Version: 1.6

YAML Reference

This page lists all components, interpolation variables and interpolation macros that can be used when defining a low code YAML file.

For the technical JSON schema definition that low code manifests are validated against, see here.

Components

DeclarativeSource `object`

An API source that extracts data according to its declarative components.

Properties:

check
Type:
- #/definitions/CheckStream
- #/definitions/CheckDynamicStream
streams array
dynamic_streams array #/definitions/DynamicDeclarativeStream
version string
The version of the Airbyte CDK used to build and test the source.
schemas #/definitions/Schemas
spec #/definitions/Spec
concurrency_level #/definitions/ConcurrencyLevel
api_budget #/definitions/HTTPAPIBudget
max_concurrent_async_job_count integerstring
Maximum number of concurrent asynchronous jobs to run. This property is only relevant for sources/streams that support asynchronous job execution through the AsyncRetriever (e.g. a report-based stream that initiates a job, polls the job status, and then fetches the job results). This is often set by the API's maximum number of concurrent jobs on the account level. Refer to the API's documentation for this information.
Examples:
```
3
```
```
{{ config['max_concurrent_async_job_count'] }}
```
metadata object
For internal Airbyte use only - DO NOT modify manually. Used by consumers of declarative manifests for storing related metadata.
description string
A description of the connector. It will be presented on the Source documentation page.

AddedFieldDefinition `object`

Defines the field to add on a record.

Properties:

path array
List of strings defining the path where to add the value on the record.
Examples:
```
[
  "segment_id"
]
```
```
[
  "metadata",
  "segment_id"
]
```
value string
Value of the new field. Use {{ record['existing_field'] }} syntax to refer to other fields in the record.
Available variables:
Examples:
```
{{ record['updates'] }}
```
```
{{ record['MetaData']['LastUpdatedTime'] }}
```
```
{{ stream_partition['segment_id'] }}
```
value_type #/definitions/ValueType
Type of the value. If not specified, the type will be inferred from the value.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

AddFields `object`

Transformation which adds field to an output record. The path of the added field can be nested.

Properties:

fields array #/definitions/AddedFieldDefinition
List of transformations (path and corresponding value) that will be added to the record.
condition string
Fields will be added if expression is evaluated to True.
Available variables:
Examples:
```
{{ property|string == '' }}
```
```
{{ property is integer }}
```
```
{{ property|length > 5 }}
```
```
{{ property == 'some_string_to_match' }}
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

ApiKeyAuthenticator `object`

Authenticator for requests authenticated with an API token injected as an HTTP request header.

Properties:

api_token string
The API key to inject in the request. Fill it in the user inputs.
Available variables:
- config
Examples:
```
{{ config['api_key'] }}
```
```
Token token={{ config['api_key'] }}
```
header string
The name of the HTTP header that will be set to the API key. This setting is deprecated, use inject_into instead. Header and inject_into can not be defined at the same time.
Available variables:
- config
Examples:
```
Authorization
```
```
Api-Token
```
```
X-Auth-Token
```
inject_into #/definitions/RequestOption
Configure how the API Key will be sent in requests to the source API. Either inject_into or header has to be defined.
Examples:
```
{
  "inject_into": "header",
  "field_name": "Authorization"
}
```
```
{
  "inject_into": "request_parameter",
  "field_name": "authKey"
}
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

AuthFlow `object`

Additional and optional specification object to describe what an 'advanced' Auth flow would need to function.

A connector should be able to fully function with the configuration as described by the ConnectorSpecification in a 'basic' mode.
The 'advanced' mode provides easier UX for the user with UI improvements and automations. However, this requires further setup on the server side by instance or workspace admins beforehand. The trade-off is that the user does not have to provide as many technical inputs anymore and the auth process is faster and easier to complete.

Properties:

auth_flow_type string
The type of auth to use
predicate_key array
JSON path to a field in the connectorSpecification that should exist for the advanced auth to be applicable.
Example:
```
[
  "credentials",
  "auth_type"
]
```
predicate_value string
Value of the predicate_key fields for the advanced auth to be applicable.
Example:
```
Oauth
```
oauth_config_specification #/definitions/OAuthConfigSpecification

BasicHttpAuthenticator `object`

Authenticator for requests authenticated with the Basic HTTP authentication scheme, which encodes a username and an optional password in the Authorization request header.

Properties:

username string
The username that will be combined with the password, base64 encoded and used to make requests. Fill it in the user inputs.
Available variables:
- config
Examples:
```
{{ config['username'] }}
```
```
{{ config['api_key'] }}
```
password string
The password that will be combined with the username, base64 encoded and used to make requests. Fill it in the user inputs.
Available variables:
- config
Examples:
```
{{ config['password'] }}
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

BearerAuthenticator `object`

Authenticator for requests authenticated with a bearer token injected as a request header of the form Authorization: Bearer <token>.

Properties:

api_token string
Token to inject as request header for authenticating with the API.
Available variables:
- config
Examples:
```
{{ config['api_key'] }}
```
```
{{ config['token'] }}
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

SelectiveAuthenticator `object`

Authenticator that selects concrete authenticator based on config property.

Properties:

authenticator_selection_path array
Path of the field in config with selected authenticator name
Examples:
```
[
  "auth"
]
```
```
[
  "auth",
  "type"
]
```

authenticators `object`

Authenticators to select from.

Example:

{
  "authenticators": {
    "token": "#/definitions/ApiKeyAuthenticator",
    "oauth": "#/definitions/OAuthAuthenticator",
    "jwt": "#/definitions/JwtAuthenticator"
  }
}

$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CheckStream `object`

Defines the streams to try reading when running a check operation.

Properties:

stream_names array
Names of the streams to try reading from when running a check operation.
Examples:
```
[
  "users"
]
```
```
[
  "users",
  "contacts"
]
```
dynamic_streams_check_configs array #/definitions/DynamicStreamCheckConfig

DynamicStreamCheckConfig `object`

Properties:

dynamic_stream_name string
The dynamic stream name.
stream_count integer
The number of streams to attempt reading from during a check operation. If stream_count exceeds the total number of available streams, the minimum of the two values will be used.

CheckDynamicStream `object`

(This component is experimental. Use at your own risk.) Defines the dynamic streams to try reading when running a check operation.

Properties:

stream_count integer
Numbers of the streams to try reading from when running a check operation.
use_check_availability boolean
Enables stream check availability. This field is automatically set by the CDK.

CompositeErrorHandler `object`

Error handler that sequentially iterates over a list of error handlers.

Properties:

error_handlers array
List of error handlers to iterate on to determine how to handle a failed response.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

ConcurrencyLevel `object`

Defines the amount of parallelization for the streams that are being synced. The factor of parallelization is how many partitions or streams are synced at the same time. For example, with a concurrency_level of 10, ten streams or partitions of data will processed at the same time. Note that a value of 1 could create deadlock if a stream has a very high number of partitions.

Properties:

default_concurrency
The amount of concurrency that will applied during a sync. This value can be hardcoded or user-defined in the config if different users have varying volume thresholds in the target API.
Type:
- integer
- string
Available variables:
- config
Examples:
```
10
```
```
{{ config['num_workers'] or 10 }}
```
max_concurrency integer
The maximum level of concurrency that will be used during a sync. This becomes a required field when the default_concurrency derives from the config, because it serves as a safeguard against a user-defined threshold that is too high.
Examples:
```
20
```
```
100
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

ConstantBackoffStrategy `object`

Backoff strategy with a constant backoff interval.

Properties:

backoff_time_in_seconds
Backoff time in seconds.
Type:
- number
- string
Available variables:
- config
Examples:
```
30
```
```
30.5
```
```
{{ config['backoff_time'] }}
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CursorPagination `object`

Pagination strategy that evaluates an interpolated string to define the next page to fetch.

Properties:

cursor_value string
Value of the cursor defining the next page to fetch.
Available variables:
Examples:
```
{{ headers.link.next.cursor }}
```
```
{{ last_record['key'] }}
```
```
{{ response['nextPage'] }}
```
page_size integer
The number of records to include in each pages.
Example:
```
100
```
stop_condition string
Template string evaluating when to stop paginating.
Available variables:
Examples:
```
{{ response.data.has_more is false }}
```
```
{{ 'next' not in headers['link'] }}
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomAuthenticator `object`

Authenticator component whose behavior is derived from a custom code implementation of the connector.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom authentication strategy. Has to be a sub class of DeclarativeAuthenticator. The format is source_<name>.<package>.<class_name>.
Example:
```
source_railz.components.ShortLivedTokenAuthenticator
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomBackoffStrategy `object`

Backoff strategy component whose behavior is derived from a custom code implementation of the connector.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom backoff strategy. The format is source_<name>.<package>.<class_name>.
Example:
```
source_railz.components.MyCustomBackoffStrategy
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomErrorHandler `object`

Error handler component whose behavior is derived from a custom code implementation of the connector.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom error handler. The format is source_<name>.<package>.<class_name>.
Example:
```
source_railz.components.MyCustomErrorHandler
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomIncrementalSync `object`

Incremental component whose behavior is derived from a custom code implementation of the connector.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom incremental sync. The format is source_<name>.<package>.<class_name>.
Example:
```
source_railz.components.MyCustomIncrementalSync
```
cursor_field string
The location of the value on a record that will be used as a bookmark during sync.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomPaginationStrategy `object`

Pagination strategy component whose behavior is derived from a custom code implementation of the connector.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom pagination strategy. The format is source_<name>.<package>.<class_name>.
Example:
```
source_railz.components.MyCustomPaginationStrategy
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomRecordExtractor `object`

Record extractor component whose behavior is derived from a custom code implementation of the connector.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom record extraction strategy. The format is source_<name>.<package>.<class_name>.
Example:
```
source_railz.components.MyCustomRecordExtractor
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomRecordFilter `object`

Record filter component whose behavior is derived from a custom code implementation of the connector.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom record filter strategy. The format is source_<name>.<package>.<class_name>.
Example:
```
source_railz.components.MyCustomCustomRecordFilter
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomRequester `object`

Requester component whose behavior is derived from a custom code implementation of the connector.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom requester strategy. The format is source_<name>.<package>.<class_name>.
Example:
```
source_railz.components.MyCustomRecordExtractor
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomRetriever `object`

Retriever component whose behavior is derived from a custom code implementation of the connector.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom retriever strategy. The format is source_<name>.<package>.<class_name>.
Example:
```
source_railz.components.MyCustomRetriever
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomPartitionRouter `object`

Partition router component whose behavior is derived from a custom code implementation of the connector.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom partition router. The format is source_<name>.<package>.<class_name>.
Example:
```
source_railz.components.MyCustomPartitionRouter
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomSchemaLoader `object`

Schema Loader component whose behavior is derived from a custom code implementation of the connector.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom schema loader. The format is source_<name>.<package>.<class_name>.
Example:
```
source_railz.components.MyCustomSchemaLoader
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomSchemaNormalization `object`

Schema normalization component whose behavior is derived from a custom code implementation of the connector.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom normalization. The format is source_<name>.<package>.<class_name>.
Example:
```
source_amazon_seller_partner.components.LedgerDetailedViewReportsTypeTransformer
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomStateMigration `object`

Apply a custom transformation on the input state.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom state migration. The format is source_<name>.<package>.<class_name>.
Example:
```
source_railz.components.MyCustomStateMigration
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

CustomTransformation `object`

Transformation component whose behavior is derived from a custom code implementation of the connector.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom transformation. The format is source_<name>.<package>.<class_name>.
Example:
```
source_railz.components.MyCustomTransformation
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

LegacyToPerPartitionStateMigration `object`

Transforms the input state for per-partitioned streams from the legacy format to the low-code format. The cursor field and partition ID fields are automatically extracted from the stream's DatetimebasedCursor and SubstreamPartitionRouter. Example input state: { "13506132": { "last_changed": "2022-12-27T08:34:39+00:00" } Example output state: { "partition": {"id": "13506132"}, "cursor": {"last_changed": "2022-12-27T08:34:39+00:00"} }

Properties:

IncrementingCountCursor `object`

Cursor that allows for incremental sync according to a continuously increasing integer.

Properties:

cursor_field string
The location of the value on a record that will be used as a bookmark during sync. To ensure no data loss, the API must return records in ascending order based on the cursor field. Nested fields are not supported, so the field must be at the top level of the record. You can use a combination of Add Field and Remove Field transformations to move the nested field to the top.
Available variables:
- config
Examples:
```
created_at
```
```
{{ config['record_cursor'] }}
```
start_value
The value that determines the earliest record that should be synced.
Type:
- string
- integer
Available variables:
- config
Examples:
```
0
```
```
{{ config['start_value'] }}
```
start_value_option #/definitions/RequestOption
Optionally configures how the start value will be sent in requests to the source API.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

DatetimeBasedCursor `object`

Cursor to provide incremental capabilities over datetime.

Properties:

clamping object
This option is used to adjust the upper and lower boundaries of each datetime window to beginning and end of the provided target period (day, week, month)
cursor_field string
The location of the value on a record that will be used as a bookmark during sync. To ensure no data loss, the API must return records in ascending order based on the cursor field. Nested fields are not supported, so the field must be at the top level of the record. You can use a combination of Add Field and Remove Field transformations to move the nested field to the top.
Available variables:
- config
Examples:
```
created_at
```
```
{{ config['record_cursor'] }}
```
datetime_format string
The datetime format used to format the datetime values that are sent in outgoing requests to the API. Use placeholders starting with "%" to describe the format the API is using. The following placeholders are available:
- %s: Epoch unix timestamp - 1686218963
- %s_as_float: Epoch unix timestamp in seconds as float with microsecond precision - 1686218963.123456
- %ms: Epoch unix timestamp (milliseconds) - 1686218963123
- %a: Weekday (abbreviated) - Sun
- %A: Weekday (full) - Sunday
- %w: Weekday (decimal) - 0 (Sunday), 6 (Saturday)
- %d: Day of the month (zero-padded) - 01, 02, ..., 31
- %b: Month (abbreviated) - Jan
- %B: Month (full) - January
- %m: Month (zero-padded) - 01, 02, ..., 12
- %y: Year (without century, zero-padded) - 00, 01, ..., 99
- %Y: Year (with century) - 0001, 0002, ..., 9999
- %H: Hour (24-hour, zero-padded) - 00, 01, ..., 23
- %I: Hour (12-hour, zero-padded) - 01, 02, ..., 12
- %p: AM/PM indicator
- %M: Minute (zero-padded) - 00, 01, ..., 59
- %S: Second (zero-padded) - 00, 01, ..., 59
- %f: Microsecond (zero-padded to 6 digits) - 000000
- %_ms: Millisecond (zero-padded to 3 digits) - 000
- %z: UTC offset - (empty), +0000, -04:00
- %Z: Time zone name - (empty), UTC, GMT
- %j: Day of the year (zero-padded) - 001, 002, ..., 366
- %U: Week number of the year (starting Sunday) - 00, ..., 53
- %W: Week number of the year (starting Monday) - 00, ..., 53
- %c: Date and time - Tue Aug 16 21:30:00 1988
- %x: Date standard format - 08/16/1988
- %X: Time standard format - 21:30:00
- %%: Literal '%' character
Some placeholders depend on the locale of the underlying system - in most cases this locale is configured as en/US. For more information see the Python documentation.
Examples:
```
%Y-%m-%dT%H:%M:%S.%f%z
```
```
%Y-%m-%d
```
```
%s
```
```
%ms
```
```
%s_as_float
```
start_datetime
The datetime that determines the earliest record that should be synced.
Type:
- string
- #/definitions/MinMaxDatetime
Available variables:
- config
Examples:
```
2020-01-1T00:00:00Z
```
```
{{ config['start_time'] }}
```
cursor_datetime_formats array
The possible formats for the cursor field, in order of preference. The first format that matches the cursor field value will be used to parse it. If not provided, the datetime_format will be used.
cursor_granularity string
Smallest increment the datetime_format has (ISO 8601 duration) that is used to ensure the start of a slice does not overlap with the end of the previous one, e.g. for %Y-%m-%d the granularity should be P1D, for %Y-%m-%dT%H:%M:%SZ the granularity should be PT1S. Given this field is provided, step needs to be provided as well.
Example:
```
PT1S
```
end_datetime
The datetime that determines the last record that should be synced. If not provided, {{ now_utc() }} will be used.
Type:
- string
- #/definitions/MinMaxDatetime
Available variables:
- config
Examples:
```
2021-01-1T00:00:00Z
```
```
{{ now_utc() }}
```
```
{{ day_delta(-1) }}
```
end_time_option #/definitions/RequestOption
Optionally configures how the end datetime will be sent in requests to the source API.
is_data_feed boolean
A data feed API is an API that does not allow filtering and paginates the content from the most recent to the least recent. Given this, the CDK needs to know when to stop paginating and this field will generate a stop condition for pagination.
is_client_side_incremental boolean
If the target API endpoint does not take cursor values to filter records and returns all records anyway, the connector with this cursor will filter out records locally, and only emit new records from the last sync, hence incremental. This means that all records would be read from the API, but only new records will be emitted to the destination.
is_compare_strictly boolean
Set to True if the target API does not accept queries where the start time equal the end time.
global_substream_cursor boolean
This setting optimizes performance when the parent stream has thousands of partitions by storing the cursor as a single value rather than per partition. Notably, the substream state is updated only at the end of the sync, which helps prevent data loss in case of a sync failure. See more info in the docs.
lookback_window string
Time interval before the start_datetime to read data for, e.g. P1M for looking back one month.
Available variables:
- config
Examples:
```
P1D
```
```
P{{ config['lookback_days'] }}D
```
partition_field_end string
Name of the partition start time field.
Example:
```
ending_time
```
partition_field_start string
Name of the partition end time field.
Example:
```
starting_time
```
start_time_option #/definitions/RequestOption
Optionally configures how the start datetime will be sent in requests to the source API.
step string
The size of the time window (ISO8601 duration). Given this field is provided, cursor_granularity needs to be provided as well.
Examples:
```
P1W
```
```
{{ config['step_increment'] }}
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

JwtAuthenticator `object`

Authenticator for requests using JWT authentication flow.

Properties:

secret_key string
Secret used to sign the JSON web token.
Example:
```
{{ config['secret_key'] }}
```
base64_encode_secret_key boolean
When set to true, the secret key will be base64 encoded prior to being encoded as part of the JWT. Only set to "true" when required by the API.
algorithm string
Algorithm used to sign the JSON web token.
Examples:
```
ES256
```
```
HS256
```
```
RS256
```
```
{{ config['algorithm'] }}
```
token_duration integer
The amount of time in seconds a JWT token can be valid after being issued.
Examples:
```
1200
```
```
3600
```
header_prefix string
The prefix to be used within the Authentication header.
Examples:
```
Bearer
```
```
Basic
```
jwt_headers object
JWT headers used when signing JSON web token.
additional_jwt_headers object
Additional headers to be included with the JWT headers object.
jwt_payload object
JWT Payload used when signing JSON web token.
additional_jwt_payload object
Additional properties to be added to the JWT payload.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

OAuthAuthenticator `object`

Authenticator for requests using OAuth 2.0 authorization flow.

Properties:

client_id_name string
The name of the property to use to refresh the access_token.
Example:
```
custom_app_id
```
client_id string
The OAuth client ID. Fill it in the user inputs.
Examples:
```
{{ config['client_id }}
```
```
{{ config['credentials']['client_id }}
```
client_secret_name string
The name of the property to use to refresh the access_token.
Example:
```
custom_app_secret
```
client_secret string
The OAuth client secret. Fill it in the user inputs.
Examples:
```
{{ config['client_secret }}
```
```
{{ config['credentials']['client_secret }}
```
refresh_token_name string
The name of the property to use to refresh the access_token.
Example:
```
custom_app_refresh_value
```
refresh_token string
Credential artifact used to get a new access token.
Examples:
```
{{ config['refresh_token'] }}
```
```
{{ config['credentials]['refresh_token'] }}
```
token_refresh_endpoint string
The full URL to call to obtain a new access token.
Example:
```
https://connect.squareup.com/oauth2/token
```
access_token_name string
The name of the property which contains the access token in the response from the token refresh endpoint.
Example:
```
access_token
```
access_token_value string
The value of the access_token to bypass the token refreshing using refresh_token.
Example:
```
secret_access_token_value
```
expires_in_name string
The name of the property which contains the expiry date in the response from the token refresh endpoint.
Example:
```
expires_in
```
grant_type_name string
The name of the property to use to refresh the access_token.
Example:
```
custom_grant_type
```
grant_type string
Specifies the OAuth2 grant type. If set to refresh_token, the refresh_token needs to be provided as well. For client_credentials, only client id and secret are required. Other grant types are not officially supported.
Examples:
```
refresh_token
```
```
client_credentials
```

refresh_request_body `object`

Body of the request sent to get a new access token.

Example:

{
  "applicationId": "{{ config['application_id'] }}",
  "applicationSecret": "{{ config['application_secret'] }}",
  "token": "{{ config['token'] }}"
}

refresh_request_headers `object`

Headers of the request sent to get a new access token.

Example:

{
  "Authorization": "<AUTH_TOKEN>",
  "Content-Type": "application/x-www-form-urlencoded"
}

scopes `array`

List of scopes that should be granted to the access token.

Example:

[
  "crm.list.read",
  "crm.objects.contacts.read",
  "crm.schema.contacts.read"
]

token_expiry_date string
The access token expiry date.
Examples:
```
2023-04-06T07:12:10.421833+00:00
```
```
1680842386
```
token_expiry_date_format string
The format of the time to expiration datetime. Provide it if the time is returned as a date-time string instead of seconds.
Example:
```
%Y-%m-%d %H:%M:%S.%f+00:00
```
refresh_token_updater
When the token updater is defined, new refresh tokens, access tokens and the access token expiry date are written back from the authentication response to the config object. This is important if the refresh token can only used once.
profile_assertion #/definitions/JwtAuthenticator
The authenticator being used to authenticate the client authenticator.
use_profile_assertion boolean
Enable using profile assertion as a flow for OAuth authorization.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

DeclarativeStream `object`

A stream whose behavior is described by a set of declarative low code components.

Properties:

name string
The stream name.
retriever
Component used to coordinate how records are extracted across stream slices and request pages.
Type:
incremental_sync
Component used to fetch data incrementally based on a time field in the data.
Type:
primary_key #/definitions/PrimaryKey
schema_loader
One or many schema loaders can be used to retrieve the schema for the current stream. When multiple schema loaders are defined, schema properties will be merged together. Schema loaders defined first taking precedence in the event of a conflict.
Type:
transformations array
A list of transformations to be applied to each output record.
state_migrations array
Array of state migrations to be applied on the input state
file_uploader object
(experimental) Describes how to fetch a file
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

HTTPAPIBudget `object`

Defines how many requests can be made to the API in a given time frame. HTTPAPIBudget extracts the remaining call count and the reset time from HTTP response headers using the header names provided by ratelimit_remaining_header and ratelimit_reset_header. Only requests using HttpRequester are rate-limited; custom components that bypass HttpRequester are not covered by this budget.

Properties:

policies array
List of call rate policies that define how many calls are allowed.
ratelimit_reset_header string
The HTTP response header name that indicates when the rate limit resets.
ratelimit_remaining_header string
The HTTP response header name that indicates the number of remaining allowed calls.
status_codes_for_ratelimit_hit array
List of HTTP status codes that indicate a rate limit has been hit.

FixedWindowCallRatePolicy `object`

A policy that allows a fixed number of calls within a specific time window.

Properties:

period string
The time interval for the rate limit window.
call_limit integer
The maximum number of calls allowed within the period.
matchers array #/definitions/HttpRequestRegexMatcher
List of matchers that define which requests this policy applies to.

MovingWindowCallRatePolicy `object`

A policy that allows a fixed number of calls within a moving time window.

Properties:

rates array #/definitions/Rate
List of rates that define the call limits for different time intervals.
matchers array #/definitions/HttpRequestRegexMatcher
List of matchers that define which requests this policy applies to.

UnlimitedCallRatePolicy `object`

A policy that allows unlimited calls for specific requests.

Properties:

matchers array #/definitions/HttpRequestRegexMatcher
List of matchers that define which requests this policy applies to.

Rate `object`

Defines a rate limit with a specific number of calls allowed within a time interval.

Properties:

limit
The maximum number of calls allowed within the interval.
Type:
- integer
- string
Available variables:
- config
interval string
The time interval for the rate limit.
Examples:
```
PT1H
```
```
P1D
```

HttpRequestRegexMatcher `object`

Matches HTTP requests based on method, base URL, URL path pattern, query parameters, and headers. Use url_base to specify the scheme and host (without trailing slash) and url_path_pattern to apply a regex to the request path.

Properties:

method string
The HTTP method to match (e.g., GET, POST).
url_base string
The base URL (scheme and host, e.g. "https://api.example.com") to match.
url_path_pattern string
A regular expression pattern to match the URL path.
params object
The query parameters to match.
headers object
The headers to match.

DefaultErrorHandler `object`

Component defining how to handle errors. Default behavior includes only retrying server errors (HTTP 5XX) and too many requests (HTTP 429) with an exponential backoff.

Properties:

backoff_strategies array
List of backoff strategies to use to determine how long to wait before retrying a retryable request.
max_retries integer
The maximum number of time to retry a retryable request before giving up and failing.
Examples:
```
5
```
```
0
```
```
10
```
response_filters array #/definitions/HttpResponseFilter
List of response filters to iterate on when deciding how to handle an error. When using an array of multiple filters, the filters will be applied sequentially and the response will be selected if it matches any of the filter's predicate.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

DefaultPaginator `object`

Default pagination implementation to request pages of results with a fixed size until the pagination strategy no longer returns a next_page_token.

Properties:

pagination_strategy
Strategy defining how records are paginated.
Type:
page_size_option #/definitions/RequestOption
page_token_option
Type:
- #/definitions/RequestOption
- #/definitions/RequestPath
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

DpathExtractor `object`

Record extractor that searches a decoded response over a path defined as an array of fields.

Properties:

field_path array
List of potentially nested fields describing the full path of the field to extract. Use "*" to extract all values from an array. See more info in the docs.
Available variables:
- config
Examples:
```
[
  "data"
]
```
```
[
  "data",
  "records"
]
```
```
[
  "data",
  "{{ parameters.name }}"
]
```
```
[
  "data",
  "*",
  "record"
]
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

ResponseToFileExtractor `object`

A record extractor designed for handling large responses that may exceed memory limits (to prevent OOM issues). It downloads a CSV file to disk, reads the data from disk, and deletes the file once it has been fully processed.

Properties:

$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

ExponentialBackoffStrategy `object`

Backoff strategy with an exponential backoff interval. The interval is defined as factor * 2^attempt_count.

Properties:

factor
Multiplicative constant applied on each retry.
Type:
- number
- string
Available variables:
- config
Examples:
```
5
```
```
5.5
```
```
10
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

GroupByKeyMergeStrategy

Record merge strategy that combines records according to fields on the record.

Properties:

key
The name of the field on the record whose value will be used to group properties that were retrieved through multiple API requests.
Type:
- string
- array
Examples:
```
id
```
```
[
  "parent_id",
  "end_date"
]
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

SessionTokenAuthenticator `object`

Authenticator for requests using the session token as an API key that's injected into the request.

Properties:

login_requester `#/definitions/HttpRequester`

Description of the request to perform to obtain a session token to perform data requests. The response body is expected to be a JSON object with a session token property.

Example:

{
  "type": "HttpRequester",
  "url_base": "https://my_api.com",
  "path": "/login",
  "authenticator": {
    "type": "BasicHttpAuthenticator",
    "username": "{{ config.username }}",
    "password": "{{ config.password }}"
  }
}

session_token_path array
The path in the response body returned from the login requester to the session token.
Examples:
```
[
  "access_token"
]
```
```
[
  "result",
  "token"
]
```
expiration_duration string
The duration in ISO 8601 duration notation after which the session token expires, starting from the time it was obtained. Omitting it will result in the session token being refreshed for every request.
Examples:
```
PT1H
```
```
P1D
```
request_authentication
Authentication method to use for requests sent to the API, specifying how to inject the session token.
Type:
- #/definitions/SessionTokenRequestApiKeyAuthenticator
- #/definitions/SessionTokenRequestBearerAuthenticator
decoder
Component used to decode the response.
Type:
- #/definitions/JsonDecoder
- #/definitions/XmlDecoder
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

SessionTokenRequestApiKeyAuthenticator `object`

Authenticator for requests using the session token as an API key that's injected into the request.

Properties:

inject_into `#/definitions/RequestOption`

Configure how the API Key will be sent in requests to the source API.

Examples:

{
  "inject_into": "header",
  "field_name": "Authorization"
}

{
  "inject_into": "request_parameter",
  "field_name": "authKey"
}

SessionTokenRequestBearerAuthenticator

Authenticator for requests using the session token as a standard bearer token.

Properties:

HttpRequester `object`

Requester submitting HTTP requests and extracting records from the response.

Properties:

url_base string
Deprecated, use the url instead. Base URL of the API source. Do not put sensitive information (e.g. API tokens) into this field - Use the Authenticator component for this.
Available variables:
Examples:
```
https://connect.squareup.com/v2
```
```
{{ config['base_url'] or 'https://app.posthog.com'}}/api
```
```
https://connect.squareup.com/v2/quotes/{{ stream_partition['id'] }}/quote_line_groups
```
```
https://example.com/api/v1/resource/{{ next_page_token['id'] }}
```
url string
The URL of the source API endpoint. Do not put sensitive information (e.g. API tokens) into this field - Use the Authenticator component for this.
Available variables:
Examples:
```
https://connect.squareup.com/v2
```
```
{{ config['url'] or 'https://app.posthog.com'}}/api
```
```
https://connect.squareup.com/v2/quotes/{{ stream_partition['id'] }}/quote_line_groups
```
```
https://example.com/api/v1/resource/{{ next_page_token['id'] }}
```
path string
Deprecated, use the url instead. Path the specific API endpoint that this stream represents. Do not put sensitive information (e.g. API tokens) into this field - Use the Authenticator component for this.
Available variables:
Examples:
```
/products
```
```
/quotes/{{ stream_partition['id'] }}/quote_line_groups
```
```
/trades/{{ config['symbol_id'] }}/history
```
http_method string
The HTTP method used to fetch data from the source (can be GET or POST).
Examples:
```
GET
```
```
POST
```
authenticator
Authentication method to use for requests sent to the API.
Type:
fetch_properties_from_endpoint #/definitions/PropertiesFromEndpoint
Allows for retrieving a dynamic set of properties from an API endpoint which can be injected into outbound request using the stream_partition.extra_fields.

request_parameters

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

Type:

object
string

Available variables:

next_page_token
stream_interval
stream_partition
stream_slice

Examples:

{
  "unit": "day"
}

{
  "query": "last_event_time BETWEEN TIMESTAMP \"{{ stream_interval.start_time }}\" AND TIMESTAMP \"{{ stream_interval.end_time }}\""
}

{
  "searchIn": "{{ ','.join(config.get('search_in', [])) }}"
}

{
  "sort_by[asc]": "updated_at"
}

request_headers
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
Type:
- object
- string
Available variables:
Examples:
```
{
  "Output-Format": "JSON"
}
```
```
{
  "Version": "{{ config['version'] }}"
}
```
request_body_data
Specifies how to populate the body of the request with a non-JSON payload. Plain text will be sent as is, whereas objects will be converted to a urlencoded form.
Type:
- object
- string
Available variables:
Example:
```
[{"clause": {"type": "timestamp", "operator": 10, "parameters":
    [{"value": {{ stream_interval['start_time'] | int * 1000 }} }]
  }, "orderBy": 1, "columnName": "Timestamp"}]/
```
request_body_json
Specifies how to populate the body of the request with a JSON payload. Can contain nested objects.
Type:
- object
- string
Available variables:
Examples:
```
{
  "sort_order": "ASC",
  "sort_field": "CREATED_AT"
}
```
```
{
  "key": "{{ config['value'] }}"
}
```
```
{
  "sort": {
    "field": "updated_at",
    "order": "ascending"
  }
}
```

request_body

Specifies how to populate the body of the request with a payload. Can contain nested objects.

Type:

#/definitions/RequestBodyPlainText
#/definitions/RequestBodyUrlEncodedForm
#/definitions/RequestBodyJsonObject
#/definitions/RequestBodyGraphQL

Available variables:

next_page_token
stream_interval
stream_partition
stream_slice

Examples:

{
  "type": "RequestBodyJsonObject",
  "value": {
    "sort_order": "ASC",
    "sort_field": "CREATED_AT"
  }
}

{
  "type": "RequestBodyJsonObject",
  "value": {
    "key": "{{ config['value'] }}"
  }
}

{
  "type": "RequestBodyJsonObject",
  "value": {
    "sort": {
      "field": "updated_at",
      "order": "ascending"
    }
  }
}

{
  "type": "RequestBodyPlainText",
  "value": "plain_text_body"
}

{
  "type": "RequestBodyUrlEncodedForm",
  "value": {
    "param1": "value1",
    "param2": "{{ config['param2_value'] }}"
  }
}

{
  "type": "RequestBodyGraphQL",
  "value": {
    "query": {
      "param1": "value1",
      "param2": "{{ config['param2_value'] }}"
    }
  }
}

error_handler
Error handler component that defines how to handle errors.
Type:
use_cache boolean
Enables stream requests caching. This field is automatically set by the CDK.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

HttpResponseFilter `object`

A filter that is used to select on properties of the HTTP response received. When used with additional filters, a response will be selected if it matches any of the filter's criteria.

Properties:

action string
Action to execute if a response matches the filter.
Examples:
```
SUCCESS
```
```
FAIL
```
```
RETRY
```
```
IGNORE
```
```
RATE_LIMITED
```
failure_type string
Failure type of traced exception if a response matches the filter.
Examples:
```
system_error
```
```
config_error
```
```
transient_error
```
error_message string
Error Message to display if the response matches the filter.
Available variables:
error_message_contains string
Match the response if its error message contains the substring.
http_codes array
Match the response if its HTTP code is included in this list.
Examples:
```
[
  420,
  429
]
```
```
[
  500
]
```

predicate `string`

Match the response if the predicate evaluates to true.

Available variables:

response
headers

Examples:

{{ 'Too much requests' in response }}

{{ 'error_code' in response and response['error_code'] == 'ComplexityException' }}

$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

ComplexFieldType `object`

(This component is experimental. Use at your own risk.) Represents a complex field type.

Properties:

field_type string
items
Type:
- string
- #/definitions/ComplexFieldType

TypesMap `object`

(This component is experimental. Use at your own risk.) Represents a mapping between a current type and its corresponding target type.

Properties:

target_type
Type:
- string
- array
- #/definitions/ComplexFieldType
current_type
Type:
- string
- array
condition string
Available variables:
- raw_schema

SchemaTypeIdentifier `object`

(This component is experimental. Use at your own risk.) Identifies schema details for dynamic schema extraction and processing.

Properties:

schema_pointer array
List of nested fields defining the schema field path to extract. Defaults to [].
Available variables:
- config
key_pointer array
List of potentially nested fields describing the full path of the field key to extract.
Available variables:
- config
type_pointer array
List of potentially nested fields describing the full path of the field type to extract.
Available variables:
- config
types_mapping array #/definitions/TypesMap
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

DynamicSchemaLoader `object`

(This component is experimental. Use at your own risk.) Loads a schema by extracting data from retrieved records.

Properties:

retriever
Component used to coordinate how records are extracted across stream slices and request pages.
Type:
schema_filter
Responsible for filtering fields to be added to json schema.
Type:
- #/definitions/RecordFilter
- #/definitions/CustomRecordFilter
schema_transformations array
A list of transformations to be applied to the schema.
schema_type_identifier #/definitions/SchemaTypeIdentifier
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

InlineSchemaLoader `object`

Loads a schema that is defined directly in the manifest file.

Properties:

schema object
Describes a streams' schema. Refer to the <a href="https://docs.airbyte.com/understanding-airbyte/supported-data-types/">Data Types documentation</a> for more details on which types are valid.

JsonFileSchemaLoader `object`

Loads the schema from a json file.

Properties:

file_path string
Path to the JSON file defining the schema. The path is relative to the connector module's root.
Available variables:
- config
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

JsonDecoder `object`

Select 'JSON' if the response is formatted as a JSON object.

Properties:

JsonlDecoder `object`

Select 'JSON Lines' if the response consists of JSON objects separated by new lines ('\n') in JSONL format.

Properties:

KeysToLower `object`

A transformation that renames all keys to lower case.

Properties:

$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

KeysToSnakeCase `object`

A transformation that renames all keys to snake case.

Properties:

$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

FlattenFields `object`

A transformation that flatten record to single level format.

Properties:

flatten_lists boolean
Whether to flatten lists or leave it as is. Default is True.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

KeyTransformation `object`

Properties:

prefix string
Prefix to add for object keys. If not provided original keys remain unchanged.
Example:
```
flattened_
```
suffix string
Suffix to add for object keys. If not provided original keys remain unchanged.
Example:
```
_flattened
```

DpathFlattenFields `object`

A transformation that flatten field values to the to top of the record.

Properties:

field_path array
A path to field that needs to be flattened.
Examples:
```
[
  "data"
]
```
```
[
  "data",
  "*",
  "field"
]
```
delete_origin_value boolean
Whether to delete the origin value or keep it. Default is False.
replace_record boolean
Whether to replace the origin record or not. Default is False.
key_transformation object #/definitions/KeyTransformation
Transformation for object keys. If not provided, original key will be used.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

KeysReplace `object`

A transformation that replaces symbols in keys.

Properties:

old string
Old value to replace.
Available variables:
Examples:
```
 
```
```
{{ record.id }}
```
```
{{ config['id'] }}
```
```
{{ stream_slice['id'] }}
```
new string
New value to set.
Available variables:
Examples:
```
_
```
```
{{ record.id }}
```
```
{{ config['id'] }}
```
```
{{ stream_slice['id'] }}
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

IterableDecoder `object`

Select 'Iterable' if the response consists of strings separated by new lines (\n). The string will then be wrapped into a JSON object with the record key.

Properties:

XmlDecoder `object`

Select 'XML' if the response consists of XML-formatted data.

Properties:

CustomDecoder `object`

Use this to implement custom decoder logic.

Properties:

class_name string
Fully-qualified name of the class that will be implementing the custom decoding. Has to be a sub class of Decoder. The format is source_<name>.<package>.<class_name>.
Example:
```
source_amazon_ads.components.GzipJsonlDecoder
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

ZipfileDecoder `object`

Select 'ZIP file' for response data that is returned as a zipfile. Requires specifying an inner data type/decoder to parse the unzipped data.

Properties:

decoder
Parser to parse the decompressed data from the zipfile(s).
Type:

ListPartitionRouter `object`

A Partition router that specifies a list of attributes where each attribute describes a portion of the complete data set for a stream. During a sync, each value is iterated over and can be used as input to outbound API requests.

Properties:

cursor_field string
While iterating over list values, the name of field used to reference a list value. The partition value can be accessed with string interpolation. e.g. "{{ stream_partition['my_key'] }}" where "my_key" is the value of the cursor_field.
Available variables:
- config
Examples:
```
section
```
```
{{ config['section_key'] }}
```
values
The list of attributes being iterated over and used as input for the requests made to the source API.
Type:
- string
- array
Available variables:
- config
Examples:
```
[
  "section_a",
  "section_b",
  "section_c"
]
```
```
{{ config['sections'] }}
```
request_option #/definitions/RequestOption
A request option describing where the list value should be injected into and under what field name if applicable.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

MinMaxDatetime `object`

Compares the provided date against optional minimum or maximum times. The max_datetime serves as the ceiling and will be returned when datetime exceeds it. The min_datetime serves as the floor.

Properties:

datetime string
Datetime value.
Available variables:
- config
Examples:
```
2021-01-01
```
```
2021-01-01T00:00:00Z
```
```
{{ config['start_time'] }}
```
datetime_format string
Format of the datetime value. Defaults to "%Y-%m-%dT%H:%M:%S.%f%z" if left empty. Use placeholders starting with "%" to describe the format the API is using. The following placeholders are available:
- %s: Epoch unix timestamp - 1686218963
- %s_as_float: Epoch unix timestamp in seconds as float with microsecond precision - 1686218963.123456
- %ms: Epoch unix timestamp - 1686218963123
- %a: Weekday (abbreviated) - Sun
- %A: Weekday (full) - Sunday
- %w: Weekday (decimal) - 0 (Sunday), 6 (Saturday)
- %d: Day of the month (zero-padded) - 01, 02, ..., 31
- %b: Month (abbreviated) - Jan
- %B: Month (full) - January
- %m: Month (zero-padded) - 01, 02, ..., 12
- %y: Year (without century, zero-padded) - 00, 01, ..., 99
- %Y: Year (with century) - 0001, 0002, ..., 9999
- %H: Hour (24-hour, zero-padded) - 00, 01, ..., 23
- %I: Hour (12-hour, zero-padded) - 01, 02, ..., 12
- %p: AM/PM indicator
- %M: Minute (zero-padded) - 00, 01, ..., 59
- %S: Second (zero-padded) - 00, 01, ..., 59
- %f: Microsecond (zero-padded to 6 digits) - 000000, 000001, ..., 999999
- %_ms: Millisecond (zero-padded to 3 digits) - 000, 001, ..., 999
- %z: UTC offset - (empty), +0000, -04:00
- %Z: Time zone name - (empty), UTC, GMT
- %j: Day of the year (zero-padded) - 001, 002, ..., 366
- %U: Week number of the year (Sunday as first day) - 00, 01, ..., 53
- %W: Week number of the year (Monday as first day) - 00, 01, ..., 53
- %c: Date and time representation - Tue Aug 16 21:30:00 1988
- %x: Date representation - 08/16/1988
- %X: Time representation - 21:30:00
- %%: Literal '%' character
Some placeholders depend on the locale of the underlying system - in most cases this locale is configured as en/US. For more information see the Python documentation.
Examples:
```
%Y-%m-%dT%H:%M:%S.%f%z
```
```
%Y-%m-%d
```
```
%s
```
max_datetime string
Ceiling applied on the datetime value. Must be formatted with the datetime_format field.
Available variables:
- config
Examples:
```
2021-01-01T00:00:00Z
```
```
2021-01-01
```
min_datetime string
Floor applied on the datetime value. Must be formatted with the datetime_format field.
Available variables:
- config
Examples:
```
2010-01-01T00:00:00Z
```
```
2010-01-01
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

NoAuth `object`

Authenticator for requests requiring no authentication.

Properties:

$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

NoPagination `object`

Pagination implementation that never returns a next page.

Properties:

OAuthConfigSpecification `object`

Specification describing how an 'advanced' Auth flow would need to function.

Properties:

oauth_user_input_from_connector_config_specification object
OAuth specific blob. This is a Json Schema used to validate Json configurations used as input to OAuth. Must be a valid non-nested JSON that refers to properties from ConnectorSpecification.connectionSpecification using special annotation 'path_in_connector_config'. These are input values the user is entering through the UI to authenticate to the connector, that might also shared as inputs for syncing data via the connector. Examples: if no connector values is shared during oauth flow, oauth_user_input_from_connector_config_specification=[] if connector values such as 'app_id' inside the top level are used to generate the API url for the oauth flow, oauth_user_input_from_connector_config_specification={ app_id: { type: string path_in_connector_config: ['app_id'] } } if connector values such as 'info.app_id' nested inside another object are used to generate the API url for the oauth flow, oauth_user_input_from_connector_config_specification={ app_id: { type: string path_in_connector_config: ['info', 'app_id'] } }
Examples:
```
{
  "app_id": {
    "type": "string",
    "path_in_connector_config": [
      "app_id"
    ]
  }
}
```
```
{
  "app_id": {
    "type": "string",
    "path_in_connector_config": [
      "info",
      "app_id"
    ]
  }
}
```
oauth_connector_input_specification object
The DeclarativeOAuth specific blob. Pertains to the fields defined by the connector relating to the OAuth flow.

Interpolation capabilities:
- The variables placeholders are declared as {{my_var}}.
- The nested resolution variables like {{ {{my_nested_var}} }} is allowed as well.
- The allowed interpolation context is:
  - base64Encoder - encode to base64, {{ {{my_var_a}}:{{my_var_b}} | base64Encoder }}
  - base64Decorer - decode from base64 encoded string, {{ {{my_string_variable_or_string_value}} | base64Decoder }}
  - urlEncoder - encode the input string to URL-like format, {{ https://test.host.com/endpoint | urlEncoder}}
  - urlDecorer - decode the input url-encoded string into text format, {{ urlDecoder:https%3A%2F%2Fairbyte.io | urlDecoder}}
  - codeChallengeS256 - get the codeChallenge encoded value to provide additional data-provider specific authorisation values, {{ {{state_value}} | codeChallengeS256 }}
Examples:
- The TikTok Marketing DeclarativeOAuth spec: { "oauth_connector_input_specification": { "type": "object", "additionalProperties": false, "properties": { "consent_url": "https://ads.tiktok.com/marketing_api/auth?{{client_id_key}}={{client_id_value}}&{{redirect_uri_key}}={{ {{redirect_uri_value}} | urlEncoder}}&{{state_key}}={{state_value}}", "access_token_url": "https://business-api.tiktok.com/open_api/v1.3/oauth2/access_token/", "access_token_params": { "{{ auth_code_key }}": "{{ auth_code_value }}", "{{ client_id_key }}": "{{ client_id_value }}", "{{ client_secret_key }}": "{{ client_secret_value }}" }, "access_token_headers": { "Content-Type": "application/json", "Accept": "application/json" }, "extract_output": ["data.access_token"], "client_id_key": "app_id", "client_secret_key": "secret", "auth_code_key": "auth_code" } } }
complete_oauth_output_specification object
OAuth specific blob. This is a Json Schema used to validate Json configurations produced by the OAuth flows as they are returned by the distant OAuth APIs. Must be a valid JSON describing the fields to merge back to ConnectorSpecification.connectionSpecification. For each field, a special annotation path_in_connector_config can be specified to determine where to merge it, Examples: complete_oauth_output_specification={ refresh_token: { type: string, path_in_connector_config: ['credentials', 'refresh_token'] } }
Example:
```
{
  "refresh_token": {
    "type": "string,",
    "path_in_connector_config": [
      "credentials",
      "refresh_token"
    ]
  }
}
```
complete_oauth_server_input_specification object
OAuth specific blob. This is a Json Schema used to validate Json configurations persisted as Airbyte Server configurations. Must be a valid non-nested JSON describing additional fields configured by the Airbyte Instance or Workspace Admins to be used by the server when completing an OAuth flow (typically exchanging an auth code for refresh token). Examples: complete_oauth_server_input_specification={ client_id: { type: string }, client_secret: { type: string } }
Example:
```
{
  "client_id": {
    "type": "string"
  },
  "client_secret": {
    "type": "string"
  }
}
```
complete_oauth_server_output_specification object
OAuth specific blob. This is a Json Schema used to validate Json configurations persisted as Airbyte Server configurations that also need to be merged back into the connector configuration at runtime. This is a subset configuration of complete_oauth_server_input_specification that filters fields out to retain only the ones that are necessary for the connector to function with OAuth. (some fields could be used during oauth flows but not needed afterwards, therefore they would be listed in the complete_oauth_server_input_specification but not complete_oauth_server_output_specification) Must be a valid non-nested JSON describing additional fields configured by the Airbyte Instance or Workspace Admins to be used by the connector when using OAuth flow APIs. These fields are to be merged back to ConnectorSpecification.connectionSpecification. For each field, a special annotation path_in_connector_config can be specified to determine where to merge it, Examples: complete_oauth_server_output_specification={ client_id: { type: string, path_in_connector_config: ['credentials', 'client_id'] }, client_secret: { type: string, path_in_connector_config: ['credentials', 'client_secret'] } }
Example:
```
{
  "client_id": {
    "type": "string,",
    "path_in_connector_config": [
      "credentials",
      "client_id"
    ]
  },
  "client_secret": {
    "type": "string,",
    "path_in_connector_config": [
      "credentials",
      "client_secret"
    ]
  }
}
```

OffsetIncrement `object`

Pagination strategy that returns the number of records reads so far and returns it as the next page token.

Properties:

page_size
The number of records to include in each pages.
Type:
- integer
- string
Available variables:
- config
- response
Examples:
```
100
```
```
{{ config['page_size'] }}
```
inject_on_first_request boolean
Using the offset with value 0 during the first request
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

PageIncrement `object`

Pagination strategy that returns the number of pages reads so far and returns it as the next page token.

Properties:

page_size
The number of records to include in each pages.
Type:
- integer
- string
Available variables:
- config
Examples:
```
100
```
```
100
```
```
{{ config['page_size'] }}
```
start_from_page integer
Index of the first page to request.
Examples:
```
0
```
```
1
```
inject_on_first_request boolean
Using the page number with value defined by start_from_page during the first request
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

ParentStreamConfig `object`

Describes how to construct partitions from the records retrieved from the parent stream..

Properties:

lazy_read_pointer array
If set, this will enable lazy reading, using the initial read of parent records to extract child records.
Available variables:
- config
parent_key string
The primary key of records from the parent stream that will be used during the retrieval of records for the current substream. This parent identifier field is typically a characteristic of the child records being extracted from the source API.
Examples:
```
id
```
```
{{ config['parent_record_id'] }}
```
stream
Reference to the parent stream.
Type:
- #/definitions/DeclarativeStream
- #/definitions/StateDelegatingStream
partition_field string
While iterating over parent records during a sync, the parent_key value can be referenced by using this field.
Examples:
```
parent_id
```
```
{{ config['parent_partition_field'] }}
```
request_option #/definitions/RequestOption
A request option describing where the parent key value should be injected into and under what field name if applicable.
incremental_dependency boolean
Indicates whether the parent stream should be read incrementally based on updates in the child stream.
extra_fields array
Array of field paths to include as additional fields in the stream slice. Each path is an array of strings representing keys to access fields in the respective parent record. Accessible via stream_slice.extra_fields. Missing fields are set to None.
Available variables:
- config
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

PrimaryKey

The stream field to be used to distinguish unique records. Can either be a single field, an array of fields representing a composite key, or an array of arrays representing a composite key where the fields are nested fields.

Examples:

id

[
  "code",
  "type"
]

PropertiesFromEndpoint `object`

Defines the behavior for fetching the list of properties from an API that will be loaded into the requests to extract records.

Properties:

property_field_path array
Describes the path to the field that should be extracted
Available variables:
- config
- parameters
Example:
```
[
  "name"
]
```
retriever
Requester component that describes how to fetch the properties to query from a remote API endpoint.
Type:
- #/definitions/SimpleRetriever
- #/definitions/CustomRetriever
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

PropertyChunking `object`

For APIs with restrictions on the amount of properties that can be requester per request, property chunking can be applied to make multiple requests with a subset of the properties.

Properties:

property_limit_type
The type used to determine the maximum number of properties per chunk
property_limit integer
The maximum amount of properties that can be retrieved per request according to the limit type.
record_merge_strategy #/definitions/GroupByKeyMergeStrategy
Dictates how to records that require multiple requests to get all properties should be emitted to the destination
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

QueryProperties `object`

For APIs that require explicit specification of the properties to query for, this component specifies which property fields and how they are supplied to outbound requests.

Properties:

property_list
The set of properties that will be queried for in the outbound request. This can either be statically defined or dynamic based on an API endpoint
Type:
- array
- #/definitions/PropertiesFromEndpoint
always_include_properties array
The list of properties that should be included in every set of properties when multiple chunks of properties are being requested.
property_chunking #/definitions/PropertyChunking
Defines how query properties will be grouped into smaller sets for APIs with limitations on the number of properties fetched per API request.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

RecordFilter `object`

Filter applied on a list of records.

Properties:

condition string
The predicate to filter a record. Records will be removed if evaluated to False.
Available variables:
Examples:
```
{{ record['created_at'] >= stream_interval['start_time'] }}
```
```
{{ record.status in ['active', 'expired'] }}
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

RecordSelector `object`

Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.

Properties:

extractor
Type:
- #/definitions/DpathExtractor
- #/definitions/CustomRecordExtractor
record_filter
Responsible for filtering records to be emitted by the Source.
Type:
- #/definitions/RecordFilter
- #/definitions/CustomRecordFilter
schema_normalization
Responsible for normalization according to the schema.
Type:
- #/definitions/SchemaNormalization
- #/definitions/CustomSchemaNormalization
transform_before_filtering boolean
If true, transformation will be applied before record filtering.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

SchemaNormalization `string`

Responsible for normalization according to the schema.

Examples:

Default

None

RemoveFields `object`

A transformation which removes fields from a record. The fields removed are designated using FieldPointers. During transformation, if a field or any of its parents does not exist in the record, no error is thrown.

Properties:

condition string
The predicate to filter a property by a property value. Property will be removed if it is empty OR expression is evaluated to True.,
Available variables:
Examples:
```
{{ property|string == '' }}
```
```
{{ property is integer }}
```
```
{{ property|length > 5 }}
```
```
{{ property == 'some_string_to_match' }}
```
field_pointers array
Array of paths defining the field to remove. Each item is an array whose field describe the path of a field to remove.
Examples:
```
[
  "tags"
]
```
```
[
  [
    "content",
    "html"
  ],
  [
    "content",
    "plain_text"
  ]
]
```

RequestPath `object`

Specifies where in the request path a component's value should be inserted.

Properties:

RequestOption `object`

Specifies the key field or path and where in the request a component's value should be injected.

Properties:

field_name string
Configures which key should be used in the location that the descriptor is being injected into. We hope to eventually deprecate this field in favor of field_path for all request_options, but must currently maintain it for backwards compatibility in the Builder.
Available variables:
- config
- parameters
Example:
```
segment_id
```
field_path array
Configures a path to be used for nested structures in JSON body requests (e.g. GraphQL queries)
Available variables:
- config
- parameters
Example:
```
[
  "data",
  "viewer",
  "id"
]
```
inject_into
Configures where the descriptor should be set on the HTTP requests. Note that request parameters that are already encoded in the URL path will not be duplicated.
Examples:
```
request_parameter
```
```
header
```
```
body_data
```
```
body_json
```

Schemas `object`

The stream schemas representing the shape of the data emitted by the stream.

LegacySessionTokenAuthenticator `object`

Deprecated - use SessionTokenAuthenticator instead. Authenticator for requests authenticated using session tokens. A session token is a random value generated by a server to identify a specific user for the duration of one interaction session.

Properties:

header string
The name of the session token header that will be injected in the request
Example:
```
X-Session
```
login_url string
Path of the login URL (do not include the base URL)
Example:
```
session
```
session_token string
Session token to use if using a pre-defined token. Not needed if authenticating with username + password pair
session_token_response_key string
Name of the key of the session token to be extracted from the response
Example:
```
id
```
username string
Username used to authenticate and obtain a session token
Example:
```
 {{ config['username'] }}
```
password string
Password used to authenticate and obtain a session token
Examples:
```
{{ config['password'] }}
```
validate_session_url string
Path of the URL to use to validate that the session token is valid (do not include the base URL)
Example:
```
user/current
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

StateDelegatingStream `object`

(This component is experimental. Use at your own risk.) Orchestrate the retriever's usage based on the state value.

Properties:

name string
The stream name.
full_refresh_stream #/definitions/DeclarativeStream
Component used to coordinate how records are extracted across stream slices and request pages when the state is empty or not provided.
incremental_stream #/definitions/DeclarativeStream
Component used to coordinate how records are extracted across stream slices and request pages when the state provided.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

SimpleRetriever `object`

Retrieves records by synchronously sending requests to fetch records. The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the partition router.

Properties:

requester
Requester component that describes how to prepare HTTP requests to send to the source API.
Type:
- #/definitions/HttpRequester
- #/definitions/CustomRequester
decoder
Component decoding the response so records can be extracted.
Type:
record_selector #/definitions/RecordSelector
Component that describes how to extract records from a HTTP response.
paginator
Paginator component that describes how to navigate through the API's pages.
Type:
- #/definitions/DefaultPaginator
- #/definitions/NoPagination
ignore_stream_slicer_parameters_on_paginated_requests boolean
If true, the partition router and incremental request options will be ignored when paginating requests. Request options set directly on the requester will not be ignored.
partition_router
PartitionRouter component that describes how to partition the stream, enabling incremental syncs and checkpointing.
Type:
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

GzipDecoder `object`

Select 'gzip' for response data that is compressed with gzip. Requires specifying an inner data type/decoder to parse the decompressed data.

Properties:

decoder
Type:

CsvDecoder `object`

Select 'CSV' for response data that is formatted as CSV (comma-separated values). Can specify an encoding (default: 'utf-8') and a delimiter (default: ',').

Properties:

encoding string
delimiter string

AsyncJobStatusMap `object`

Matches the api job status to Async Job Status.

Properties:

running array
completed array
failed array
timeout array

AsyncRetriever `object`

Retrieves records by Asynchronously sending requests to fetch records. The retriever acts as an orchestrator between the requester, the record selector, the paginator, and the partition router.

Properties:

record_selector #/definitions/RecordSelector
Component that describes how to extract records from a HTTP response.
status_mapping
Async Job Status to Airbyte CDK Async Job Status mapping.
Type:
- #/definitions/AsyncJobStatusMap
status_extractor
Responsible for fetching the actual status of the async job.
Type:
- #/definitions/DpathExtractor
- #/definitions/CustomRecordExtractor
download_target_extractor
Responsible for fetching the final result urls provided by the completed / finished / ready async job.
Type:
- #/definitions/DpathExtractor
- #/definitions/CustomRecordExtractor
download_extractor
Responsible for fetching the records from provided urls.
Type:
creation_requester
Requester component that describes how to prepare HTTP requests to send to the source API to create the async server-side job.
Type:
- #/definitions/HttpRequester
- #/definitions/CustomRequester
polling_requester
Requester component that describes how to prepare HTTP requests to send to the source API to fetch the status of the running async job.
Type:
- #/definitions/HttpRequester
- #/definitions/CustomRequester
polling_job_timeout
The time in minutes after which the single Async Job should be considered as Timed Out.
Type:
- integer
- string
Available variables:
- config
download_target_requester
Requester component that describes how to prepare HTTP requests to send to the source API to extract the url from polling response by the completed async job.
Type:
- #/definitions/HttpRequester
- #/definitions/CustomRequester
download_requester
Requester component that describes how to prepare HTTP requests to send to the source API to download the data provided by the completed async job.
Type:
- #/definitions/HttpRequester
- #/definitions/CustomRequester
download_paginator
Paginator component that describes how to navigate through the API's pages during download.
Type:
- #/definitions/DefaultPaginator
- #/definitions/NoPagination
abort_requester
Requester component that describes how to prepare HTTP requests to send to the source API to abort a job once it is timed out from the source's perspective.
Type:
- #/definitions/HttpRequester
- #/definitions/CustomRequester
delete_requester
Requester component that describes how to prepare HTTP requests to send to the source API to delete a job once the records are extracted.
Type:
- #/definitions/HttpRequester
- #/definitions/CustomRequester
partition_router
PartitionRouter component that describes how to partition the stream, enabling incremental syncs and checkpointing.
Type:
decoder
Component decoding the response so records can be extracted.
Type:
download_decoder
Component decoding the download response so records can be extracted.
Type:
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

Spec `object`

A source specification made up of connector metadata and how it can be configured.

Properties:

connection_specification object
A connection specification describing how a the connector can be configured.
documentation_url string
URL of the connector's documentation page.
Example:
```
https://docs.airbyte.com/integrations/sources/dremio
```
advanced_auth #/definitions/AuthFlow
Advanced specification for configuring the authentication flow.
config_normalization_rules object

ConfigMigration `object`

A config migration that will be applied on the incoming config at the start of a sync.

Properties:

description string
The description/purpose of the config migration.
transformations array
The list of transformations that will attempt to be applied on an incoming unmigrated config. The transformations will be applied in the order they are defined.

SubstreamPartitionRouter `object`

Partition router that is used to retrieve records that have been partitioned according to records from the specified parent streams. An example of a parent stream is automobile brands and the substream would be the various car models associated with each branch.

Properties:

parent_stream_configs array #/definitions/ParentStreamConfig
Specifies which parent streams are being iterated over and how parent records should be used to partition the child stream data set.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

ValueType `string`

A schema type.

WaitTimeFromHeader `object`

Extract wait time from a HTTP header in the response.

Properties:

header string
The name of the response header defining how long to wait before retrying.
Available variables:
- config
Example:
```
Retry-After
```
regex string
Optional regex to apply on the header to extract its value. The regex should define a capture group defining the wait time.
Example:
```
([-+]?\d+)
```
max_waiting_time_in_seconds number
Given the value extracted from the header is greater than this value, stop the stream.
Example:
```
3600
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

GroupingPartitionRouter `object`

A decorator on top of a partition router that groups partitions into batches of a specified size. This is useful for APIs that support filtering by multiple partition keys in a single request. Note that per-partition incremental syncs may not work as expected because the grouping of partitions might change between syncs, potentially leading to inconsistent state tracking.

Properties:

group_size integer
The number of partitions to include in each group. This determines how many partition values are batched together in a single slice.
Examples:
```
10
```
```
50
```
underlying_partition_router
The partition router whose output will be grouped. This can be any valid partition router component.
Type:
deduplicate boolean
If true, ensures that partitions are unique within each group by removing duplicates based on the partition key.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

WaitUntilTimeFromHeader `object`

Extract time at which we can retry the request from response header and wait for the difference between now and that time.

Properties:

header string
The name of the response header defining how long to wait before retrying.
Available variables:
- config
Example:
```
wait_time
```
min_wait
Minimum time to wait before retrying.
Type:
- number
- string
Available variables:
- config
Examples:
```
10
```
```
60
```
regex string
Optional regex to apply on the header to extract its value. The regex should define a capture group defining the wait time.
Available variables:
- config
Example:
```
([-+]?\d+)
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

ComponentMappingDefinition `object`

(This component is experimental. Use at your own risk.) Specifies a mapping definition to update or add fields in a record or configuration. This allows dynamic mapping of data by interpolating values into the template based on provided contexts.

Properties:

field_path array
A list of potentially nested fields indicating the full path where value will be added or updated.
Available variables:
Examples:
```
[
  "data"
]
```
```
[
  "data",
  "records"
]
```
```
[
  "data",
  1,
  "name"
]
```
```
[
  "data",
  "{{ components_values.name }}"
]
```
```
[
  "data",
  "*",
  "record"
]
```
```
[
  "*",
  "**",
  "name"
]
```
value string
The dynamic or static value to assign to the key. Interpolated values can be used to dynamically determine the value during runtime.
Available variables:
Examples:
```
{{ components_values['updates'] }}
```
```
{{ components_values['MetaData']['LastUpdatedTime'] }}
```
```
{{ config['segment_id'] }}
```
```
{{ stream_slice['parent_id'] }}
```
```
{{ stream_slice['extra_fields']['name'] }}
```
value_type #/definitions/ValueType
The expected data type of the value. If omitted, the type will be inferred from the value provided.
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

HttpComponentsResolver `object`

(This component is experimental. Use at your own risk.) Component resolve and populates stream templates with components fetched via an HTTP retriever.

Properties:

retriever
Component used to coordinate how records are extracted across stream slices and request pages.
Type:
components_mapping array #/definitions/ComponentMappingDefinition
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

StreamConfig `object`

(This component is experimental. Use at your own risk.) Describes how to get streams config from the source config.

Properties:

configs_pointer array
A list of potentially nested fields indicating the full path in source config file where streams configs located.
Available variables:
- parameters
Examples:
```
[
  "data"
]
```
```
[
  "data",
  "streams"
]
```
```
[
  "data",
  "{{ parameters.name }}"
]
```
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

ConfigComponentsResolver `object`

(This component is experimental. Use at your own risk.) Resolves and populates stream templates with components fetched from the source config.

Properties:

stream_config #/definitions/StreamConfig
components_mapping array #/definitions/ComponentMappingDefinition
$parameters object
Set parameters that are inherited to all children. See the section in the advanced topics for more details.

DynamicDeclarativeStream `object`

(This component is experimental. Use at your own risk.) A component that described how will be created declarative streams based on stream template.

Properties:

name string
The dynamic stream name.
stream_template
Reference to the stream template.
Type:
- #/definitions/DeclarativeStream
- #/definitions/StateDelegatingStream
components_resolver
Component resolve and populates stream templates with components values.
Type:
- #/definitions/HttpComponentsResolver
- #/definitions/ConfigComponentsResolver

RequestBodyPlainText `object`

Request body value is sent as plain text

Properties:

value string

RequestBodyUrlEncodedForm `object`

Request body value is converted into a url-encoded form

Properties:

value object

RequestBodyJsonObject `object`

Request body value converted into a JSON object

Properties:

value object

RequestBodyGraphQL `object`

Request body value converted into a GraphQL query object

Properties:

value #/definitions/RequestBodyGraphQlQuery

RequestBodyGraphQlQuery `object`

Request body GraphQL query object

Properties:

query object
The GraphQL query to be executed

DpathValidator `object`

Validator that extracts the value located at a given field path.

Properties:

field_path array
List of potentially nested fields describing the full path of the field to validate. Use "*" to validate all values from an array.
Available variables:
- config
Examples:
```
[
  "data"
]
```
```
[
  "data",
  "records"
]
```
```
[
  "data",
  "{{ parameters.name }}"
]
```
```
[
  "data",
  "*",
  "record"
]
```
validation_strategy
The condition that the specified config value will be evaluated against
Type:
- #/definitions/ValidateAdheresToSchema

PredicateValidator `object`

Validator that applies a validation strategy to a specified value.

Properties:

value stringnumberobjectarraybooleannull
The value to be validated. Can be a literal value or interpolated from configuration.
Available variables:
- config
Examples:
```
test-value
```
```
{{ config['api_version'] }}
```
```
{{ config['tenant_id'] }}
```
```
123
```
validation_strategy
The validation strategy to apply to the value.
Type:
- #/definitions/ValidateAdheresToSchema

ValidateAdheresToSchema `object`

Validates that a user-provided schema adheres to a specified JSON schema.

Properties:

base_schema `stringobject`

The base JSON schema against which the user-provided schema will be validated.

Available variables:

config

Examples:

{{ config['report_validation_schema'] }}

'{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Person",
  "type": "object",
  "properties": {
    "name": {
      "type": "string",
      "description": "The person's name"
    },
    "age": {
      "type": "integer",
      "minimum": 0,
      "description": "The person's age"
    }
  },
  "required": ["name", "age"]
}'

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Person",
  "type": "object",
  "properties": {
    "name": {
      "type": "string",
      "description": "The person's name"
    },
    "age": {
      "type": "integer",
      "minimum": 0,
      "description": "The person's age"
    }
  },
  "required": [
    "name",
    "age"
  ]
}

ConfigRemapField `object`

Transformation that remaps a field's value to another value based on a static map.

Properties:

map objectstring
A mapping of original values to new values. When a field value matches a key in this map, it will be replaced with the corresponding value.
Available variables:
- config
Examples:
```
{
  "pending": "in_progress",
  "done": "completed",
  "cancelled": "terminated"
}
```
```
{{ config['status_mapping'] }}
```
field_path array
The path to the field whose value should be remapped. Specified as a list of path components to navigate through nested objects.
Available variables:
- config
Examples:
```
[
  "status"
]
```
```
[
  "data",
  "status"
]
```
```
[
  "data",
  "{{ config.name }}",
  "status"
]
```
```
[
  "data",
  "*",
  "status"
]
```

ConfigAddFields `object`

Transformation that adds fields to a config. The path of the added field can be nested.

Properties:

fields array #/definitions/AddedFieldDefinition
A list of transformations (path and corresponding value) that will be added to the config.

condition `string`

Fields will be added if expression is evaluated to True.

Available variables:

config
property

Examples:

{{ config['environemnt'] == 'sandbox' }}

{{ property is integer }}

{{ property|length > 5 }}

{{ property == 'some_string_to_match' }}

ConfigRemoveFields `object`

Transformation that removes a field from the config.

Properties:

field_pointers `array`

A list of field pointers to be removed from the config.

Examples:

[
  "tags"
]

[
  [
    "content",
    "html"
  ],
  [
    "content",
    "plain_text"
  ]
]

condition `string`

Fields will be removed if expression is evaluated to True.

Available variables:

config
property

Examples:

{{ config['environemnt'] == 'sandbox' }}

{{ property is integer }}

{{ property|length > 5 }}

{{ property == 'some_string_to_match' }}

Interpolation variables

All string properties that list out available variables allow jinja expressions. They can be used by placing them in double curly braces: {{ config.property }}. The following variables are available

config `object`

The connector configuration. The object's keys are the same as the the keys defined in the connection specification.

Example:

{
  "start_date": "2010-01-01",
  "api_key": "*****"
}

parameters `object`

Additional runtime parameters, to be used for string interpolation. Parameters can be passed down from a parent component to its subcomponents using the $parameters key. This can be used to avoid repetitions.

Example:

{
  "path": "automations",
  "data_export_path": "automations",
  "cursor_field": "updated_at"
}

headers `object`

The HTTP headers from the last response received from the API. The object's keys are the header names from the response.

Example:

{
  "Server": "nginx",
  "Date": "Mon, 24 Apr 2023 20:17:21 GMT",
  "Content-Type": "application/json",
  "Content-Length": "420",
  "Connection": "keep-alive",
  "referrer-policy": "strict-origin-when-cross-origin",
  "x-content-type-options": "nosniff",
  "x-ratelimit-limit": "600",
  "x-ratelimit-remaining": "598",
  "x-ratelimit-reset": "39"
}

last_record `object`

Last record extracted from the response received from the API.

Example:

{
  "name": "Test List: 19",
  "id": "0236d6d2",
  "contact_count": 20,
  "_metadata": {
    "self": "https://api.sendgrid.com/v3/marketing/lists/0236d6d2"
  }
}

last_page_size `object`

Number of records extracted from the last response received from the API.

Example:

next_page_token `object`

Object describing the token to fetch the next page of records. The object has a single key "next_page_token".

Examples:

{
  "next_page_token": 3
}

{
  "next_page_token": "https://api.sendgrid.com/v3/marketing/lists/0236d6d2-75d2-42c5-962d-603e0deaf8d1"
}

record `object`

The record being processed. The object's keys are the same keys as the records produced by the RecordSelector.

response `object`

The body of the last response received from the API. The object's keys are the same keys as the response body's.

Example:

{
  "result": [
    {
      "name": "Test List: 19",
      "id": "0236d6d2-75d2-42c5-962d-603e0deaf8d1",
      "contact_count": 20,
      "_metadata": {
        "self": "https://api.sendgrid.com/v3/marketing/lists/0236d6d2"
      }
    }
  ],
  "_metadata": {
    "self": "https://api.sendgrid.com/v3/marketing/lists?page_size=1&page_token=",
    "next": "https://api.sendgrid.com/v3/marketing/lists?page_size=1&page_token=0236d6d2",
    "count": 82
  }
}

creation_response `object`

The response received from the creation_requester in the AsyncRetriever component.

Example:

{
  "id": "1234"
}

polling_response `object`

The response received from the polling_requester in the AsyncRetriever component.

Example:

{
  "id": "1234"
}

download_target `string`

The URL received from the polling_requester in the AsyncRetriever with jobStatus as COMPLETED.

Example:

https://api.sendgrid.com/v3/marketing/lists?page_size=1&page_token=0236d6d2&filename=xxx_yyy_zzz.csv

stream_interval `object`

The current stream interval being processed. The keys are defined by the incremental sync component. Default keys are start_time and end_time.

Example:

{
  "start_time": "2020-01-01 00:00:00.000+00:00",
  "end_time": "2020-01-02 00:00:00.000+00:00"
}

stream_partition `object`

The current stream partition being processed. The keys are defined by the partition router component.

Examples:

{
  "survey_id": 1234
}

{
  "strategy": "DESKTOP"
}

{
  "survey_id": 1234,
  "strategy": "MOBILE"
}

stream_slice `object`

This variable is deprecated. Use stream_interval or stream_partition instead.

Interpolation macros

Besides referencing variables, the following macros can be called as part of jinja expressions, for example like this: {{ now_utc() }}.

now_utc

Returns the current date and time in the UTC timezone.

Examples:

'{{ now_utc() }}' -> '2021-09-01 00:00:00+00:00'

'{{ now_utc().strftime('%Y-%m-%d') }}' -> '2021-09-01'

today_utc

Returns the current date in UTC timezone. The output is a date object.

Examples:

'{{ today_utc() }}' -> '2021-09-01'

'{{ today_utc().strftime('%Y/%m/%d')}}' -> '2021/09/01'

timestamp

Converts a number or a string representing a datetime (formatted as ISO8601) to a timestamp. If the input is a number, it is converted to an int. If no timezone is specified, the string is interpreted as UTC.

Arguments:

datetime: A string formatted as ISO8601 or an integer representing a unix timestamp

Examples:

'{{ timestamp(1646006400) }}' -> 1646006400

'{{ timestamp('2022-02-28') }}' -> 1646006400

'{{ timestamp('2022-02-28T00:00:00Z') }}' -> 1646006400

'{{ timestamp('2022-02-28 00:00:00Z') }}' -> 1646006400

'{{ timestamp('2022-02-28T00:00:00-08:00') }}' -> 1646035200

max

Returns the largest object of a iterable, or or two or more arguments.

Arguments:

args: iterable or a sequence of two or more arguments

Examples:

'{{ max(2, 3) }}' -> 3

'{{ max([2, 3]) }}' -> 3

day_delta

Returns the datetime of now() + num_days.

Arguments:

num_days: The number of days to add to now
format: How to format the output string

Examples:

'{{ day_delta(1) }}' -> '2021-09-02T00:00:00.000000+0000'

'{{ day_delta(-1) }}' -> '2021-08-31:00:00.000000+0000'

'{{ day_delta(25, format='%Y-%m-%d') }}' -> '2021-09-02'

duration

Converts an ISO8601 duration to datetime timedelta.

Arguments:

duration_string: A string representing an ISO8601 duration. See https://www.digi.com/resources/documentation/digidocs//90001488-13/reference/r_iso_8601_duration_format.htm for more details.

Examples:

'{{ duration('P1D') }}' -> '1 day, 0:00:00'

'{{ duration('P6DT23H') }}' -> '6 days, 23:00:00'

'{{ (now_utc() - duration('P1D')).strftime('%Y-%m-%dT%H:%M:%SZ') }}' -> '2021-08-31T00:00:00Z'

format_datetime

Converts a datetime or a datetime-string to the specified format.

Arguments:

datetime: The datetime object or a string to convert. If datetime is a string, it must be formatted as ISO8601.
format: The datetime format.
input_format: (optional) The datetime format in the case it is an string.

Examples:

{{ format_datetime(config['start_time'], '%Y-%m-%d') }}

{{ format_datetime(config['start_date'], '%Y-%m-%dT%H:%M:%S.%fZ') }}

{{ format_datetime(config['start_date'], '%Y-%m-%dT%H:%M:%S.%fZ', '%a, %d %b %Y %H:%M:%S %z') }}

str_to_datetime

Converts a string to a datetime object with UTC timezone.

Arguments:

s: The string to convert.

Examples:

{{ str_to_datetime('2022-01-14') }}

{{ str_to_datetime('2022-01-01 13:45:30') }}

{{ str_to_datetime('2022-01-01T13:45:30+00:00') }}

{{ str_to_datetime('2022-01-01T13:45:30.123456Z') }}

Interpolation filters

The following filters can be called as part of jinja expressions, for example like this: {{ 1 | string }}.

hash

Convert the specified value to a hashed string.

Arguments:

hash_type: Valid hash type for converts ('md5' as default value).
salt: An additional value to further protect sensitive data.

Examples:

{{ 'Test client_secret' | hash() }} -> '3032d57a12f76b61a820e47b9a5a0cbb'

{{ 'Test client_secret' | hash('md5') }} -> '3032d57a12f76b61a820e47b9a5a0cbb'

{{ 'Test client_secret' | hash('md5', salt='salt') }} -> '5011a0168579c2d94cbbe1c6ad14327c'

base64encode

Convert the specified value to a string in the base64 format.

Example:

{{ 'Test client_secret' | base64encode }} -> 'VGVzdCBjbGllbnRfc2VjcmV0'

base64decode

Decodes the specified base64 format value into a common string.

Example:

{{ 'ZmFrZSByZWZyZXNoX3Rva2VuIHZhbHVl' | base64decode }} -> 'fake refresh_token value'

string

Converts the specified value to a string.

Examples:

{{ 1 | string }} -> "1"

{{ ["hello", "world" | string }} -> "["hello", "world"]"

regex_search

Match the input string against a regular expression and return the first match.

Arguments:

regex: The regular expression to search for. It must include a capture group.

Example:

{{ "goodbye, cruel world" | regex_search("goodbye,\s(.*)$") }} -> "cruel world"

Components​

DeclarativeSource object​

check

streams array

dynamic_streams array #/definitions/DynamicDeclarativeStream

version string

schemas #/definitions/Schemas

spec #/definitions/Spec

concurrency_level #/definitions/ConcurrencyLevel

api_budget #/definitions/HTTPAPIBudget

max_concurrent_async_job_count integerstring

metadata object

description string

AddedFieldDefinition object​

path array

value string

value_type #/definitions/ValueType

$parameters object

AddFields object​

fields array #/definitions/AddedFieldDefinition

condition string

$parameters object

ApiKeyAuthenticator object​

api_token string

header string

inject_into #/definitions/RequestOption

$parameters object

AuthFlow object​

auth_flow_type string

predicate_key array

predicate_value string

oauth_config_specification #/definitions/OAuthConfigSpecification

BasicHttpAuthenticator object​

username string

password string

$parameters object

BearerAuthenticator object​

api_token string

$parameters object

SelectiveAuthenticator object​

authenticator_selection_path array

authenticators object

$parameters object

CheckStream object​

stream_names array

dynamic_streams_check_configs array #/definitions/DynamicStreamCheckConfig

DynamicStreamCheckConfig object​

dynamic_stream_name string

stream_count integer

CheckDynamicStream object​

stream_count integer

use_check_availability boolean

CompositeErrorHandler object​

error_handlers array

$parameters object

ConcurrencyLevel object​

default_concurrency

max_concurrency integer

$parameters object

ConstantBackoffStrategy object​

backoff_time_in_seconds

$parameters object

CursorPagination object​

cursor_value string

page_size integer

stop_condition string

$parameters object

CustomAuthenticator object​

class_name string

$parameters object

CustomBackoffStrategy object​

class_name string

$parameters object

CustomErrorHandler object​

class_name string

$parameters object

CustomIncrementalSync object​

class_name string

cursor_field string

$parameters object

Components

DeclarativeSource `object`

streams `array`

dynamic_streams `array` `#/definitions/DynamicDeclarativeStream`

version `string`

schemas `#/definitions/Schemas`

spec `#/definitions/Spec`

concurrency_level `#/definitions/ConcurrencyLevel`

api_budget `#/definitions/HTTPAPIBudget`

max_concurrent_async_job_count `integerstring`

metadata `object`

description `string`

AddedFieldDefinition `object`

path `array`

value `string`

value_type `#/definitions/ValueType`

$parameters `object`

AddFields `object`

fields `array` `#/definitions/AddedFieldDefinition`

condition `string`

$parameters `object`

ApiKeyAuthenticator `object`

api_token `string`

header `string`

inject_into `#/definitions/RequestOption`

$parameters `object`

AuthFlow `object`

auth_flow_type `string`

predicate_key `array`

predicate_value `string`

oauth_config_specification `#/definitions/OAuthConfigSpecification`

BasicHttpAuthenticator `object`

username `string`

password `string`

$parameters `object`

BearerAuthenticator `object`

api_token `string`

$parameters `object`

SelectiveAuthenticator `object`

authenticator_selection_path `array`

authenticators `object`

$parameters `object`

CheckStream `object`

stream_names `array`

dynamic_streams_check_configs `array` `#/definitions/DynamicStreamCheckConfig`

DynamicStreamCheckConfig `object`

dynamic_stream_name `string`

stream_count `integer`

CheckDynamicStream `object`

stream_count `integer`

use_check_availability `boolean`

CompositeErrorHandler `object`

error_handlers `array`

$parameters `object`

ConcurrencyLevel `object`

max_concurrency `integer`

$parameters `object`

ConstantBackoffStrategy `object`

$parameters `object`

CursorPagination `object`

cursor_value `string`

page_size `integer`

stop_condition `string`

$parameters `object`

CustomAuthenticator `object`

class_name `string`

$parameters `object`

CustomBackoffStrategy `object`

class_name `string`

$parameters `object`

CustomErrorHandler `object`

class_name `string`

$parameters `object`

CustomIncrementalSync `object`

class_name `string`

cursor_field `string`

$parameters `object`

CustomPaginationStrategy `object`

class_name `string`

$parameters `object`