Skip to main content

Datasource

View as Markdown

A Lakewatch datasource: a pipeline that ingests security data and writes it to Unity Catalog tables.

What a datasource ingests is defined by its ingestion method (autoloader, unity_catalog_connection, or existing_table), which sets how the data arrives, together with its parsers, which define the schemas that the arriving data is matched against and written as.

Datasource object​

A Lakewatch datasource: a pipeline that ingests security data and writes it to Unity Catalog tables.

What a datasource ingests is defined by its ingestion method (autoloader, unity_catalog_connection, or existing_table), which sets how the data arrives, together with its parsers, which define the schemas that the arriving data is matched against and written as.

namestringBetaIDImmutable

Resource name of the datasource, in the format datasources/{datasource_id}. Unlike the datasource's Unity Catalog location (catalog_name, schema_name, and datasource_name), the name does not change when the datasource or its catalog or schema is renamed in Unity Catalog.

Example: datasources/550e8400-e29b-41d4-a716-446655440000

datasource_idstringBetaOutput only

Unity Catalog securable ID (a UUID) of the datasource. It never changes, including when the datasource is renamed in Unity Catalog, and is the final segment of name.

catalog_namestringBetaImmutable

Name of the Unity Catalog catalog that contains the datasource. Required on create and cannot be changed afterward. Reads return the value as supplied; the catalog's current name is reported in effective_catalog_name.

schema_namestringBetaImmutable

Name of the Unity Catalog schema that contains the datasource. Required on create and cannot be changed afterward. Reads return the value as supplied; the schema's current name is reported in effective_schema_name.

effective_catalog_namestringBetaOutput only

Current name of the Unity Catalog catalog that contains the datasource, which reflects catalog renames without a change to catalog_name. Populated on get, create, and update responses; list responses omit it.

effective_schema_namestringBetaOutput only

Current name of the Unity Catalog schema that contains the datasource, which reflects schema renames without a change to schema_name. Populated on get, create, and update responses; list responses omit it.

datasource_namestringBetaImmutable

Name of the datasource in Unity Catalog, unique within its schema. Required on create and cannot be changed afterward. Reads return the value as supplied; the datasource's current Unity Catalog name, which reflects renames, is reported in effective_datasource_name.

effective_datasource_namestringBetaOutput only

Current UC leaf name of the datasource (server-owned). Tracks UC renames so callers can observe the live name without having to update the client-owned datasource_name; unset until resolved.

descriptionstringBeta

Free-form description of the datasource.

versionint64BetaOutput only

Version number of the datasource, incremented on each update. Informational only; use etag for optimistic concurrency control.

create_timestringBetaOutput only
update_timestringBetaOutput only
created_byobjectBetaOutput only
Show child attributesHide child attributes
idstringBeta

User / service-principal ID.

principal_typestringBeta

Whether this principal is a regular user, service principal, or system principal.

Values:

  • REGULAR_USER
  • SERVICE_PRINCIPAL
  • SYSTEM_USER
emailstringBeta

Email (regular users only).

display_namestringBeta

Display name (user or SP name).

updated_byobjectBetaOutput only
Show child attributesHide child attributes
idstringBeta

User / service-principal ID.

principal_typestringBeta

Whether this principal is a regular user, service principal, or system principal.

Values:

  • REGULAR_USER
  • SERVICE_PRINCIPAL
  • SYSTEM_USER
emailstringBeta

Email (regular users only).

display_namestringBeta

Display name (user or SP name).

etagstringBetaOutput only

Opaque entity tag for optimistic concurrency control. Returned by get, create, and update responses and by list responses in the DATASOURCE_VIEW_FULL view. To make an update conditional on the datasource not having changed since it was read, pass this value as the update request's etag. Changes to status do not change the etag.

annotationsobjectBeta

Free-form key-value annotations on the datasource, managed by the client.

ownerobjectBetaOutput only

Unity Catalog owner of the datasource, assigned by the server. Create responses omit it.

Show child attributesHide child attributes
idstringBeta

User / service-principal ID.

principal_typestringBeta

Whether this principal is a regular user, service principal, or system principal.

Values:

  • REGULAR_USER
  • SERVICE_PRINCIPAL
  • SYSTEM_USER
emailstringBeta

Email (regular users only).

display_namestringBeta

Display name (user or SP name).

run_as_principalstringBeta

The user or service principal that the datasource runs as, given as a user email or service principal application ID. When it is supplied on create or update, the requester must be authorized to run the datasource as that principal, or the request is rejected. When it is omitted on create, the datasource runs as the requester. On update, a mask that includes run_as_principal without a value clears it, so the datasource runs as the requester; a mask that omits it keeps the current identity. Reads return the value as supplied; the identity the datasource runs as is reported in effective_run_as_principal.

effective_run_as_principalstringBetaOutput only

The identity the datasource runs as: the resolved run_as_principal, or the requester's identity if run_as_principal was omitted on create or cleared on update. Given as a user email or service principal application ID, or as the numeric principal ID if the identity can no longer be resolved. Populated on get, create, and update responses; list responses omit it.

autoloaderobjectBeta
Show child attributesHide child attributes
formatstringBeta

File format of the source files. Required.

Values:

  • AUTOLOADER_FORMAT_JSON
  • AUTOLOADER_FORMAT_JSONL
  • AUTOLOADER_FORMAT_CSV
  • AUTOLOADER_FORMAT_PARQUET
  • AUTOLOADER_FORMAT_XML
locationstringBeta

URI of the source files: a path in a Unity Catalog volume or external location, or a cloud storage URL such as an Amazon S3 path. Required, including when new files are discovered through queue_url. Cannot be changed after creation.

format_optionsobjectBeta

Format-specific parsing options, passed to Auto Loader. Supported for the CSV, JSON, JSONL, and XML formats, each of which accepts a fixed set of options. The XML format requires the rowTag option.

service_credentialsstringBeta

Name of the Unity Catalog service credential to use.

aws_secret_scopestringBeta

Name of the workspace secret scope that holds the AWS access key ID and secret access key, under the keys awsAccessKeyId and awsSecretAccessKey.

override_partition_columnsarray of stringBeta

Hive-style partition columns to derive from the source file paths. If unset, no partition columns are derived.

raw_tableobjectBeta

Table that stores the raw data before parsers schematize it. Every Auto Loader datasource writes its raw rows to this table before any parser reads them. The table must be in the datasource's catalog and schema; an omitted catalog_name or schema_name defaults to them. Cannot be changed after creation. Reads return the value as supplied; the table the data is written to is reported in effective_raw_table.

Show child attributesHide child attributes
catalog_namestringBeta

Name of the parent Unity Catalog catalog.

schema_namestringBeta

Name of the parent Unity Catalog schema.

table_namestringBeta

Unity Catalog table name.

envelopeobjectBeta

How to unwrap a transport envelope around the source records before parsing. Supported only for the JSON and JSONL formats. If unset, each source record is treated as one event.

Show child attributesHide child attributes
json_arrayobjectBeta
Show child attributesHide child attributes
pathstringBeta

Name of the root-level key that holds the array of events, for example Records. Nested paths are not supported.

cloud_watch_logsobjectBeta
Show child attributesHide child attributes
header_retentionbooleanBeta

Whether to keep the envelope's owner, logGroup, and logStream values in the data.lw_header field of each unwrapped row. Must be set explicitly to true or false.

queue_urlstringBeta

URL of a customer-managed Amazon SQS queue that receives file-arrival notifications. Set it to discover new files through the queue. If unset, new files are discovered through the file events of the source's Unity Catalog external location when it has them configured, and by directory listing otherwise. Cannot be changed after creation.

effective_raw_tableobjectBetaOutput only

The table that the raw data is written to: raw_table in the datasource's catalog and schema, under their current Unity Catalog names. Populated on get, create, and update responses.

Show child attributesHide child attributes
catalog_namestringBeta

Name of the parent Unity Catalog catalog.

schema_namestringBeta

Name of the parent Unity Catalog schema.

table_namestringBeta

Unity Catalog table name.

include_existing_filesbooleanBeta

Whether to ingest the files already present at location when ingestion first starts. Set true to ingest the existing files and then every new file. Set false, or leave unset, to ingest only files that arrive after the datasource starts. Defaults to false. Cannot be changed after creation. Reads return the value as supplied; the setting in effect is reported in effective_include_existing_files.

effective_include_existing_filesbooleanBetaOutput only

Whether ingestion started with the files already present at location. Populated on get, create, and update responses.

unity_catalog_connectionobjectBeta
Show child attributesHide child attributes
connection_namestringBeta

Name of the Unity Catalog connection to ingest through.

connectorstringBetaOutput only

Connector type of the connection, such as SLACK_AUDIT_LOGS. Resolved by the server from connection_name through Unity Catalog; not accepted as input.

configurationobjectBeta

Free-form key-value configuration for the managed ingestion pipeline.

start_timestringBeta

Earliest source event time to ingest. Leave unset to start at creation time, with no history backfilled. Rejected if in the future, beyond the connector's history window, or for a connector without start-time support. Takes effect only at first start and cannot be changed. Reads return the value as supplied; the start time ingestion uses is reported in effective_start_time.

effective_start_timestringBetaOutput only

The start time ingestion uses: the supplied start_time, or the creation time when it was omitted. Always populated.

existing_tableobjectBeta
Show child attributesHide child attributes
namestringBeta

Name of the source table or view. Required on create and cannot be changed. Reads return the value as supplied; the table's current name is reported in effective_name.

catalogstringBeta

Name of the catalog the existing table lives in. Optional on create and defaults to the datasource's own catalog. Cannot be changed. Reads return the value as supplied; the catalog in use is reported in effective_catalog.

schemastringBeta

Name of the schema the existing table lives in. Optional, but required when catalog is set; supplying catalog without schema is rejected. Defaults to the datasource's own schema. Cannot be changed. Reads return the value as supplied; the schema in use is reported in effective_schema.

include_existing_rowsbooleanBeta

Whether to ingest the rows already present in the source table when ingestion first starts. Set true to ingest the existing rows and then every new commit. Set false, or leave unset, to ingest only rows written after the datasource starts. Defaults to false, and must be true for a Delta Sharing source table. Takes effect only at first start and cannot be changed. Reads return the value as supplied.

effective_catalogstringBetaOutput only

Current Unity Catalog catalog name of the source table, resolved from its stable table ID, so a rename of the source table is reported here without changing catalog.

effective_schemastringBetaOutput only

Current Unity Catalog schema name of the source table, resolved from its stable table ID.

effective_namestringBetaOutput only

Current Unity Catalog name of the source table, resolved from its stable table ID.

parsersarray of objectBeta

Parsers that schematize the ingested data. Each record is matched against the parsers in order and written to the parsed table of the first parser that it matches. Existing-table and Unity Catalog connection datasources require at least one parser; an Auto Loader datasource with no parsers writes only its raw table.

Show child attributesHide child attributes
idstringBeta

Identifier of the parser. Stable across renames, unlike a parser name.

versionstringBeta

Version of the parser to use, given as an explicit version number.

target_tableobjectBeta

Table that this parser's data is written to. The table must be in the datasource's catalog and schema. If unset, the table name is derived from the parser name. Reads return the value as supplied; the table the data is written to is reported in effective_target_table.

Show child attributesHide child attributes
catalog_namestringBeta

Name of the parent Unity Catalog catalog.

schema_namestringBeta

Name of the parent Unity Catalog schema.

table_namestringBeta

Unity Catalog table name.

effective_target_tableobjectBetaOutput only

The table this parser's data is written to: target_table when supplied, otherwise the table derived from the datasource's catalog and schema and the parser name. Fixed once resolved, so a later parser rename does not move it. Populated on get, create, and update responses with the resolved table.

Show child attributesHide child attributes
catalog_namestringBeta

Name of the parent Unity Catalog catalog.

schema_namestringBeta

Name of the parent Unity Catalog schema.

table_namestringBeta

Unity Catalog table name.

scheduleobjectBeta

Schedule on which the datasource's ingestion job runs. A datasource cannot be paused, so a request that sets schedule.enabled to false is rejected. Continuous schedules are not supported for Unity Catalog connection datasources.

Show child attributesHide child attributes
at_least_everystringBeta

Duration string defining the minimum cadence. May contain components for days, hours, minutes, seconds (e.g. 5d, 5d3h, 3d2h5m12s, 12m16s).

cron_expressionstringBeta

Quartz cron expression.

continuousbooleanBeta

Run continuously (sentinel — must be true when set in this branch).

compute_groupstringBeta

dedicated, inherited, automatic, or a custom group name. Reads return the value as supplied. A rule whose schedule omits it runs on dedicated.

enabledbooleanBeta

Whether the job is scheduled to run. Defaults to true.

compute_modestringBeta

Compute mode for the datasource's ingestion job. If unset, the default_compute_mode of the catalog's datasource configuration applies, or DS_COMPUTE_PERFORMANCE_OPTIMIZED if the catalog has none.

Values:

  • DS_COMPUTE_STANDARD
  • DS_COMPUTE_PERFORMANCE_OPTIMIZED
statusobjectBetaOutput only

Runtime status of the jobs and pipelines that run this datasource.

Show child attributesHide child attributes
ingestion_jobobjectBetaOutput only

Status of the datasource's ingestion job.

Show child attributesHide child attributes
job_idstringBetaOutput only

Databricks Jobs API job ID. Unset for a pure ingestion pipeline.

lifecycle_statestringBetaOutput only

Scheduling lifecycle of the resource's underlying job/pipeline: whether it is unscheduled, scheduled, stopped, or deleted. Distinct from run_state, which reports the outcome of the most recent run.

Values:

  • UNSCHEDULED
  • SCHEDULED
  • STOPPED
  • DELETED
run_statestringBetaOutput only

Outcome / progress of the most recent (or in-flight) run.

Values:

  • RUNNING
  • SUCCEEDED
  • FAILED
  • CANCELED
last_error_messagestringBetaOutput only

Most recent error message, if any. Pairs with run_state = FAILED.

create_timestringBetaOutput only
last_start_timestringBetaOutput only

When the most recent run started. Pairs with run_state = RUNNING.

last_complete_timestringBetaOutput only

When the most recent run completed. Pairs with run_state = SUCCEEDED.

last_status_change_timestringBetaOutput only
pipeline_idstringBetaOutput only

DLT ingestion-pipeline ID. Set only when this status represents the datasource's ingestion pipeline (unset for job statuses).

ingestion_pipelineobjectBetaOutput only

Status of the managed ingestion pipeline. Populated only when unity_catalog_connection is set, with pipeline_id identifying the pipeline.

Show child attributesHide child attributes
job_idstringBetaOutput only

Databricks Jobs API job ID. Unset for a pure ingestion pipeline.

lifecycle_statestringBetaOutput only

Scheduling lifecycle of the resource's underlying job/pipeline: whether it is unscheduled, scheduled, stopped, or deleted. Distinct from run_state, which reports the outcome of the most recent run.

Values:

  • UNSCHEDULED
  • SCHEDULED
  • STOPPED
  • DELETED
run_statestringBetaOutput only

Outcome / progress of the most recent (or in-flight) run.

Values:

  • RUNNING
  • SUCCEEDED
  • FAILED
  • CANCELED
last_error_messagestringBetaOutput only

Most recent error message, if any. Pairs with run_state = FAILED.

create_timestringBetaOutput only
last_start_timestringBetaOutput only

When the most recent run started. Pairs with run_state = RUNNING.

last_complete_timestringBetaOutput only

When the most recent run completed. Pairs with run_state = SUCCEEDED.

last_status_change_timestringBetaOutput only
pipeline_idstringBetaOutput only

DLT ingestion-pipeline ID. Set only when this status represents the datasource's ingestion pipeline (unset for job statuses).

warehouse_idstringBetaInput only

ID of the SQL warehouse that runs the checks Lakewatch performs to validate a create or update request, such as source and destination access checks. Required and non-blank on every create and update request; there is no default warehouse. The requester must be able to use the warehouse. Used only for those checks: it is not stored, is not returned on reads, and does not affect the compute that runs the ingestion job.

compute_workspace_idstringBetaOutput only

ID of the workspace whose compute runs this datasource's ingestion job and owns its associated resources, such as the storage the ingested data lands in. Server-assigned.

Get a datasource Beta​

GET /api/2.0/lakewatch/{name=datasources/*}

Returns a single datasource by its resource name.

API scopes: lakewatch

Parameters​

namestringRequiredpath

Response​

Returns the Datasource object.

List datasources Beta​

GET /api/2.0/lakewatch/datasources

Lists the datasources in a Unity Catalog catalog, optionally narrowed to one schema or filtered by name. Results are paginated. Datasources that are still being created or deleted are not listed.

API scopes: lakewatch

Parameters​

page_sizeint32query

Maximum number of results to return in one page. This is a best-effort upper bound, not a guarantee: the server may return fewer items than requested (even zero) while more results still exist. If unset or 0, the server picks a default.

page_tokenstringquery
filterstringquery

Filter expression that restricts the results to datasources with a given name, for example datasource_name = "my_source". Only a single exact-match comparison on datasource_name is supported, and it matches the datasource's current Unity Catalog name, which reads report in effective_datasource_name. Filtered results are returned in a single page.

catalog_namestringRequiredquery

Name of the Unity Catalog catalog to list datasources from. Required; each request lists the datasources of a single catalog.

schema_namestringquery

Name of a Unity Catalog schema in catalog_name. Optional; if set, only the datasources in this schema are returned, and if omitted, the datasources in all schemas of the catalog are returned.

viewstringquery

Which fields to return for each datasource. Defaults to DATASOURCE_VIEW_FULL. DATASOURCE_VIEW_BASIC returns only the identity and Unity Catalog location fields.

Values:

  • DATASOURCE_VIEW_BASIC
  • DATASOURCE_VIEW_FULL

Response​

Returns a list of Datasource objects.

Create a datasource Beta​

POST /api/2.0/lakewatch/datasources

Creates a new datasource.

API scopes: lakewatch

Request body​

namestringIDImmutable

Resource name of the datasource, in the format datasources/{datasource_id}. Unlike the datasource's Unity Catalog location (catalog_name, schema_name, and datasource_name), the name does not change when the datasource or its catalog or schema is renamed in Unity Catalog.

Example: datasources/550e8400-e29b-41d4-a716-446655440000

catalog_namestringRequiredImmutable

Name of the Unity Catalog catalog that contains the datasource. Required on create and cannot be changed afterward. Reads return the value as supplied; the catalog's current name is reported in effective_catalog_name.

schema_namestringRequiredImmutable

Name of the Unity Catalog schema that contains the datasource. Required on create and cannot be changed afterward. Reads return the value as supplied; the schema's current name is reported in effective_schema_name.

datasource_namestringRequiredImmutable

Name of the datasource in Unity Catalog, unique within its schema. Required on create and cannot be changed afterward. Reads return the value as supplied; the datasource's current Unity Catalog name, which reflects renames, is reported in effective_datasource_name.

descriptionstring

Free-form description of the datasource.

annotationsobject

Free-form key-value annotations on the datasource, managed by the client.

run_as_principalstring

The user or service principal that the datasource runs as, given as a user email or service principal application ID. When it is supplied on create or update, the requester must be authorized to run the datasource as that principal, or the request is rejected. When it is omitted on create, the datasource runs as the requester. On update, a mask that includes run_as_principal without a value clears it, so the datasource runs as the requester; a mask that omits it keeps the current identity. Reads return the value as supplied; the identity the datasource runs as is reported in effective_run_as_principal.

autoloaderobject
Show child attributesHide child attributes
formatstringRequired

File format of the source files. Required.

Values:

  • AUTOLOADER_FORMAT_JSON
  • AUTOLOADER_FORMAT_JSONL
  • AUTOLOADER_FORMAT_CSV
  • AUTOLOADER_FORMAT_PARQUET
  • AUTOLOADER_FORMAT_XML
locationstringRequired

URI of the source files: a path in a Unity Catalog volume or external location, or a cloud storage URL such as an Amazon S3 path. Required, including when new files are discovered through queue_url. Cannot be changed after creation.

format_optionsobject

Format-specific parsing options, passed to Auto Loader. Supported for the CSV, JSON, JSONL, and XML formats, each of which accepts a fixed set of options. The XML format requires the rowTag option.

service_credentialsstring

Name of the Unity Catalog service credential to use.

aws_secret_scopestring

Name of the workspace secret scope that holds the AWS access key ID and secret access key, under the keys awsAccessKeyId and awsSecretAccessKey.

override_partition_columnsarray of string

Hive-style partition columns to derive from the source file paths. If unset, no partition columns are derived.

raw_tableobjectRequired

Table that stores the raw data before parsers schematize it. Every Auto Loader datasource writes its raw rows to this table before any parser reads them. The table must be in the datasource's catalog and schema; an omitted catalog_name or schema_name defaults to them. Cannot be changed after creation. Reads return the value as supplied; the table the data is written to is reported in effective_raw_table.

Show child attributesHide child attributes
catalog_namestring

Name of the parent Unity Catalog catalog.

schema_namestring

Name of the parent Unity Catalog schema.

table_namestringRequired

Unity Catalog table name.

envelopeobject

How to unwrap a transport envelope around the source records before parsing. Supported only for the JSON and JSONL formats. If unset, each source record is treated as one event.

Show child attributesHide child attributes
json_arrayobject
Show child attributesHide child attributes
pathstringRequired

Name of the root-level key that holds the array of events, for example Records. Nested paths are not supported.

cloud_watch_logsobject
Show child attributesHide child attributes
header_retentionboolean

Whether to keep the envelope's owner, logGroup, and logStream values in the data.lw_header field of each unwrapped row. Must be set explicitly to true or false.

queue_urlstring

URL of a customer-managed Amazon SQS queue that receives file-arrival notifications. Set it to discover new files through the queue. If unset, new files are discovered through the file events of the source's Unity Catalog external location when it has them configured, and by directory listing otherwise. Cannot be changed after creation.

include_existing_filesboolean

Whether to ingest the files already present at location when ingestion first starts. Set true to ingest the existing files and then every new file. Set false, or leave unset, to ingest only files that arrive after the datasource starts. Defaults to false. Cannot be changed after creation. Reads return the value as supplied; the setting in effect is reported in effective_include_existing_files.

unity_catalog_connectionobject
Show child attributesHide child attributes
connection_namestringRequired

Name of the Unity Catalog connection to ingest through.

configurationobject

Free-form key-value configuration for the managed ingestion pipeline.

start_timestring

Earliest source event time to ingest. Leave unset to start at creation time, with no history backfilled. Rejected if in the future, beyond the connector's history window, or for a connector without start-time support. Takes effect only at first start and cannot be changed. Reads return the value as supplied; the start time ingestion uses is reported in effective_start_time.

existing_tableobject
Show child attributesHide child attributes
namestringRequired

Name of the source table or view. Required on create and cannot be changed. Reads return the value as supplied; the table's current name is reported in effective_name.

catalogstring

Name of the catalog the existing table lives in. Optional on create and defaults to the datasource's own catalog. Cannot be changed. Reads return the value as supplied; the catalog in use is reported in effective_catalog.

schemastring

Name of the schema the existing table lives in. Optional, but required when catalog is set; supplying catalog without schema is rejected. Defaults to the datasource's own schema. Cannot be changed. Reads return the value as supplied; the schema in use is reported in effective_schema.

include_existing_rowsboolean

Whether to ingest the rows already present in the source table when ingestion first starts. Set true to ingest the existing rows and then every new commit. Set false, or leave unset, to ingest only rows written after the datasource starts. Defaults to false, and must be true for a Delta Sharing source table. Takes effect only at first start and cannot be changed. Reads return the value as supplied.

parsersarray of object

Parsers that schematize the ingested data. Each record is matched against the parsers in order and written to the parsed table of the first parser that it matches. Existing-table and Unity Catalog connection datasources require at least one parser; an Auto Loader datasource with no parsers writes only its raw table.

Show child attributesHide child attributes
idstringRequired

Identifier of the parser. Stable across renames, unlike a parser name.

versionstringRequired

Version of the parser to use, given as an explicit version number.

target_tableobject

Table that this parser's data is written to. The table must be in the datasource's catalog and schema. If unset, the table name is derived from the parser name. Reads return the value as supplied; the table the data is written to is reported in effective_target_table.

Show child attributesHide child attributes
catalog_namestring

Name of the parent Unity Catalog catalog.

schema_namestring

Name of the parent Unity Catalog schema.

table_namestringRequired

Unity Catalog table name.

scheduleobjectRequired

Schedule on which the datasource's ingestion job runs. A datasource cannot be paused, so a request that sets schedule.enabled to false is rejected. Continuous schedules are not supported for Unity Catalog connection datasources.

Show child attributesHide child attributes
at_least_everystring

Duration string defining the minimum cadence. May contain components for days, hours, minutes, seconds (e.g. 5d, 5d3h, 3d2h5m12s, 12m16s).

cron_expressionstring

Quartz cron expression.

continuousboolean

Run continuously (sentinel — must be true when set in this branch).

compute_groupstring

dedicated, inherited, automatic, or a custom group name. Reads return the value as supplied. A rule whose schedule omits it runs on dedicated.

enabledboolean

Whether the job is scheduled to run. Defaults to true.

compute_modestring

Compute mode for the datasource's ingestion job. If unset, the default_compute_mode of the catalog's datasource configuration applies, or DS_COMPUTE_PERFORMANCE_OPTIMIZED if the catalog has none.

Values:

  • DS_COMPUTE_STANDARD
  • DS_COMPUTE_PERFORMANCE_OPTIMIZED
warehouse_idstringRequiredInput only

ID of the SQL warehouse that runs the checks Lakewatch performs to validate a create or update request, such as source and destination access checks. Required and non-blank on every create and update request; there is no default warehouse. The requester must be able to use the warehouse. Used only for those checks: it is not stored, is not returned on reads, and does not affect the compute that runs the ingestion job.

Response​

Returns the Datasource object.

Update a datasource Beta​

PATCH /api/2.0/lakewatch/{name=datasources/*}

Updates an existing datasource.

API scopes: lakewatch

Parameters​

namestringRequiredIDImmutablepath

Resource name of the datasource, in the format datasources/{datasource_id}. Unlike the datasource's Unity Catalog location (catalog_name, schema_name, and datasource_name), the name does not change when the datasource or its catalog or schema is renamed in Unity Catalog.

Example: datasources/550e8400-e29b-41d4-a716-446655440000

update_maskstringRequiredquery
etagstringquery

Optimistic-concurrency precondition: if set to a non-empty value that does not match the resource's current etag, the update is rejected as a conflict (HTTP 409). Omit this field or send an empty string to update unconditionally.

Request body​

catalog_namestringRequiredImmutable

Name of the Unity Catalog catalog that contains the datasource. Required on create and cannot be changed afterward. Reads return the value as supplied; the catalog's current name is reported in effective_catalog_name.

schema_namestringRequiredImmutable

Name of the Unity Catalog schema that contains the datasource. Required on create and cannot be changed afterward. Reads return the value as supplied; the schema's current name is reported in effective_schema_name.

datasource_namestringRequiredImmutable

Name of the datasource in Unity Catalog, unique within its schema. Required on create and cannot be changed afterward. Reads return the value as supplied; the datasource's current Unity Catalog name, which reflects renames, is reported in effective_datasource_name.

descriptionstring

Free-form description of the datasource.

annotationsobject

Free-form key-value annotations on the datasource, managed by the client.

run_as_principalstring

The user or service principal that the datasource runs as, given as a user email or service principal application ID. When it is supplied on create or update, the requester must be authorized to run the datasource as that principal, or the request is rejected. When it is omitted on create, the datasource runs as the requester. On update, a mask that includes run_as_principal without a value clears it, so the datasource runs as the requester; a mask that omits it keeps the current identity. Reads return the value as supplied; the identity the datasource runs as is reported in effective_run_as_principal.

autoloaderobject
Show child attributesHide child attributes
formatstringRequired

File format of the source files. Required.

Values:

  • AUTOLOADER_FORMAT_JSON
  • AUTOLOADER_FORMAT_JSONL
  • AUTOLOADER_FORMAT_CSV
  • AUTOLOADER_FORMAT_PARQUET
  • AUTOLOADER_FORMAT_XML
locationstringRequired

URI of the source files: a path in a Unity Catalog volume or external location, or a cloud storage URL such as an Amazon S3 path. Required, including when new files are discovered through queue_url. Cannot be changed after creation.

format_optionsobject

Format-specific parsing options, passed to Auto Loader. Supported for the CSV, JSON, JSONL, and XML formats, each of which accepts a fixed set of options. The XML format requires the rowTag option.

service_credentialsstring

Name of the Unity Catalog service credential to use.

aws_secret_scopestring

Name of the workspace secret scope that holds the AWS access key ID and secret access key, under the keys awsAccessKeyId and awsSecretAccessKey.

override_partition_columnsarray of string

Hive-style partition columns to derive from the source file paths. If unset, no partition columns are derived.

raw_tableobjectRequired

Table that stores the raw data before parsers schematize it. Every Auto Loader datasource writes its raw rows to this table before any parser reads them. The table must be in the datasource's catalog and schema; an omitted catalog_name or schema_name defaults to them. Cannot be changed after creation. Reads return the value as supplied; the table the data is written to is reported in effective_raw_table.

Show child attributesHide child attributes
catalog_namestring

Name of the parent Unity Catalog catalog.

schema_namestring

Name of the parent Unity Catalog schema.

table_namestringRequired

Unity Catalog table name.

envelopeobject

How to unwrap a transport envelope around the source records before parsing. Supported only for the JSON and JSONL formats. If unset, each source record is treated as one event.

Show child attributesHide child attributes
json_arrayobject
Show child attributesHide child attributes
pathstringRequired

Name of the root-level key that holds the array of events, for example Records. Nested paths are not supported.

cloud_watch_logsobject
Show child attributesHide child attributes
header_retentionboolean

Whether to keep the envelope's owner, logGroup, and logStream values in the data.lw_header field of each unwrapped row. Must be set explicitly to true or false.

queue_urlstring

URL of a customer-managed Amazon SQS queue that receives file-arrival notifications. Set it to discover new files through the queue. If unset, new files are discovered through the file events of the source's Unity Catalog external location when it has them configured, and by directory listing otherwise. Cannot be changed after creation.

include_existing_filesboolean

Whether to ingest the files already present at location when ingestion first starts. Set true to ingest the existing files and then every new file. Set false, or leave unset, to ingest only files that arrive after the datasource starts. Defaults to false. Cannot be changed after creation. Reads return the value as supplied; the setting in effect is reported in effective_include_existing_files.

unity_catalog_connectionobject
Show child attributesHide child attributes
connection_namestringRequired

Name of the Unity Catalog connection to ingest through.

configurationobject

Free-form key-value configuration for the managed ingestion pipeline.

start_timestring

Earliest source event time to ingest. Leave unset to start at creation time, with no history backfilled. Rejected if in the future, beyond the connector's history window, or for a connector without start-time support. Takes effect only at first start and cannot be changed. Reads return the value as supplied; the start time ingestion uses is reported in effective_start_time.

existing_tableobject
Show child attributesHide child attributes
namestringRequired

Name of the source table or view. Required on create and cannot be changed. Reads return the value as supplied; the table's current name is reported in effective_name.

catalogstring

Name of the catalog the existing table lives in. Optional on create and defaults to the datasource's own catalog. Cannot be changed. Reads return the value as supplied; the catalog in use is reported in effective_catalog.

schemastring

Name of the schema the existing table lives in. Optional, but required when catalog is set; supplying catalog without schema is rejected. Defaults to the datasource's own schema. Cannot be changed. Reads return the value as supplied; the schema in use is reported in effective_schema.

include_existing_rowsboolean

Whether to ingest the rows already present in the source table when ingestion first starts. Set true to ingest the existing rows and then every new commit. Set false, or leave unset, to ingest only rows written after the datasource starts. Defaults to false, and must be true for a Delta Sharing source table. Takes effect only at first start and cannot be changed. Reads return the value as supplied.

parsersarray of object

Parsers that schematize the ingested data. Each record is matched against the parsers in order and written to the parsed table of the first parser that it matches. Existing-table and Unity Catalog connection datasources require at least one parser; an Auto Loader datasource with no parsers writes only its raw table.

Show child attributesHide child attributes
idstringRequired

Identifier of the parser. Stable across renames, unlike a parser name.

versionstringRequired

Version of the parser to use, given as an explicit version number.

target_tableobject

Table that this parser's data is written to. The table must be in the datasource's catalog and schema. If unset, the table name is derived from the parser name. Reads return the value as supplied; the table the data is written to is reported in effective_target_table.

Show child attributesHide child attributes
catalog_namestring

Name of the parent Unity Catalog catalog.

schema_namestring

Name of the parent Unity Catalog schema.

table_namestringRequired

Unity Catalog table name.

scheduleobjectRequired

Schedule on which the datasource's ingestion job runs. A datasource cannot be paused, so a request that sets schedule.enabled to false is rejected. Continuous schedules are not supported for Unity Catalog connection datasources.

Show child attributesHide child attributes
at_least_everystring

Duration string defining the minimum cadence. May contain components for days, hours, minutes, seconds (e.g. 5d, 5d3h, 3d2h5m12s, 12m16s).

cron_expressionstring

Quartz cron expression.

continuousboolean

Run continuously (sentinel — must be true when set in this branch).

compute_groupstring

dedicated, inherited, automatic, or a custom group name. Reads return the value as supplied. A rule whose schedule omits it runs on dedicated.

enabledboolean

Whether the job is scheduled to run. Defaults to true.

compute_modestring

Compute mode for the datasource's ingestion job. If unset, the default_compute_mode of the catalog's datasource configuration applies, or DS_COMPUTE_PERFORMANCE_OPTIMIZED if the catalog has none.

Values:

  • DS_COMPUTE_STANDARD
  • DS_COMPUTE_PERFORMANCE_OPTIMIZED
warehouse_idstringRequiredInput only

ID of the SQL warehouse that runs the checks Lakewatch performs to validate a create or update request, such as source and destination access checks. Required and non-blank on every create and update request; there is no default warehouse. The requester must be able to use the warehouse. Used only for those checks: it is not stored, is not returned on reads, and does not affect the compute that runs the ingestion job.

Response​

Returns the Datasource object.

Delete a datasource Beta​

DELETE /api/2.0/lakewatch/{name=datasources/*}

Deletes a datasource by its resource name.

API scopes: lakewatch

Parameters​

namestringRequiredpath