# The Datasource object

A Lakewatch datasource: a pipeline that ingests security data and writes it to Unity
 Catalog tables.

 What a datasource ingests is defined by its ingestion method (`autoloader`,
 `unity_catalog_connection`, or `existing_table`), which sets how the data arrives,
 together with its `parsers`, which define the schemas that the arriving data is matched
 against and written as.

## Attributes

- `name` (string, Beta, ID, Immutable)
  Resource name of the datasource, in the format `datasources/{datasource_id}`. Unlike the
   datasource's Unity Catalog location (`catalog_name`, `schema_name`, and `datasource_name`),
   the name does not change when the datasource or its catalog or schema is renamed in Unity
   Catalog.
  Example: `datasources/550e8400-e29b-41d4-a716-446655440000`
- `datasource_id` (string, Beta, Output only)
  Unity Catalog securable ID (a UUID) of the datasource. It never changes, including when the
   datasource is renamed in Unity Catalog, and is the final segment of `name`.
- `catalog_name` (string, Beta, Immutable)
  Name of the Unity Catalog catalog that contains the datasource. Required on create and
   cannot be changed afterward. Reads return the value as supplied; the catalog's current name
   is reported in `effective_catalog_name`.
- `schema_name` (string, Beta, Immutable)
  Name of the Unity Catalog schema that contains the datasource. Required on create and
   cannot be changed afterward. Reads return the value as supplied; the schema's current name
   is reported in `effective_schema_name`.
- `effective_catalog_name` (string, Beta, Output only)
  Current name of the Unity Catalog catalog that contains the datasource, which reflects
   catalog renames without a change to `catalog_name`. Populated on get, create, and update
   responses; list responses omit it.
- `effective_schema_name` (string, Beta, Output only)
  Current name of the Unity Catalog schema that contains the datasource, which reflects
   schema renames without a change to `schema_name`. Populated on get, create, and update
   responses; list responses omit it.
- `datasource_name` (string, Beta, Immutable)
  Name of the datasource in Unity Catalog, unique within its schema. Required on create and
   cannot be changed afterward. Reads return the value as supplied; the datasource's current
   Unity Catalog name, which reflects renames, is reported in `effective_datasource_name`.
- `effective_datasource_name` (string, Beta, Output only)
  Current UC leaf name of the datasource (server-owned). Tracks UC renames so
   callers can observe the live name without having to update the client-owned
   `datasource_name`; unset until resolved.
- `description` (string, Beta)
  Free-form description of the datasource.
- `version` (int64, Beta, Output only)
  Version number of the datasource, incremented on each update. Informational only; use
   `etag` for optimistic concurrency control.
- `create_time` (string, Beta, Output only)
- `update_time` (string, Beta, Output only)
- `created_by` (object, Beta, Output only)
  - `id` (string, Beta)
    User / service-principal ID.
  - `principal_type` (string, Beta)
    Whether this principal is a regular user, service principal, or system principal.
    Possible values:
    - `REGULAR_USER`
    - `SERVICE_PRINCIPAL`
    - `SYSTEM_USER`
  - `email` (string, Beta)
    Email (regular users only).
  - `display_name` (string, Beta)
    Display name (user or SP name).
- `updated_by` (object, Beta, Output only)
  - `id` (string, Beta)
    User / service-principal ID.
  - `principal_type` (string, Beta)
    Whether this principal is a regular user, service principal, or system principal.
    Possible values:
    - `REGULAR_USER`
    - `SERVICE_PRINCIPAL`
    - `SYSTEM_USER`
  - `email` (string, Beta)
    Email (regular users only).
  - `display_name` (string, Beta)
    Display name (user or SP name).
- `etag` (string, Beta, Output only)
  Opaque entity tag for optimistic concurrency control. Returned by get, create, and update
   responses and by list responses in the `DATASOURCE_VIEW_FULL` view. To make an update
   conditional on the datasource not having changed since it was read, pass this value as the
   update request's `etag`. Changes to `status` do not change the etag.
- `annotations` (object, Beta)
  Free-form key-value annotations on the datasource, managed by the client.
- `owner` (object, Beta, Output only)
  Unity Catalog owner of the datasource, assigned by the server. Create responses omit it.
  - `id` (string, Beta)
    User / service-principal ID.
  - `principal_type` (string, Beta)
    Whether this principal is a regular user, service principal, or system principal.
    Possible values:
    - `REGULAR_USER`
    - `SERVICE_PRINCIPAL`
    - `SYSTEM_USER`
  - `email` (string, Beta)
    Email (regular users only).
  - `display_name` (string, Beta)
    Display name (user or SP name).
- `run_as_principal` (string, Beta)
  The user or service principal that the datasource runs as, given as a user email or
   service principal application ID. When it is supplied on create or update, the requester
   must be authorized to run the datasource as that principal, or the request is rejected.
   When it is omitted on create, the datasource runs as the requester. On update, a mask that
   includes `run_as_principal` without a value clears it, so the datasource runs as the
   requester; a mask that omits it keeps the current identity. Reads return the value as
   supplied; the identity the datasource runs as is reported in `effective_run_as_principal`.
- `effective_run_as_principal` (string, Beta, Output only)
  The identity the datasource runs as: the resolved `run_as_principal`, or the requester's
   identity if `run_as_principal` was omitted on create or cleared on update. Given as a user
   email or service principal application ID, or as the numeric principal ID if the identity
   can no longer be resolved. Populated on get, create, and update responses; list responses
   omit it.
- `autoloader` (object, Beta)
  - `format` (string, Beta)
    File format of the source files. Required.
    Possible values:
    - `AUTOLOADER_FORMAT_JSON`
    - `AUTOLOADER_FORMAT_JSONL`
    - `AUTOLOADER_FORMAT_CSV`
    - `AUTOLOADER_FORMAT_PARQUET`
    - `AUTOLOADER_FORMAT_XML`
  - `location` (string, Beta)
    URI of the source files: a path in a Unity Catalog volume or external location, or a
     cloud storage URL such as an Amazon S3 path. Required, including when new files are
     discovered through `queue_url`. Cannot be changed after creation.
  - `format_options` (object, Beta)
    Format-specific parsing options, passed to Auto Loader. Supported for the CSV, JSON, JSONL,
     and XML formats, each of which accepts a fixed set of options. The XML format requires the
     `rowTag` option.
  - `service_credentials` (string, Beta)
    Name of the Unity Catalog service credential to use.
  - `aws_secret_scope` (string, Beta)
    Name of the workspace secret scope that holds the AWS access key ID and secret access key,
     under the keys `awsAccessKeyId` and `awsSecretAccessKey`.
  - `override_partition_columns` (array of string, Beta)
    Hive-style partition columns to derive from the source file paths. If unset, no partition
     columns are derived.
  - `raw_table` (object, Beta)
    Table that stores the raw data before parsers schematize it. Every Auto Loader datasource
     writes its raw rows to this table before any parser reads them. The table must be in the
     datasource's catalog and schema; an omitted `catalog_name` or `schema_name` defaults to
     them. Cannot be changed after creation. Reads return the value as supplied; the table the
     data is written to is reported in `effective_raw_table`.
    - `catalog_name` (string, Beta)
      Name of the parent Unity Catalog catalog.
    - `schema_name` (string, Beta)
      Name of the parent Unity Catalog schema.
    - `table_name` (string, Beta)
      Unity Catalog table name.
  - `envelope` (object, Beta)
    How to unwrap a transport envelope around the source records before parsing. Supported only
     for the JSON and JSONL formats. If unset, each source record is treated as one event.
    - `json_array` (object, Beta)
      - `path` (string, Beta)
        Name of the root-level key that holds the array of events, for example
         `Records`. Nested paths are not supported.
    - `cloud_watch_logs` (object, Beta)
      - `header_retention` (boolean, Beta)
        Whether to keep the envelope's `owner`, `logGroup`, and `logStream` values in
         the `data.lw_header` field of each unwrapped row. Must be set explicitly to
         `true` or `false`.
  - `queue_url` (string, Beta)
    URL of a customer-managed Amazon SQS queue that receives file-arrival notifications. Set it
     to discover new files through the queue. If unset, new files are discovered through the
     file events of the source's Unity Catalog external location when it has them configured,
     and by directory listing otherwise. Cannot be changed after creation.
  - `effective_raw_table` (object, Beta, Output only)
    The table that the raw data is written to: `raw_table` in the datasource's catalog and
     schema, under their current Unity Catalog names. Populated on get, create, and update
     responses.
    - `catalog_name` (string, Beta)
      Name of the parent Unity Catalog catalog.
    - `schema_name` (string, Beta)
      Name of the parent Unity Catalog schema.
    - `table_name` (string, Beta)
      Unity Catalog table name.
  - `include_existing_files` (boolean, Beta)
    Whether to ingest the files already present at `location` when ingestion
     first starts. Set true to ingest the existing files and then every new file.
     Set false, or leave unset, to ingest only files that arrive after the
     datasource starts. Defaults to false. Cannot be changed after creation.
     Reads return the value as supplied; the setting in effect is reported in
     `effective_include_existing_files`.
  - `effective_include_existing_files` (boolean, Beta, Output only)
    Whether ingestion started with the files already present at `location`. Populated on get,
     create, and update responses.
- `unity_catalog_connection` (object, Beta)
  - `connection_name` (string, Beta)
    Name of the Unity Catalog connection to ingest through.
  - `connector` (string, Beta, Output only)
    Connector type of the connection, such as `SLACK_AUDIT_LOGS`. Resolved by the server from
     `connection_name` through Unity Catalog; not accepted as input.
  - `configuration` (object, Beta)
    Free-form key-value configuration for the managed ingestion pipeline.
  - `start_time` (string, Beta)
    Earliest source event time to ingest. Leave unset to start at creation time, with no
     history backfilled. Rejected if in the future, beyond the connector's history window, or
     for a connector without start-time support. Takes effect only at first start and cannot
     be changed. Reads return the value as supplied; the start time ingestion uses is reported
     in `effective_start_time`.
  - `effective_start_time` (string, Beta, Output only)
    The start time ingestion uses: the supplied `start_time`, or the creation time when it was
     omitted. Always populated.
- `existing_table` (object, Beta)
  - `name` (string, Beta)
    Name of the source table or view. Required on create and cannot be changed.
     Reads return the value as supplied; the table's current name is reported in
     `effective_name`.
  - `catalog` (string, Beta)
    Name of the catalog the existing table lives in. Optional on create and
     defaults to the datasource's own catalog. Cannot be changed. Reads return
     the value as supplied; the catalog in use is reported in `effective_catalog`.
  - `schema` (string, Beta)
    Name of the schema the existing table lives in. Optional, but required when
     `catalog` is set; supplying `catalog` without `schema` is rejected. Defaults
     to the datasource's own schema. Cannot be changed. Reads return the value as
     supplied; the schema in use is reported in `effective_schema`.
  - `include_existing_rows` (boolean, Beta)
    Whether to ingest the rows already present in the source table when
     ingestion first starts. Set true to ingest the existing rows and then every
     new commit. Set false, or leave unset, to ingest only rows written after the
     datasource starts. Defaults to false, and must be true for a Delta Sharing
     source table. Takes effect only at first start and cannot be changed. Reads
     return the value as supplied.
  - `effective_catalog` (string, Beta, Output only)
    Current Unity Catalog catalog name of the source table, resolved from its
     stable table ID, so a rename of the source table is reported here without
     changing `catalog`.
  - `effective_schema` (string, Beta, Output only)
    Current Unity Catalog schema name of the source table, resolved from its
     stable table ID.
  - `effective_name` (string, Beta, Output only)
    Current Unity Catalog name of the source table, resolved from its stable
     table ID.
- `parsers` (array of object, Beta)
  Parsers that schematize the ingested data. Each record is matched against the parsers in
   order and written to the parsed table of the first parser that it matches. Existing-table
   and Unity Catalog connection datasources require at least one parser; an Auto Loader
   datasource with no parsers writes only its raw table.
  - `id` (string, Beta)
    Identifier of the parser. Stable across renames, unlike a parser name.
  - `version` (string, Beta)
    Version of the parser to use, given as an explicit version number.
  - `target_table` (object, Beta)
    Table that this parser's data is written to. The table must be in the datasource's catalog
     and schema. If unset, the table name is derived from the parser name. Reads return the value
     as supplied; the table the data is written to is reported in `effective_target_table`.
    - `catalog_name` (string, Beta)
      Name of the parent Unity Catalog catalog.
    - `schema_name` (string, Beta)
      Name of the parent Unity Catalog schema.
    - `table_name` (string, Beta)
      Unity Catalog table name.
  - `effective_target_table` (object, Beta, Output only)
    The table this parser's data is written to: `target_table` when supplied, otherwise the
     table derived from the datasource's catalog and schema and the parser name. Fixed once
     resolved, so a later parser rename does not move it. Populated on get, create, and update
     responses with the resolved table.
    - `catalog_name` (string, Beta)
      Name of the parent Unity Catalog catalog.
    - `schema_name` (string, Beta)
      Name of the parent Unity Catalog schema.
    - `table_name` (string, Beta)
      Unity Catalog table name.
- `schedule` (object, Beta)
  Schedule on which the datasource's ingestion job runs. A datasource cannot be paused, so a
   request that sets `schedule.enabled` to `false` is rejected. Continuous schedules are not
   supported for Unity Catalog connection datasources.
  - `at_least_every` (string, Beta)
    Duration string defining the minimum cadence. May contain components
     for days, hours, minutes, seconds (e.g. `5d`, `5d3h`, `3d2h5m12s`,
     `12m16s`).
  - `cron_expression` (string, Beta)
    Quartz cron expression.
  - `continuous` (boolean, Beta)
    Run continuously (sentinel — must be `true` when set in this branch).
  - `compute_group` (string, Beta)
    `dedicated`, `inherited`, `automatic`, or a custom group name. Reads return
     the value as supplied. A rule whose schedule omits it runs on `dedicated`.
  - `enabled` (boolean, Beta)
    Whether the job is scheduled to run. Defaults to true.
- `compute_mode` (string, Beta)
  Compute mode for the datasource's ingestion job. If unset, the `default_compute_mode` of
   the catalog's datasource configuration applies, or `DS_COMPUTE_PERFORMANCE_OPTIMIZED` if
   the catalog has none.
  Possible values:
  - `DS_COMPUTE_STANDARD`
  - `DS_COMPUTE_PERFORMANCE_OPTIMIZED`
- `status` (object, Beta, Output only)
  Runtime status of the jobs and pipelines that run this datasource.
  - `ingestion_job` (object, Beta, Output only)
    Status of the datasource's ingestion job.
    - `job_id` (string, Beta, Output only)
      Databricks Jobs API job ID. Unset for a pure ingestion pipeline.
    - `lifecycle_state` (string, Beta, Output only)
      Scheduling lifecycle of the resource's underlying job/pipeline: whether it
       is unscheduled, scheduled, stopped, or deleted. Distinct from `run_state`,
       which reports the outcome of the most recent run.
      Possible values:
      - `UNSCHEDULED`
      - `SCHEDULED`
      - `STOPPED`
      - `DELETED`
    - `run_state` (string, Beta, Output only)
      Possible values:
      - `RUNNING`
      - `SUCCEEDED`
      - `FAILED`
      - `CANCELED`
    - `last_error_message` (string, Beta, Output only)
      Most recent error message, if any. Pairs with `run_state = FAILED`.
    - `create_time` (string, Beta, Output only)
    - `last_start_time` (string, Beta, Output only)
      When the most recent run started. Pairs with `run_state = RUNNING`.
    - `last_complete_time` (string, Beta, Output only)
      When the most recent run completed. Pairs with `run_state = SUCCEEDED`.
    - `last_status_change_time` (string, Beta, Output only)
    - `pipeline_id` (string, Beta, Output only)
      DLT ingestion-pipeline ID. Set only when this status represents the
       datasource's ingestion pipeline (unset for job statuses).
  - `ingestion_pipeline` (object, Beta, Output only)
    Status of the managed ingestion pipeline. Populated only when `unity_catalog_connection`
     is set, with `pipeline_id` identifying the pipeline.
    - `job_id` (string, Beta, Output only)
      Databricks Jobs API job ID. Unset for a pure ingestion pipeline.
    - `lifecycle_state` (string, Beta, Output only)
      Scheduling lifecycle of the resource's underlying job/pipeline: whether it
       is unscheduled, scheduled, stopped, or deleted. Distinct from `run_state`,
       which reports the outcome of the most recent run.
      Possible values:
      - `UNSCHEDULED`
      - `SCHEDULED`
      - `STOPPED`
      - `DELETED`
    - `run_state` (string, Beta, Output only)
      Possible values:
      - `RUNNING`
      - `SUCCEEDED`
      - `FAILED`
      - `CANCELED`
    - `last_error_message` (string, Beta, Output only)
      Most recent error message, if any. Pairs with `run_state = FAILED`.
    - `create_time` (string, Beta, Output only)
    - `last_start_time` (string, Beta, Output only)
      When the most recent run started. Pairs with `run_state = RUNNING`.
    - `last_complete_time` (string, Beta, Output only)
      When the most recent run completed. Pairs with `run_state = SUCCEEDED`.
    - `last_status_change_time` (string, Beta, Output only)
    - `pipeline_id` (string, Beta, Output only)
      DLT ingestion-pipeline ID. Set only when this status represents the
       datasource's ingestion pipeline (unset for job statuses).
- `warehouse_id` (string, Beta, Input only)
  ID of the SQL warehouse that runs the checks Lakewatch performs to validate a create or update
   request, such as source and destination access checks. Required and non-blank on every create
   and update request; there is no default warehouse. The requester must be able to use the
   warehouse. Used only for those checks: it is not stored, is not returned on reads, and does
   not affect the compute that runs the ingestion job.
- `compute_workspace_id` (string, Beta, Output only)
  ID of the workspace whose compute runs this datasource's ingestion job and
   owns its associated resources, such as the storage the ingested data lands
   in. Server-assigned.

## Example

```json
{
  "name": "datasources/550e8400-e29b-41d4-a716-446655440000",
  "datasource_id": "string",
  "catalog_name": "string",
  "schema_name": "string",
  "effective_catalog_name": "string",
  "effective_schema_name": "string",
  "datasource_name": "string",
  "effective_datasource_name": "string",
  "description": "string",
  "version": 0,
  "create_time": "string",
  "update_time": "string",
  "created_by": {
    "id": "string",
    "principal_type": "REGULAR_USER",
    "email": "string",
    "display_name": "string"
  },
  "updated_by": {
    "id": "string",
    "principal_type": "REGULAR_USER",
    "email": "string",
    "display_name": "string"
  },
  "etag": "string",
  "annotations": {},
  "owner": {
    "id": "string",
    "principal_type": "REGULAR_USER",
    "email": "string",
    "display_name": "string"
  },
  "run_as_principal": "string",
  "effective_run_as_principal": "string",
  "autoloader": {
    "format": "AUTOLOADER_FORMAT_JSON",
    "location": "string",
    "format_options": {},
    "service_credentials": "string",
    "aws_secret_scope": "string",
    "override_partition_columns": [
      "string"
    ],
    "raw_table": {},
    "envelope": {},
    "queue_url": "string",
    "effective_raw_table": {},
    "include_existing_files": true,
    "effective_include_existing_files": true
  },
  "unity_catalog_connection": {
    "connection_name": "string",
    "connector": "string",
    "configuration": {},
    "start_time": "string",
    "effective_start_time": "string"
  },
  "existing_table": {
    "name": "string",
    "catalog": "string",
    "schema": "string",
    "include_existing_rows": true,
    "effective_catalog": "string",
    "effective_schema": "string",
    "effective_name": "string"
  },
  "parsers": [
    {
      "id": "string",
      "version": "string",
      "target_table": {},
      "effective_target_table": {}
    }
  ],
  "schedule": {
    "at_least_every": "string",
    "cron_expression": "string",
    "continuous": true,
    "compute_group": "string",
    "enabled": true
  },
  "compute_mode": "DS_COMPUTE_STANDARD",
  "status": {
    "ingestion_job": {},
    "ingestion_pipeline": {}
  },
  "warehouse_id": "string",
  "compute_workspace_id": "string"
}
```