# Create a datasource

Launch stage: Beta

`POST /api/2.0/lakewatch/datasources`

Creates a new datasource.

API scopes: lakewatch

## Request body

- `name` (string, optional, ID, Immutable)
  Resource name of the datasource, in the format `datasources/{datasource_id}`. Unlike the
   datasource's Unity Catalog location (`catalog_name`, `schema_name`, and `datasource_name`),
   the name does not change when the datasource or its catalog or schema is renamed in Unity
   Catalog.
  Example: `datasources/550e8400-e29b-41d4-a716-446655440000`
- `catalog_name` (string, required, Immutable)
  Name of the Unity Catalog catalog that contains the datasource. Required on create and
   cannot be changed afterward. Reads return the value as supplied; the catalog's current name
   is reported in `effective_catalog_name`.
- `schema_name` (string, required, Immutable)
  Name of the Unity Catalog schema that contains the datasource. Required on create and
   cannot be changed afterward. Reads return the value as supplied; the schema's current name
   is reported in `effective_schema_name`.
- `datasource_name` (string, required, Immutable)
  Name of the datasource in Unity Catalog, unique within its schema. Required on create and
   cannot be changed afterward. Reads return the value as supplied; the datasource's current
   Unity Catalog name, which reflects renames, is reported in `effective_datasource_name`.
- `description` (string, optional)
  Free-form description of the datasource.
- `annotations` (object, optional)
  Free-form key-value annotations on the datasource, managed by the client.
- `run_as_principal` (string, optional)
  The user or service principal that the datasource runs as, given as a user email or
   service principal application ID. When it is supplied on create or update, the requester
   must be authorized to run the datasource as that principal, or the request is rejected.
   When it is omitted on create, the datasource runs as the requester. On update, a mask that
   includes `run_as_principal` without a value clears it, so the datasource runs as the
   requester; a mask that omits it keeps the current identity. Reads return the value as
   supplied; the identity the datasource runs as is reported in `effective_run_as_principal`.
- `autoloader` (object, optional)
  - `format` (string, required)
    File format of the source files. Required.
    Possible values:
    - `AUTOLOADER_FORMAT_JSON`
    - `AUTOLOADER_FORMAT_JSONL`
    - `AUTOLOADER_FORMAT_CSV`
    - `AUTOLOADER_FORMAT_PARQUET`
    - `AUTOLOADER_FORMAT_XML`
  - `location` (string, required)
    URI of the source files: a path in a Unity Catalog volume or external location, or a
     cloud storage URL such as an Amazon S3 path. Required, including when new files are
     discovered through `queue_url`. Cannot be changed after creation.
  - `format_options` (object, optional)
    Format-specific parsing options, passed to Auto Loader. Supported for the CSV, JSON, JSONL,
     and XML formats, each of which accepts a fixed set of options. The XML format requires the
     `rowTag` option.
  - `service_credentials` (string, optional)
    Name of the Unity Catalog service credential to use.
  - `aws_secret_scope` (string, optional)
    Name of the workspace secret scope that holds the AWS access key ID and secret access key,
     under the keys `awsAccessKeyId` and `awsSecretAccessKey`.
  - `override_partition_columns` (array of string, optional)
    Hive-style partition columns to derive from the source file paths. If unset, no partition
     columns are derived.
  - `raw_table` (object, required)
    Table that stores the raw data before parsers schematize it. Every Auto Loader datasource
     writes its raw rows to this table before any parser reads them. The table must be in the
     datasource's catalog and schema; an omitted `catalog_name` or `schema_name` defaults to
     them. Cannot be changed after creation. Reads return the value as supplied; the table the
     data is written to is reported in `effective_raw_table`.
    - `catalog_name` (string, optional)
      Name of the parent Unity Catalog catalog.
    - `schema_name` (string, optional)
      Name of the parent Unity Catalog schema.
    - `table_name` (string, required)
      Unity Catalog table name.
  - `envelope` (object, optional)
    How to unwrap a transport envelope around the source records before parsing. Supported only
     for the JSON and JSONL formats. If unset, each source record is treated as one event.
    - `json_array` (object, optional)
      - `path` (string, required)
        Name of the root-level key that holds the array of events, for example
         `Records`. Nested paths are not supported.
    - `cloud_watch_logs` (object, optional)
      - `header_retention` (boolean, optional)
        Whether to keep the envelope's `owner`, `logGroup`, and `logStream` values in
         the `data.lw_header` field of each unwrapped row. Must be set explicitly to
         `true` or `false`.
  - `queue_url` (string, optional)
    URL of a customer-managed Amazon SQS queue that receives file-arrival notifications. Set it
     to discover new files through the queue. If unset, new files are discovered through the
     file events of the source's Unity Catalog external location when it has them configured,
     and by directory listing otherwise. Cannot be changed after creation.
  - `include_existing_files` (boolean, optional)
    Whether to ingest the files already present at `location` when ingestion
     first starts. Set true to ingest the existing files and then every new file.
     Set false, or leave unset, to ingest only files that arrive after the
     datasource starts. Defaults to false. Cannot be changed after creation.
     Reads return the value as supplied; the setting in effect is reported in
     `effective_include_existing_files`.
- `unity_catalog_connection` (object, optional)
  - `connection_name` (string, required)
    Name of the Unity Catalog connection to ingest through.
  - `configuration` (object, optional)
    Free-form key-value configuration for the managed ingestion pipeline.
  - `start_time` (string, optional)
    Earliest source event time to ingest. Leave unset to start at creation time, with no
     history backfilled. Rejected if in the future, beyond the connector's history window, or
     for a connector without start-time support. Takes effect only at first start and cannot
     be changed. Reads return the value as supplied; the start time ingestion uses is reported
     in `effective_start_time`.
- `existing_table` (object, optional)
  - `name` (string, required)
    Name of the source table or view. Required on create and cannot be changed.
     Reads return the value as supplied; the table's current name is reported in
     `effective_name`.
  - `catalog` (string, optional)
    Name of the catalog the existing table lives in. Optional on create and
     defaults to the datasource's own catalog. Cannot be changed. Reads return
     the value as supplied; the catalog in use is reported in `effective_catalog`.
  - `schema` (string, optional)
    Name of the schema the existing table lives in. Optional, but required when
     `catalog` is set; supplying `catalog` without `schema` is rejected. Defaults
     to the datasource's own schema. Cannot be changed. Reads return the value as
     supplied; the schema in use is reported in `effective_schema`.
  - `include_existing_rows` (boolean, optional)
    Whether to ingest the rows already present in the source table when
     ingestion first starts. Set true to ingest the existing rows and then every
     new commit. Set false, or leave unset, to ingest only rows written after the
     datasource starts. Defaults to false, and must be true for a Delta Sharing
     source table. Takes effect only at first start and cannot be changed. Reads
     return the value as supplied.
- `parsers` (array of object, optional)
  Parsers that schematize the ingested data. Each record is matched against the parsers in
   order and written to the parsed table of the first parser that it matches. Existing-table
   and Unity Catalog connection datasources require at least one parser; an Auto Loader
   datasource with no parsers writes only its raw table.
  - `id` (string, required)
    Identifier of the parser. Stable across renames, unlike a parser name.
  - `version` (string, required)
    Version of the parser to use, given as an explicit version number.
  - `target_table` (object, optional)
    Table that this parser's data is written to. The table must be in the datasource's catalog
     and schema. If unset, the table name is derived from the parser name. Reads return the value
     as supplied; the table the data is written to is reported in `effective_target_table`.
    - `catalog_name` (string, optional)
      Name of the parent Unity Catalog catalog.
    - `schema_name` (string, optional)
      Name of the parent Unity Catalog schema.
    - `table_name` (string, required)
      Unity Catalog table name.
- `schedule` (object, required)
  Schedule on which the datasource's ingestion job runs. A datasource cannot be paused, so a
   request that sets `schedule.enabled` to `false` is rejected. Continuous schedules are not
   supported for Unity Catalog connection datasources.
  - `at_least_every` (string, optional)
    Duration string defining the minimum cadence. May contain components
     for days, hours, minutes, seconds (e.g. `5d`, `5d3h`, `3d2h5m12s`,
     `12m16s`).
  - `cron_expression` (string, optional)
    Quartz cron expression.
  - `continuous` (boolean, optional)
    Run continuously (sentinel — must be `true` when set in this branch).
  - `compute_group` (string, optional)
    `dedicated`, `inherited`, `automatic`, or a custom group name. Reads return
     the value as supplied. A rule whose schedule omits it runs on `dedicated`.
  - `enabled` (boolean, optional)
    Whether the job is scheduled to run. Defaults to true.
- `compute_mode` (string, optional)
  Compute mode for the datasource's ingestion job. If unset, the `default_compute_mode` of
   the catalog's datasource configuration applies, or `DS_COMPUTE_PERFORMANCE_OPTIMIZED` if
   the catalog has none.
  Possible values:
  - `DS_COMPUTE_STANDARD`
  - `DS_COMPUTE_PERFORMANCE_OPTIMIZED`
- `warehouse_id` (string, required, Input only)
  ID of the SQL warehouse that runs the checks Lakewatch performs to validate a create or update
   request, such as source and destination access checks. Required and non-blank on every create
   and update request; there is no default warehouse. The requester must be able to use the
   warehouse. Used only for those checks: it is not stored, is not returned on reads, and does
   not affect the compute that runs the ingestion job.

## Returns

Returns the Datasource object.

## Response

```json
{
  "name": "datasources/550e8400-e29b-41d4-a716-446655440000",
  "datasource_id": "string",
  "catalog_name": "string",
  "schema_name": "string",
  "effective_catalog_name": "string",
  "effective_schema_name": "string",
  "datasource_name": "string",
  "effective_datasource_name": "string",
  "description": "string",
  "version": 0,
  "create_time": "string",
  "update_time": "string",
  "created_by": {
    "id": "string",
    "principal_type": "REGULAR_USER",
    "email": "string",
    "display_name": "string"
  },
  "updated_by": {
    "id": "string",
    "principal_type": "REGULAR_USER",
    "email": "string",
    "display_name": "string"
  },
  "etag": "string",
  "annotations": {},
  "owner": {
    "id": "string",
    "principal_type": "REGULAR_USER",
    "email": "string",
    "display_name": "string"
  },
  "run_as_principal": "string",
  "effective_run_as_principal": "string",
  "autoloader": {
    "format": "AUTOLOADER_FORMAT_JSON",
    "location": "string",
    "format_options": {},
    "service_credentials": "string",
    "aws_secret_scope": "string",
    "override_partition_columns": [
      "string"
    ],
    "raw_table": {},
    "envelope": {},
    "queue_url": "string",
    "effective_raw_table": {},
    "include_existing_files": true,
    "effective_include_existing_files": true
  },
  "unity_catalog_connection": {
    "connection_name": "string",
    "connector": "string",
    "configuration": {},
    "start_time": "string",
    "effective_start_time": "string"
  },
  "existing_table": {
    "name": "string",
    "catalog": "string",
    "schema": "string",
    "include_existing_rows": true,
    "effective_catalog": "string",
    "effective_schema": "string",
    "effective_name": "string"
  },
  "parsers": [
    {
      "id": "string",
      "version": "string",
      "target_table": {},
      "effective_target_table": {}
    }
  ],
  "schedule": {
    "at_least_every": "string",
    "cron_expression": "string",
    "continuous": true,
    "compute_group": "string",
    "enabled": true
  },
  "compute_mode": "DS_COMPUTE_STANDARD",
  "status": {
    "ingestion_job": {},
    "ingestion_pipeline": {}
  },
  "compute_workspace_id": "string"
}
```