Rename an ingested table

Applies to: API-based pipeline authoring

By default, target tables created by Lakeflow Connect managed ingestion pipelines have the same name as the source tables. However, you can optionally rename an ingested table. For example, if you ingest an object into two target tables in the same schema, you'll have to rename one of the target tables to differentiate between them (duplicates aren't supported). If you rename a table in your pipeline, it becomes an API-only pipeline, and you can no longer edit the pipeline in the UI.

Example: Google Analytics

Databricks Asset Bundles

YAML

resources:
  pipelines:
    pipeline_ga4:
      name: <pipeline-name>
      catalog: <target-catalog>	# Location of the pipeline event log
      schema: <target-schema> # Location of the pipeline event log
      ingestion_definition:
        connection_name: <connection>
        objects:
	        - table:
              source_url: <project-id>
              source_schema: <property-name>
              destination_catalog: <target-catalog>
              destination_schema: <target-schema>
              destination_table: <custom-target-table-name> # Table rename

Databricks notebook

Google Analytics:

Python

pipeline_spec = """
{
  "name": "<pipeline>",
  "ingestion_definition": {
    "connection_name": "<connection>",
    "objects": [
      {
        "table": {
          "source_catalog": "<project-id>",
          "source_schema": "<property-name>",
          "source_table": "<source-table>",
          "destination_catalog": "<target-catalog>",
          "destination_schema": "<target-schema>",
          "destination_table": "<custom-target-table-name>",
        }
      }
    ]
  }
}
"""

Databricks CLI

JSON

{
  "resources": {
    "pipelines": {
      "pipeline_ga4": {
        "name": "<pipeline>",
        "catalog": "<target-catalog>",
        "schema": "<target-schema>",
        "ingestion_definition": {
          "connection_name": "<connection>",
          "objects": [
            {
              "table": {
                "source_url": "<project-id>",
                "source_schema": "<property-name>",
                "destination_catalog": "<destination-catalog>",
                "destination_schema": "<destination-schema>",
                "destination_table": "<custom-destination-table-name>"
              }
            }
          ]
        }
      }
    }
  }
}

Example: Salesforce

Databricks Asset Bundles

YAML

resources:
  pipelines:
    pipeline_sfdc:
      name: <pipeline-name>
      catalog: <target-catalog>	# Location of the pipeline event log
      schema: <target-schema> # Location of the pipeline event log
      ingestion_definition:
        connection_name: <connection>
        objects:
	        - table:
              source_schema: <source-schema>
              source_table: <source-table>
              destination_catalog: <target-catalog>
              destination_schema: <target-schema>
              destination_table: <custom-target-table-name> # Table rename

Databricks notebook

Python

pipeline_spec = """
{
  "name": "<pipeline>",
  "ingestion_definition": {
    "connection_name": "<connection>",
    "objects": [
      {
        "table": {
          "source_catalog": "<source-catalog>",
          "source_schema": "<source-schema>",
          "source_table": "<source-table>",
          "destination_catalog": "<target-catalog>",
          "destination_schema": "<target-schema>",
          "destination_table": "<custom-target-table-name>",
        }
      }
    ]
  }
}
"""

Databricks CLI

JSON

{
  "resources": {
    "pipelines": {
      "pipeline_sfdc": {
        "name": "<pipeline>",
        "catalog": "<target-catalog>",
        "schema": "<target-schema>",
        "ingestion_definition": {
          "connection_name": "<connection>",
          "objects": [
            {
              "table": {
                "source_schema": "<source-schema>",
                "source_table": "<source-table>",
                "destination_catalog": "<destination-catalog>",
                "destination_schema": "<destination-schema>",
                "destination_table": "<custom-destination-table-name>"
              }
            }
          ]
        }
      }
    }
  }
}

Example: Workday

Databricks Asset Bundles

YAML

resources:
  pipelines:
    pipeline_workday:
      name: <pipeline>
      catalog: <target-catalog>	# Location of the pipeline event log
      schema: <target-schema> # Location of the pipeline event log
      ingestion_definition:
        connection_name: <connection>
        objects:
	        - report:
              source_url: <report-url>
              destination_catalog: <target-catalog>
              destination_schema: <target-schema>
              destination_table: <custom-target-table-name> # Table rename

Databricks notebook

Python

pipeline_spec = """
{
  "name": "<pipeline>",
  "ingestion_definition": {
    "connection_name": "<connection>",
    "objects": [
      {
        "report": {
          "source_url": "<report-url>",
          "destination_catalog": "<target-catalog>",
          "destination_schema": "<target-schema>",
          "destination_table": "<custom-target-table-name>",
        }
      }
    ]
  }
}
"""

Databricks CLI

JSON

{
  "resources": {
    "pipelines": {
      "pipeline_workday": {
        "name": "<pipeline>",
        "catalog": "<target-catalog>",
        "schema": "<target-schema>",
        "ingestion_definition": {
          "connection_name": "<connection>",
          "objects": [
            {
              "report": {
                "source_url": "<report-url>",
                "destination_catalog": "<destination-catalog>",
                "destination_schema": "<destination-schema>",
                "destination_table": "<custom-destination-table-name>"
              }
            }
          ]
        }
      }
    }
  }
}