Skip to main content
Last updated on

Column selection for MySQL

Use column selection to ingest only the table columns you need. This page covers the requirements that are specific to MySQL.

note

Column selection for MySQL is available only for integrated change data capture (CDC) pipelines. See Create an integrated CDC pipeline for MySQL.

Select columns to ingest​

By default, the pipeline ingests all columns in the tables you select. To ingest only some columns, set include_columns or exclude_columns in the table configuration. For a description of each property and example pipeline definitions, see Select columns to ingest.

Required key columns​

CDC keeps your destination table in sync by matching each change to a row using the source table's key. The connector picks the key in this order, and your column selection must keep all of that key's columns:

  1. Primary key: If the table has a primary key, keep all primary key columns. Unique key columns are optional.
  2. Unique key: If the table has no primary key, keep all columns of the first unique key whose columns are all NOT NULL. Unique keys are ordered alphabetically by index name. For example, if uk_email and uk_phone both have only NOT NULL columns, the connector uses uk_email. To see a table's index names, run SHOW INDEX FROM <table>.
  3. No usable key: If the table has neither, you can include or exclude any columns. A unique key with a nullable column doesn't count as a key.

To keep a column, list it in include_columns, or leave it out of exclude_columns.

If your column selection drops a required key column, the table fails to ingest with the error INGESTION_GATEWAY_KEY_COLUMN_NOT_SELECTED. To recover, update include_columns or exclude_columns so that the key is fully selected, and then run a full refresh of the table. See Fully refresh target tables.

Change column selection for a table that is already ingesting​

You can edit include_columns or exclude_columns for a table at any time, and the change is saved to your pipeline. However, changing the column selection for a table that is already being ingested requires a full refresh of that table. If you change the selection and don't run a full refresh of the affected table in the same update, the pipeline update fails with the error CANNOT_CHANGE_COLUMN_SELECTION_WITHOUT_FULL_REFRESH.

On this page