Skip to main content

MySQL connector FAQ

This page answers common questions about the MySQL connector, such as supported versions and platforms, authentication methods, tested scale, and how to recover when binlog files are cleaned up before ingestion.

What MySQL versions and platforms are supported?​

The MySQL connector supports the following versions and platforms:

  • Amazon RDS for MySQL: Version 5.7.44 and later (both standalone and HA deployments)
  • Amazon Aurora MySQL: Version 5.7.mysql_aurora.2.12.2 and later (for HA setups, support is only from primary instance)
  • Amazon Aurora MySQL Serverless: Supported
  • Azure Database for MySQL Flexible Servers: Version 5.7.44 and later (both standalone and HA deployments)
  • Google Cloud SQL for MySQL: Version 5.7.44 and later
  • MySQL on EC2: Version 5.7.44 and later

What authentication methods are supported?​

The MySQL connector supports the following authentication plugins based on your MySQL version:

  • MySQL 5.7.44: Only sha256_password is supported. The replication user must be created using this authentication plugin.
  • MySQL 8.0 and later: Both sha256_password and caching_sha2_password are supported.

Does the connector support GTID-based replication?​

No, the MySQL connector doesn't support GTID (Global Transaction Identifier)-based replication. The connector uses position-based binlog replication.

You can still use the connector if GTID is enabled on your MySQL server, but the connector uses a binlog file and position-based replication regardless.

Because a binlog file and position are only meaningful on the server that produced them, a pipeline can't continue on a different server or replica without a full refresh. See Reconnecting to a different source node.

Are XA transactions supported?​

No, the MySQL connector doesn't support XA transactions (distributed transactions). If an XA transaction is performed, the table is skipped from the pipeline.

Can I ingest tables with spatial data types?​

No, spatial data types (GEOMETRY, POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, GEOMETRYCOLLECTION) aren't supported.

If a table contains spatial columns, you must exclude the entire table from ingestion. The tables with spatial types are skipped when they are detected or when a new column with spatial types is added.

Can I create multiple pipelines with the same destination table?​

No, one destination table can be managed by only one Managed Ingestion pipeline. You can't create two different Managed Ingestion pipelines with overlapping destination tables.

Can I ingest tables with the same name from different schemas?​

No, you can't ingest two tables with the same name in the same pipeline, even if they come from different source schemas. For example, you can't ingest both schema1.customers and schema2.customers in one pipeline.

To work around this, see Create multi-destination pipelines.

How do I rotate MySQL credentials?​

To rotate credentials for an existing connection:

  1. Update the password in MySQL
  2. In Databricks, go to Catalog Explorer
  3. Navigate to the connection
  4. Click Edit and update the password
  5. Save the changes

The ingestion gateway and pipeline will automatically use the new credentials on the next run.

Are table and column names case-sensitive?​

Yes, MySQL table and schema names are case-sensitive in the MySQL connector. MySQL schema and table names are case sensitive, whereas Unity Catalog catalog, schema, and table names are case insensitive. If there is a conflict due to case (for example, mytable vs MyTable), use multi destination functionality to resolve conflicts.

For details, see Identifier Case Sensitivity in the MySQL documentation.

What scale has been tested for the MySQL connector?​

The connector has been tested for 100 tables in a single pipeline with total snapshot data less than 1 TB.

Machine type recommendations for the ingestion gateway:

For gateway-based pipelines, the following are the default machine types for the ingestion gateway:

  • AWS: r5n.xlarge
  • AZURE: Standard_E4d_v4
  • GCP: n2-highmem-4

Consider using r5n.2xlarge, Standard_E8d_v4, or n2-highmem-8 for better snapshot performance.

Pipeline limits:

  • Databricks recommends 250 or fewer tables per pipeline
  • Hard limit: 1,000 flows per pipeline (which effectively supports up to 500 tables)

Note: While the hard limit supports up to 500 tables, Databricks recommends 250 or fewer for optimal performance.

Does the ingestion gateway support triggered mode execution?​

No, the ingestion gateway does not support triggered mode and must run continuously to avoid the need for full refreshes due to log cleanups.

The ingestion pipeline can run on a schedule or be triggered, but the gateway must remain running continuously.

This question applies to gateway-based pipelines. Integrated change data capture (CDC) pipelines don't use an ingestion gateway and run in triggered mode by default. See Create an integrated CDC pipeline for MySQL.

When should I perform a full refresh?​

Perform a full refresh in the following scenarios:

  • When a table flow is marked as failed in the ingestion pipeline.
  • When an incompatible schema change causes the table flow to fail.
  • When binlog files are cleaned up before the connector processes them.
  • When you manually need to re-sync a table.

To perform a full refresh, see Fully refresh target tables.

What should I do if binlog files are cleaned up before the connector processes them?​

If binlog files are purged before the connector processes them, run a full refresh of all affected tables in the pipeline. The pipeline reports the issue in the event log. In gateway-based pipelines, the ingestion gateway also skips all affected tables.

To prevent this from happening:

  • Configure adequate binlog retention (one day minimum, seven days recommended).
  • For gateway-based pipelines, keep the ingestion gateway running continuously.

My ingestion gateway takes a long time to start. How do I fix it?​

Gateways run on classic compute and provision a virtual machine (VM) on each start. If startup takes longer than a few minutes, consider the following:

  • Switch to the current pipeline channel. This is the most common fix. Preview channel builds have longer startup times. You can change this in the UI (in the pipeline's advanced settings under Channel), bundle resource file, or pipeline specification.
  • Don't restart the gateway between ingestion runs. The gateway is designed to run continuously. Stopping and restarting it re-provisions the VM on each restart and risks missing change logs if the source truncates them while the gateway is down.

If the gateway is stuck in a starting state for 15 minutes or more, create a support ticket.

This only applies to gateways. Ingestion pipelines run on Serverless compute and start quickly.

Does the connector support on-premises MySQL deployments?​

Yes, on-premises MySQL deployments are supported when connected to a Databricks workspace through:

  • Azure ExpressRoute
  • AWS Direct Connect
  • VPN connection

Ensure that:

  • Sufficient network bandwidth is available for data transfer
  • Network connectivity is stable and reliable
  • Firewall rules allow traffic on the MySQL port (default 3306)
  • Binlog retention is configured appropriately

For more information, contact the Databricks Support team. For details about cross-cloud connectivity, see Network connectivity.