Skip to main content
Last updated on

Zendesk Support connector limitations

General limitations​

  • When you run a scheduled pipeline, alerts don't trigger immediately. Instead, they trigger when the next update runs.
  • When a source table is deleted, the destination table is not automatically deleted. You must delete the destination table manually. This behavior is not consistent with Spark Declarative Pipelines on Lakeflow behavior.
  • During source maintenance periods, Databricks might not be able to access your data.
  • If a source table name conflicts with an existing destination table name, the pipeline update fails.
  • Multi-destination pipeline support is API-only.
  • You can optionally rename a table that you ingest. If you rename a table in your pipeline, it becomes an API-only pipeline, and you can no longer edit the pipeline in the UI.
  • If you select a column after a pipeline has already started, the connector does not automatically backfill data for the new column. To ingest historical data, manually run a full refresh on the table.
  • Databricks can't ingest two or more tables with the same name in the same pipeline, even if they come from different source schemas.
  • The source system assumes that the cursor columns are monotonically increasing.
  • The connector ingests raw data without transformations. Use downstream Spark Declarative Pipelines on Lakeflow pipelines for transformations.

Supported Zendesk products​

Zendesk offers a variety of products, including Zendesk Support, Zendesk Chat, Zendesk Talk, and others. Databricks only supports ingestion from Zendesk Support, including ticket data, knowledge base content, and community forum data.

API rate limits​

Zendesk enforces rate limits on its REST API, particularly for incremental data endpoints. To ensure consistent ingestion performance, Databricks recommends limiting the number of tables that you ingest at the same time. For example, you can split out tables across multiple pipelines and schedule them at different times.

Nested and custom fields​

Some fields in the schema may be nested within complex structures, and the inner-level fields can include custom attributes. To ensure compatibility and consistency, such fields are represented as a string data type. For example, the custom_fields column in the tickets table is an array of custom objects, which can have any number of subfields.

Audit records can change after creation​

Zendesk documents ticket audit records as immutable, but some fields inside an audit can continue to change after the audit is first created. For example, the chat transcript in the ChatStartedEvent event of the ticket_audits table can keep growing for several minutes after the audit's created_at timestamp, as later messages in the conversation are added.

Because the connector ingests the ticket_audits table incrementally using the created_at cursor, it reads each audit once and does not revisit it. If a field changes after the connector ingested the audit, the target table keeps the value that was present at ingestion time and does not reflect the later update. The connector surfaces no error, because it persists the exact data returned by the Zendesk API at the time of the read.

To get the current values for affected records, run a full refresh of the ticket_audits table.

On this page