Skip to main content
Last updated on

Troubleshoot Oracle ingestion

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

This page describes common issues with the Oracle integrated CDC connector in Databricks Lakeflow Connect and how to resolve them.

General pipeline troubleshooting​

The troubleshooting steps in this section apply to all ingestion pipelines in Lakeflow Connect.

If a pipeline fails while executing, click the step that failed and confirm whether the error message provides sufficient information about the nature of the error.

View pipeline event logs in the UI

You can also check and download the cluster logs from the pipeline details page by clicking Update details in the right-hand panel, then Logs. Scan the logs for errors or exceptions.

View pipeline update details in the UI

Verify the source configuration​

If a pipeline update fails, confirm that the source database is configured correctly:

  1. Confirm that archive log mode is enabled:

    SQL
    SELECT LOG_MODE FROM V$DATABASE;

    The query must return ARCHIVELOG.

  2. Confirm that supplemental logging is enabled:

    SQL
    SELECT supplemental_log_data_min, supplemental_log_data_pk, supplemental_log_data_all FROM V$DATABASE;

    This query reports supplemental logging at the database level only. If you enabled supplemental logging on individual tables instead, these columns can still return NO while your tables are configured correctly. To check a specific table, query ALL_LOG_GROUPS:

    SQL
    SELECT log_group_type FROM ALL_LOG_GROUPS WHERE owner = '<schema>' AND table_name = '<table>';

  3. Verify that the replication user has the privileges granted by DBX_ORACLE_SETUP_UTIL.GRANT_PERMISSIONS. See Oracle database user requirements.

Supplemental logging validation fails​

Databricks validates that each table you replicate has at least primary-key supplemental logging. A table satisfies this check when it has supplemental logging enabled directly, or when it inherits it from the catalog or database. Minimal supplemental logging alone does not satisfy this check.

Enable primary-key supplemental logging on the table:

SQL
ALTER TABLE <schema>.<table> ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;

For how to choose between primary-key and full supplemental logging, see Which supplemental logging method should I choose?.

TDE-encrypted data with a closed wallet​

If your database has Transparent Data Encryption (TDE) encrypted tablespaces or columns, the encryption wallet (keystore) must be open. Otherwise, LogMiner reports Unsupported Type for encrypted columns and validation fails. Confirm the wallet status:

SQL
SELECT STATUS FROM V$ENCRYPTION_WALLET;

The status must be OPEN. For a multi-tenant database that uses a united keystore, open the wallet in CDB$ROOT.

ORA-12514: listener does not currently know of service requested​

The service name in the Unity Catalog connection cannot be reached. If your database has DB_DOMAIN set, the pluggable database (PDB) services register with a domain-qualified name. Use the domain-qualified CDB$ROOT service name in the connection. See Database domain.

Multi-tenant database: user cannot see change data​

For a multi-tenant (CDB) database, confirm that:

  • The replication user is a common user (the C## prefix).
  • The user has CONTAINER_DATA=ALL set. The setup tool sets this automatically. See Container access for CDB environments.
  • The source_catalog in the pipeline is the CDB$ROOT service name.

Archive logs purged before processing​

If Oracle purges archive logs before the pipeline can process them, the affected tables require a full refresh. Increase the archive log retention period to avoid this. Databricks recommends retaining archive logs for at least 48 hours.

PERMISSION_DENIED: You are not authorized to create clusters​

Contact a Databricks account admin to grant you Unrestricted cluster creation permissions, or use a custom cluster policy. See Requirements.

A table is skipped during ingestion​

LogMiner ignores tables that contain unsupported data types or storage attributes (for example, BFILE, nested tables, or identity columns). See LogMiner limitations.

Schema or table discovery times out in the ingestion wizard​

In large Oracle environments (for example, 100 or more schemas or 1,000 or more tables), the step that discovers schemas and tables can time out. Grant the replication user SELECT only on the schemas and tables you intend to replicate. This reduces the discovery scope and avoids the timeout. See Grant SELECT privileges on tables.

default auth: cannot configure default credentials​

If you receive this error, there is an issue with discovering the current user credentials. Try replacing the following:

w = WorkspaceClient()

with:

w = WorkspaceClient(host=input('Databricks Workspace URL: '), token=input('Token: '))

See Authentication in the Databricks SDK for Python documentation.

On this page