Skip to main content
Last updated on

Troubleshoot common sharing issues in OpenSharing

The following sections describe common errors that might occur when you try to access data in a share.

Resource limit exceeded errors​

Issue: Your query on a shared table returns the error RESOURCE_LIMIT_EXCEEDED.

  • "RESOURCE_LIMIT_EXCEEDED","message":"The table metadata size exceeded limits"
  • "RESOURCE_LIMIT_EXCEEDED","message":"The number of files in the table to return exceeded limits, consider contact your provider to optimize the table"

Possible causes: There are limits on the number of files in metadata allowed for a shared table:

  • Active files: A shared table supports a maximum of 400,000 active files (AddFile actions). If the number of active files exceeds this limit, queries return the "number of files exceeded" error.
  • Remove file actions: A shared table supports a maximum of 100,000 RemoveFile actions in the Delta log. If the number of RemoveFile actions exceeds this limit, queries return the "metadata size exceeded" error.

You can request a limit increase for the active files limit. See Resource limits.

Recommended fix: Contact your data provider and ask them to optimize the shared table by running OPTIMIZE and VACUUM to compact small files and remove stale RemoveFile entries from the Delta log. For additional guidance, see RESOURCE_LIMIT_EXCEEDED error when querying a Delta Sharing table in the Databricks Knowledge Base.

AWS S3 bucket name issue​

Issue: You see an error message that throws a file not found or certificate exception.

Spark error example:

FileReadException: Error while reading file delta-sharing:/%252Ftmp%252Fexample.share%2523example.tpc_ds.example/XXXXXXXXXXXXX/XXXXXXXX.

Caused by: SSLPeerUnverifiedException: Certificate for - <[workspace name].cloud.databricks.com.s3.us-east-1.amazonaws.com> doesn't match any of the subject alternative names [s3.amazonaws.com, *.s3.amazonaws.com…]:

Pandas error example:

FileNotFoundError(path)
FileNotFoundError: https://xxxx.xxxxxx.s3.xx-xxxx-1.amazonaws.com/xxxxxx/part-00000-xxxxx-Amz-Algorithm=Axxxxxx-Amz-Date=xxxxxxxx&X-Amz-SignedHeaders=host&X-Amz-Expires=xxx&X-Amz-Credential=xxxxxxx_request&X-Amz-Signature=xxxxx

Power BI error example:

DataSource.Error: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
Details:
    https://xxxx.xxxxxxxxx.s3.xx-xxxx-1.amazonaws.com/xxxxxxxx/part-00000-xxxxxxx.snappy.parquet

Possible cause: Typically you see this error because your bucket name uses dot or period notation (for example, incorrect.bucket.name.notation). This is an AWS limitation. See the AWS bucket naming rules.

You might get this error even if your bucket name is formatted correctly. For example, you might encounter an SSL error (SSLCertVerificationError) when you execute code on PyCharm.

Recommended fix: If your bucket name uses invalid AWS bucket naming notation, use a different bucket for Unity Catalog and OpenSharing.

If your bucket uses valid naming conventions and you still face a FileNotFoundError in Python, enable debug logging to help isolate the issue:

Python
import logging
logging.basicConfig(level=logging.DEBUG)

Vacuumed data file issue​

Issue: You see an error message that throws a “404 The specified [path|key] does not exist” exception.

Spark error examples:

java.lang.Throwable: HTTP request failed with status: HTTP/1.1 404 The specified path does not exist.

or

HTTP request failed with status: HTTP/1.1 404 Not Found <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message>

Possible cause: Typically you see this error because the data file corresponding to the pre-signed URL is vacuumed in the shared table and the data file belongs to a historical table version.

Workaround: Query the latest snapshot.

Schema mismatch error with Open Source Spark​

Issue: When using Open Source Spark (OSS), you see a schema mismatch error when reading OpenSharing tables.

Error example:

py4j.protocol.Py4JJavaError: An error occurred while calling o85.count.: org.apache.spark.SparkException: The schema or partition columns of your Delta table has changed since your DataFrame was created. Please redefine your DataFrame

Possible cause: The schema or partition columns of the Delta table changed after the DataFrame was created.

Recommended fix: Set the Spark configuration flag spark.delta.sharing.client.useStructuralSchemaMatch to true.

note

The spark.delta.sharing.client.useStructuralSchemaMatch configuration is only available in delta-sharing-client 1.2.3 or above, which requires Apache Spark 4.0.0 or above.

Python
spark.conf.set("spark.delta.sharing.client.useStructuralSchemaMatch", "true")

Spark connector setup errors with Open Source Spark​

The following errors can occur when you start PySpark or spark-shell with the OpenSharing Spark connector, or when you first read a shared table. For setup instructions and compatible version combinations, see Apache Spark: Read shared data.

Java version is too old​

Issue: PySpark fails to start with an UnsupportedClassVersionError.

Error: LinkageError occurred while loading main class org.apache.spark.launcher.Main
	java.lang.UnsupportedClassVersionError: org/apache/spark/launcher/Main has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0

Possible cause: Your Java version is older than your version of Apache Spark requires. Apache Spark 4.x requires Java 17 or above. Class file version 61.0 corresponds to Java 17.

Recommended fix: Install Java 17 or above, and set the JAVA_HOME environment variable to the new Java installation. Then run java -version to confirm the version.

Spark connector built for a different Scala version​

Issue: Reading a shared table fails with a NoClassDefFoundError for a scala.collection class.

java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: io.delta.sharing.spark.DeltaSharingDataSource Unable to get public no-arg constructor
Caused by: java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce

The Caused by line might also be:

Caused by: java.lang.NoClassDefFoundError: scala/collection/IterableOnce

Possible cause: The Scala version in the Spark connector's Maven coordinate doesn't match the Scala version of your Apache Spark installation. The missing class tells you which way they're mismatched:

  • scala/collection/GenTraversableOnce: You're using a delta-sharing-spark_2.12 connector with a Scala 2.13 build of Apache Spark, such as PySpark 4.x.
  • scala/collection/IterableOnce: You're using a delta-sharing-spark_2.13 connector with a Scala 2.12 build of Apache Spark, such as PySpark 3.5.

Recommended fix: Run pyspark --version to find your Scala version. Then restart PySpark with a Spark connector for that Scala version: delta-sharing-spark_2.13 for Scala 2.13, or delta-sharing-spark_2.12 for Scala 2.12. See Choose a Spark connector version.

Spark connector built for a different Spark version​

Issue: Reading a shared table fails with a NoSuchMethodError.

Caused by: java.lang.NoSuchMethodError: 'org.apache.spark.sql.SparkSession$implicits$ org.apache.spark.sql.SparkSession.implicits()'

Possible cause: The Spark connector release was built for a different major version of Apache Spark. For example, Spark connector 3.3.3 is built for Apache Spark 3.5, so it doesn't work with Apache Spark 4.x.

Recommended fix: Restart PySpark with a Spark connector release that matches your Apache Spark version. See Choose a Spark connector version.

Spark connector can't be downloaded​

Issue: PySpark fails to start with an unresolved dependency error.

::          UNRESOLVED DEPENDENCIES         ::
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: io.delta#delta-sharing-spark_2.13;4.4.1: not found]

Possible causes:

  • Your network blocks access to Maven Central.
  • The Maven coordinate in the --packages option contains a typo or a version that doesn't exist.

Recommended fix: Check the Maven coordinate against Choose a Spark connector version. If your network blocks access to Maven Central, add the --repositories option with the URL of a Maven repository mirror that your organization provides.

SQL statement fails with a syntax error in the PySpark shell​

Issue: Running a SQL statement in the PySpark shell fails with SyntaxError: invalid syntax.

Possible cause: The PySpark shell runs Python. It can't run SQL statements directly.

Recommended fix: Run SQL statements using spark.sql(). See Query shared data using SQL.

Shared materialization asset access issue​

Issue: Your query on a shared view, materialized view, or streaming table returns the error DS_MATERIALIZATION_QUERY_FAILED.

"DS_MATERIALIZATION_QUERY_FAILED": "The shared asset could not be materialized due to the asset not being accessible in the materialization workspace. Please ask data provider to contact :re[DB] support to override the materialization workspace."

Possible causes: The provider does not have read-write access to the asset they are trying to share.

Recommended fix: Contact your data provider to ensure they have read-write access to the shared data asset.

Network access error during data materialization​

Issue: Your query on a shared data asset returns an error about accessing the data provider's cloud storage.

There was an issue accessing the data provider's cloud storage. Shared view materialization uses the Serverless compute of data provider's region to perform the materialization. Please contact the data provider to allowlist Serverless compute IPs of their corresponding region to access the view's dependent tables storage location.

Possible causes: The storage location for the materialized data has network restrictions (such as a firewall or private link) that prevent Databricks serverless compute from accessing it. When sharing views, materialized views, or streaming tables, the data is temporarily materialized on the provider's side. The materialization storage location is the asset's parent schema or catalog storage location.

Recommended fix: The data provider needs to allowlist serverless compute IPs of their corresponding region to access the view's dependent tables storage location. To configure your firewall, see Serverless compute firewall configuration.

Shared materialized view read error caused by a bucket policy​

Issue: Your query on a shared materialized view returns a FAILED_READ_FILE error with SQLSTATE KD001, but queries on shared tables in the same S3 bucket succeed.

[FAILED_READ_FILE.NO_HINT] Error while reading file delta-sharing:/... SQLSTATE: KD001

Possible cause: The provider's S3 bucket policy restricts presigned URLs, for example with a Deny statement on the s3:signatureAge condition key. Recipients read shared materialized views through presigned URLs, so the policy blocks those reads. Shared tables are read with signed request headers instead, so the same policy doesn't affect them. The error message doesn't mention the bucket policy.

Recommended fix: Contact your data provider and ask them to update the bucket policy so that it allows presigned URL access to the materialized view's storage location. See Amazon S3 Signature Version 4 authentication specific policy keys in the AWS documentation.

Data asset "does not exist" error​

If you click on a shared asset and encounter an object "does not exist" error, the share owner on the provider side might not have sufficient permissions on the asset. Contact your data provider and ask them to verify that the share owner has the required permissions on all shared assets. For more information about permissions required for a share owner to share a data asset, see Requirements and Grant recipient access to share.

On this page