Receive Delta Sharing shares using a Python client and Open ID Connect (OIDC) federation in a machine-to-machine flow (open sharing)

Preview

This feature is in Public Preview.

This page describes how data recipients can use a Python client registered in their own identity provider (IdP) to establish access to Delta Sharing shares created in Databricks. This "machine-to-machine" (M2M) OAuth Client Credentials grant flow is typically used in scenarios where an application, such as a nightly job running on a virtual machine, accesses data autonomously. This authentication flow uses OIDC federation, allowing JSON Web Tokens (JWTs) issued by the recipient’s IdP to be used as short-lived OAuth tokens that are authenticated by Databricks. This Databricks-to-open sharing authentication flow is for recipients who do not have access to a Unity Catalog-enabled Databricks workspace.

Open OIDC federation is an alternative to using long-lived Databricks-issued bearer tokens to connect non-Databricks recipients to providers. In the OAuth Client Credentials grant flow, an OAuth application is registered as a Service Principal (SP) in the recipient’s IdP. No long-lived secrets or credentials are shared between Databricks, the provider, and the recipient. For information about using bearer tokens to manage authentication to shares instead, see Create a recipient object for non-Databricks users using bearer tokens (open sharing).

This article is intended for recipients. For information about how providers can enable OIDC federation for recipients in Databricks, see Use Open ID Connect (OIDC) federation to enable authentication to Delta Sharing shares (open sharing). For information about the "user-to-machine" (U2M) flow, see Receive Delta Sharing shares using Open ID Connect (OIDC) federation in a user-to-machine flow (open sharing).

Register an app in your IdP

Before you can use OIDC federation to give your client application access to Delta Sharing shares, you must register an OAuth application in your IdP. This section describes how to register an OAuth application in Microsoft Entra ID. For other IdPs, see their documentation.

Register an app in Microsoft Entra ID

These instructions are intended as general guidance and are not guaranteed to be kept up-to-date. For detailed app registration instructions, see this Microsoft quickstart.

1. Sign in to the Microsoft Entra admin center as at least an Application Developer.
2. Go to Identity > Applications> App registrations and create a new registration without a redirect URL.
3. Go to Certificates & Secrets > Create a secret for your app.
4. Copy the secret value and store it securely.
5. On the App registrations > Overview page for the app, copy the Application (client) ID
6. Navigate to Enterprise Applications > Properties and copy the Object ID.
7. Modify the app to be a V2 application by updating the manifest:
   1. In the app's Manage section, select Manifest.
   2. In the editor, set accessTokenAcceptedVersion to 2.
   3. Save the changes.

Send required information to the Databricks data provider

If you, as a recipient, use Microsoft Entra ID, you can get the fields required by the provider by following these instructions. Always refer to Microsoft Entra ID documentation for the most up-to-date instructions.

• Issuer URL: https://login.microsoftonline.com/{tenentId}/v2.0, replacing {tenentId} with your Entra tenant ID. If you don't know your tenant ID, see the Microsoft Entra ID documentation.

• Subject Claim: Defines the JWT field used to identify the app. Supported values: oid, sub, groups.

• in Entra ID, if you use object ID of the service principal of the OAuth app, subject claim will be oid.

• Subject: Get the object ID of the service principal object of the registered OAuth app. You should have copied this in the previous step. If not, navigate to the Microsoft Entra admin center, search for App registrations, select your registered OAuth application, navigate to Enterprise Applications, and find Object ID in the properties section.

  Alternatively, you can use azp (which corresponds to clientId) as the subject-claim, but this requires calling the REST API directly. They are not accessible through the UI.

• Audience: For machine-to-Databricks authentication, typically you use the resource's clientId, but you can specify any other valid resource identifier.

  You should have copied this in the previous step. If not, navigate to the Microsoft Entra admin center, search for App registrations, select your registered application, and then locate the Application (client) ID on the Overview page. You could also use a different resource ID.

Share issuer, subject claim, subject, and audience with the provider.

Configure your app to use the OAuth profile file shared by the Databricks provider

To configure your app to access Delta Sharing shares from the provider:

1. Go to the OIDC profile portal URL that the Databricks provider shared with you.

   Request the URL if you haven't yet received it.

2. On the portal page, select the M2M tile and, under For OAuth, click Download file.

3. Modify the downloaded oauth_config.share JSON file to add your clientID, clientSecret, and scope.

   You should have copied the client ID and client secret when you registered your app. You cannot retrieve the client secret again. To retrieve the client ID, see the instructions in the previous section.

   If you choose to use the app's {clientId} as the audience, the scope should be {clientId}/.default. For example, if the audience is 61a80fb9-ce0c-4794-9f7f-2ba42a7b76f6, the scope should be 61a80fb9-ce0c-4794-9f7f-2ba42a7b76f6/.default.

   Sample profile:

   JSON

   {
     "shareCredentialsVersion": 2,
     "endpoint": "https://oregon.cloud.databricks.com/api/2.0/delta-sharing/metastores/11a11aaa-11aa-11a12-11aa-111a1aa11111/recipients/a11da11aa1-a1a1-11a1-a11a-1111a11111aa",
     "tokenEndpoint": "https://login.microsoftonline.com/a111a111-1111-1aaa-1aa1-1aa1111aa1/oauth2/v2.0/token",
     "type": "oauth_client_credentials",
     "clientId": "[REPLACE_WITH_YOUR_CLIENT_ID]",
     "clientSecret": "[REPLACE_WITH_YOUR_CLIENT_SECRET]",
     "scope": "[REPLACE_WITH_YOUR_SCOPE]"
   }

4. Install and configure the latest Delta Sharing Python OSS client.

   You must have the latest version of the Delta Sharing Python OSS client.

   Bash

   python3 -m venv .venv
   source .venv/bin/activate
   pip3 install "delta-sharing>=1.3.1"

5. Save the updated oauth_config.share file.

6. Test the configuration:

   Create a test script, test.py:

   Python

   import delta_sharing
   
   # Point to the profile file. It can be a file on the local file system or a file on a remote storage.
   
    profile_file = "oauth_config.share"
   
    # Create a SharingClient.
    client = delta_sharing.SharingClient(profile_file)
    #
    # List all shared tables.
    tables = client.list_all_tables()
   
    print(tables)
   
    # replace the following line with the coordinates of the shared table
    #table_url = profile_file + "#sample_share.sample_db.sample_table"
   
    # Fetch 10 rows from a table and convert it to a Pandas DataFrame.
    # This can be used to read sample data from a table that cannot fit in the memory.
    #df = delta_sharing.load_as_pandas(table_url, limit=10)
   
    #print(df)

   Run the script:

   Python

   python3 test.py

   The script should list the shared tables.