Skip to main content
Last updated on

Authentication

This page describes common authentication methods for the Kinesis connector on Databricks. For the full list of Kinesis options, see Kinesis.

Authenticate with a Unity Catalog connection​

In Databricks Runtime 19 and above, Databricks recommends authenticating to Kinesis with a Unity Catalog connection. A connection references a service credential by name, so you don't include access keys or role ARNs in your notebooks, queries, or logs. You govern access to the connection with Unity Catalog privileges, like any other securable. If you need a method that a connection doesn't support, use a service credential or an alternate method. For more information, see Unity Catalog connections.

To authenticate with a Unity Catalog connection, do the following:

  1. Create a Databricks service credential using an IAM role with the necessary permissions to access Kinesis. See Step 1: Create an IAM role.

    • Confirm that the IAM role attached to your service credential has the ListShards, GetRecords, and GetShardIterator permissions.

  2. Create a connection of type KINESIS that references the service credential.

    • You need the CREATE CONNECTION privilege on the metastore.
    • You can also create the connection in Catalog Explorer. Click Catalog > Create > Create a connection. See Create a connection.
    SQL
    CREATE CONNECTION IF NOT EXISTS <connection-name> TYPE KINESIS
    OPTIONS (aws_region '<aws-region>', credential '<service-credential-name>');

  3. Grant the users or groups that run the stream access to both the connection and the service credential that it references.

    SQL
    GRANT USE CONNECTION ON CONNECTION <connection-name> TO <principal>;
    GRANT ACCESS ON SERVICE CREDENTIAL <service-credential-name> TO <principal>;

  4. To run a stream using the connection, set the databricks.connection source option to your connection's name:

Python
df = (spark.readStream
  .format("kinesis")
  .option("databricks.connection", "<connection-name>")
  .option("streamName", "<stream-name>")
  .load()
)
note

When you use a Unity Catalog connection, the connection has the credential and region. Don't set the following Spark options with databricks.connection, or the stream fails with a UC_CONNECTION_OPTION_CONFLICT error:

  • serviceCredential
  • awsAccessKey and awsSecretKey
  • region
  • roleArn, roleExternalId, and roleSessionName
  • endpoint and stsEndpoint

A stream ARN includes a region, but the connector ignores it and uses the connection's aws_region instead. Set aws_region to the region where your stream is located.

Authenticate with a service credential​

In Databricks Runtime 16.1 and above, you can authenticate to Kinesis with a Databricks service credential. Databricks recommends this method in Databricks Runtime 16.1 through 18. In Databricks Runtime 19 and above, use a service credential only when a Unity Catalog connection doesn't fit your scenario. See Create service credentials.

To use a service credential, do the following:

  1. Create a Databricks service credential using an IAM role with the necessary permissions to access Kinesis. See Step 1: Create an IAM role.
  2. Specify the service credential's name with the serviceCredential option when defining a streaming read.
note

The Kinesis source requires ListShards, GetRecords, and GetShardIterator permissions. If you encounter Amazon: Access Denied exceptions, verify that your IAM role has these permissions. See Controlling Access to Amazon Kinesis Data Streams Resources Using IAM.

Alternate authentication methods​

In Databricks Runtime 16.0 and below, Databricks service credentials aren't available. Databricks has the following alternate authentication methods:

Instance profile​

Attach an instance profile during compute configuration. See Instance profiles.

Instance profiles aren't supported in standard access mode (formerly shared access mode). See Standard compute requirements and limitations.

Use access keys​

Set the awsAccessKey and awsSecretKey options.

Databricks recommends storing your keys using Databricks secrets. See Secret management.

Assume IAM role​

To assume an IAM role, set the roleArn option to the role's ARN. Your compute must have permission to assume the role, so either launch the cluster with that permission or set the awsAccessKey and awsSecretKey options. You can optionally set roleExternalId and roleSessionName.

This method supports cross-account authentication. For more information, see Delegate Access Across AWS Accounts Using IAM Roles.

On this page