Google Cloud Storage

This article describes how to read from and write to Google Cloud Storage (GCS) tables in Databricks. To read or write from a GCS bucket, you must create an attached service account and you must associate the bucket with the service account when creating a cluster.

Connect to the bucket directly with a key that you generate for the service account.

Access a GCS bucket directly

To read and write directly to a bucket, you configure a key defined in your Spark config.

Step 1: Set up Google Cloud service account using Google Cloud Console

You must create a service account for the Databricks cluster. Databricks recommends giving this service account the least privileges needed to perform its tasks.

1. Click IAM and Admin in the left navigation pane.

2. Click Service Accounts.

3. Click + CREATE SERVICE ACCOUNT.

4. Enter the service account name and description.

   

5. Click CREATE.

6. Click CONTINUE.

7. Click DONE.

8. Create a key. See Create a key to access GCS bucket directly.

Create a key to access GCS bucket directly

Warning

The JSON key you generate for the service account is a private key that should only be shared with authorized users as it controls access to datasets and resources in your Google Cloud account.

1. In the Google Cloud console, in the service accounts list, click the newly created account.

2. In the Keys section, click ADD KEY > Create new key.

   

3. Accept the JSON key type.

4. Click CREATE. The key file is downloaded to your computer.

Step 2: Configure the GCS bucket

Create a bucket

If you do not already have a bucket, create one:

1. Click Storage in the left navigation pane.

2. Click CREATE BUCKET.

   

3. Click CREATE.

Configure the bucket

1. Configure the bucket details.

2. Click the Permissions tab.

3. Next to the Permissions label, click ADD.

   

4. Provide the Storage Admin permission to the service account on the bucket from the Cloud Storage roles.

   

5. Click SAVE.

Step 3: Set up Databricks cluster

When you configure your cluster:

1. In the Databricks Runtime Version drop-down, select 7.3 LTS or above.

2. In the Spark Config tab, add all of the following Spark configuration. Replace <client_email>, <project_id>, <private_key>, and <private_key_id> with the values of those exact field names from your key JSON file.

   Important

   The value for <private_key> spans multiple lines. Paste the entire private key. Do not include the leading and trailing quotes.

   spark.hadoop.google.cloud.auth.service.account.enable true
   spark.hadoop.fs.gs.auth.service.account.email <client_email>
   spark.hadoop.fs.gs.project.id <project_id>
   spark.hadoop.fs.gs.auth.service.account.private.key <private_key>
   spark.hadoop.fs.gs.auth.service.account.private.key.id <private_key_id>
   

Step 4: Usage

To read from the GCS bucket, use a Spark read command in any supported format, for example:

df = spark.read.format("parquet").load("gs://<bucket-name>/<path>")

To write to the GCS bucket, use a Spark write command in any supported format, for example:

df.write.format("parquet").mode("<mode>").save("gs://<bucket-name>/<path>")

Replace <bucket-name> with the name of the bucket you created in Step 2: Configure the GCS bucket.

Example notebooks

Read from Google Cloud Storage notebook

Open notebook in new tab Copy link for import

Write to Google Cloud Storage notebook

Open notebook in new tab Copy link for import