Configure Databricks S3 commit service-related settings

Databricks runs a commit service that coordinates writes to Amazon S3 from multiple clusters. This service runs in the Databricks control plane. For additional security, you can disable the service’s direct upload optimization as described in Disable the direct upload optimization. To further restrict access to your S3 buckets, see Additional bucket security restrictions.

About the commit service

The S3 commit service helps guarantee consistency of writes across multiple clusters on a single table in specific cases. For example, the commit service helps Delta Lake implement ACID transactions.

In the default configuration, Databricks sends temporary AWS credentials from the data plane to the control plane in the commit service API call. Instance profile credentials are valid for six hours.

The data plane writes data directly to S3, and then the control plane provides concurrency control by finalizing the commit log upload (completing the multipart upload described below). The commit service does not read any data from S3. It puts a new file in S3 if it does not exist.

The most common data that is written to S3 by the Databricks commit service is the Delta log, which contains statistical aggregates from your data, such as the column’s minimum and maximum values. Most Delta log data is sent to S3 from the data plane using an Amazon S3 multipart upload.

After the cluster stages the multipart data to write the Delta log to S3, the S3 commit service in the Databricks control plane finishes the S3 multipart upload by letting S3 know that it is complete. As a performance optimization for very small updates, by default the commit service sometimes pushes small updates directly from the control plane to S3. This direct update optimization can be disabled. See Disable the direct upload optimization.

In addition to Delta Lake, the following Databricks features use the same S3 commit service:

• Structured Streaming
• Auto Loader
• The SQL command COPY INTO

The commit service is necessary because Amazon doesn’t provide an operation that puts an object only if it does not yet exist. Amazon S3 is a distributed system. If S3 receives multiple write requests for the same object simultaneously, it overwrites all but the last object written. Without the ability to centrally verify commits, simultaneous commits from different clusters would corrupt tables.

Disable the direct upload optimization

As a performance optimization for very small updates, by default the commit service sometimes pushes small updates directly from the control plane to S3. To disable this optimization, set the Spark parameter spark.hadoop.fs.s3a.databricks.s3commit.directPutFileSizeThreshold to 0. You can apply this setting in the cluster’s Spark config or set it in a global init script.

Disabling this feature may result in a small performance impact for near real-time Structured Streaming queries with constant small updates. Consider testing the performance impact with your data before disabling this feature in production.

Additional bucket security restrictions

The following bucket policy configurations further restrict access to your S3 buckets.

• Limit the bucket access to specific IP addresses and S3 operations. If you are interested in additional controls on your bucket, you can limit specific S3 buckets to be accessible only from specific IP addresses. For example, you can restrict access to only your own environment and the IP addresses for the Databricks control plane, including the S3 commit service. See Restrict access to your S3 buckets. This configuration limits the risk that credentials are used from other locations.

• Limit S3 operation types outside the required directories. You can deny access from the Databricks control plane to your S3 bucket outside the required directories for the S3 commit service. You also can limit the operations in those directories to just the required S3 operations put and list from Databricks IP addresses. The Databricks control plane (including the S3 commit service) does not require get access on the bucket.

  {
    "Sid": "LimitCommitServiceActions",
    "Effect": "Deny",
    "Principal": "*",
    "NotAction": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:PutObject"
    ],
    "Resource": [
        "arn:aws:s3:::<bucket-name>/*",
        "arn:aws:s3:::<bucket-name>"
    ],
    "Condition": {
        "IpAddress": {
            "aws:SourceIp": "<control-plane-ip>"
        }
    }
  },
  {
    "Sid": "LimitCommitServicePut",
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:PutObject",
    "NotResource": [
        "arn:aws:s3:::<bucket-name>/*_delta_log/*",
        "arn:aws:s3:::<bucket-name>/*_spark_metadata/*",
        "arn:aws:s3:::<bucket-name>/*offsets/*",
        "arn:aws:s3:::<bucket-name>/*sources/*",
        "arn:aws:s3:::<bucket-name>/*sinks/*",
        "arn:aws:s3:::<bucket-name>/*_schemas/*"
    ],
    "Condition": {
        "IpAddress": {
            "aws:SourceIp": "<control-plane-ip>"
        }
    }
  },
  {
    "Sid": "LimitCommitServiceList",
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:ListBucket",
    "Resource": "arn:aws:s3:::<bucket-name>",
    "Condition": {
        "StringNotLike": {
            "s3:Prefix": [
                "*_delta_log/*",
                "*_spark_metadata/*",
                "*offsets/*",
                "*sources/*",
                "*sinks/*",
                "*_schemas/*"
            ]
        },
        "IpAddress": {
            "aws:SourceIp": "<control-plane-ip>"
        }
    }
  }
  

  Replace <control-plane-ip> with your regional IP address for the Databricks control plane. Replace <bucket-name> with your S3 bucket name.