Skip to main content
Last updated on

Use the Files API

Use the Files API to read, write, list, and delete files and directories by path. It works with files in Unity Catalog volumes, which you refer to by their absolute volume path, such as /Volumes/<catalog>/<schema>/<volume>/<path>. Files API 3.0 requests use the /api/3.0/fs endpoint prefix and go to your workspace's regional hostname. For every operation, parameter, and header, see the Files API 3.0 reference.

Beta

This feature is in Beta. To use it, a workspace admin must turn on Files API V3 with Large File Upload Support from the Previews page. See Manage Databricks previews.

Use of the Files API might incur Databricks data transfer charges.

Send requests to the regional hostname​

Send Files API 3.0 requests to the regional hostname of the workspace that the requesting user belongs to. The regional hostname has the following format:

<workspace-id>.storage.<workspace-region>.<databricks-domain>

For example, 1234567890123456.storage.us-west-2.cloud.databricks.com.

To find your workspace ID and region, use the account console. See Workspace instance names, URLs, and IDs. You can also find the workspace ID in the deployment URL: in https://<databricks-instance>/?o=1234567890123456, the number after o= is the workspace ID. The workspace UI shows the region below the workspace name.

Configure network connectivity​

Files API 3.0 requires network connectivity from your network to Databricks and from Databricks to your storage.

From your network to Databricks​

If you allow public internet access to your workspace but restrict it with a firewall, allowlist the control plane IP ranges for your region.

Include the IPs under Default storage, OpenSharing SecureConnect, Zerobus Ingestion, and Lakebase Autoscaling. See Inbound IPs.

If you don't allow internet access to Databricks and connect through a front-end private link, create a second front-end private link for performance-intensive services. See Configure inbound PrivateLink for performance-intensive services.

From Databricks to your storage​

If you allow public access to your storage but apply network restrictions, for in-region storage, allowlist the Databricks serverless OrgPath. Otherwise, allowlist the Databricks outbound IPs.

If you don't allow public access to your storage, create a private link to Amazon S3. See Configure private connectivity to AWS-managed resources.

note

Serverless network billing applies. See Understand Databricks networking costs.

Upload large files​

A single upload request accepts files up to 5000 MiB. To upload a larger file, or a file whose size is unknown when the upload starts, use a Files API 3.0 multipart upload: start a session with the initiate-upload action, upload each part with PUT, then assemble the parts with the complete-upload action. For part limits and request details, see Upload a file and Perform an action on a file.

Resume an interrupted download​

To resume an interrupted download, request the remaining bytes with a Range header that starts at the first byte you have not received. To make sure the file did not change between requests, also send If-Match with the ETag from your first response:

HTTP
GET /api/3.0/fs/files/Volumes/main/default/my-volume/large-file.bin
Range: bytes=1048576-
If-Match: "33a64df551425fcc55e4d42a148795d9f25f89d4"

206 Partial Content
Content-Range: bytes 1048576-10485759/10485760
<binary data>

If the file changed, the server returns 412 Precondition Failed, and you must start the download again. Combining bytes from different versions of a file corrupts the result. For Range header syntax, see Download a file.

Conditional requests​

Use conditional headers to run a request only when the file's current ETag or Last-Modified value meets a condition that you set. For the headers that each operation supports, see the Files API 3.0 reference. When you use them, note the following:

  • Store the ETag exactly as the server returns it, including the surrounding quotes, and send it back unchanged. Don't parse or construct an ETag.
  • Last-Modified has a precision of one second, so If-Modified-Since and If-Unmodified-Since can't detect several changes within the same second. To prevent concurrent writers from overwriting each other's changes, use If-Match or If-None-Match instead.
  • A read request can include at most two conditional headers. When you send two, their precedence follows RFC 9110.

Files API versions​

Files API 2.0 sends requests to your workspace hostname with the /api/2.0/fs endpoint prefix. It doesn't support multipart upload, and its only conditional header is If-Unmodified-Since on downloads. See the Files API 2.0 reference.

To move code from Files API 2.0 to 3.0, make the following changes:

  • Send requests to the regional hostname with the /api/3.0/fs endpoint prefix, and configure network connectivity.
  • Send the Content-Length header on every upload. If you don't know the file size in advance, use multipart upload.
  • Expect 200 OK instead of 204 No Content from a successful upload.
  • Expect 204 No Content when you delete a file that doesn't exist. To get 404 Not Found instead, as in Files API 2.0, send If-Match: *.
On this page