Skip to main content
Last updated on

Upload large files with multipart upload

Beta

This feature is in Beta. To use it, a workspace admin must turn on Files API V3 with Large File Upload Support from the Previews page. See Manage Databricks previews.

Multipart upload lets you upload large files to Unity Catalog volumes by splitting them into smaller parts that upload independently and are then assembled into a single file. This approach follows the same model used by major cloud providers.

A single-request upload, which uploads a file in a single operation, accepts files up to 5000 MiB. To upload a larger file, or a file whose total size is unknown when the upload starts, use multipart upload.

Requirements​

How multipart upload works​

Multipart upload has three steps:

  1. Initiate the upload to start a session and receive an upload ID.
  2. Upload parts, sending each part's content with the upload ID and a part number.
  3. Complete the upload by providing the list of uploaded parts.

You can also cancel an in-progress upload at any time with the abort-upload action.

Part numbers range from 1 to 10000. Every part except the last must be at least 5 MiB, and each part can be at most 4000 MiB. Databricks assembles the parts in ascending part-number order when the upload completes.

Example​

The following example uploads a file in two parts.

To start the upload session, initiate the upload. The response contains the upload_id that identifies the session:

HTTP
POST /api/3.0/fs/files/Volumes/c/s/v/file?action=initiate-upload

200 OK
Content-Type: application/json
{
  "upload_id": "dvOR6FE39mNb"
}

To upload the first part, send its content with the upload_id and part_number. The response includes the part's ETag:

HTTP
PUT /api/3.0/fs/files/Volumes/c/s/v/file?upload_id=dvOR6FE39mNb&part_number=1
Content-Length: 268435456
Content-Type: application/octet-stream
<binary data>

200 OK
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"

To upload the second part, repeat the request with the next part_number:

HTTP
PUT /api/3.0/fs/files/Volumes/c/s/v/file?upload_id=dvOR6FE39mNb&part_number=2
Content-Length: 268435456
Content-Type: application/octet-stream
<binary data>

200 OK
ETag: "954fb694be262821177cfbc47f9d4a0aa06c349b"

To finish, complete the upload with a body that lists every part and its stored ETag:

HTTP
POST /api/3.0/fs/files/Volumes/c/s/v/file?action=complete-upload&upload_id=dvOR6FE39mNb
Content-Type: application/json
{
  "parts": [
    { "part_number": 1, "etag": "\"33a64df551425fcc55e4d42a148795d9f25f89d4\"" },
    { "part_number": 2, "etag": "\"954fb694be262821177cfbc47f9d4a0aa06c349b\"" }
  ]
}

200 OK
ETag: "b3f1c2a94d6e7085f2a1c0b9e8d7654321fedcba"

The ETag in this response belongs to the whole assembled file, not to a part. You can use it in later conditional requests, for example to overwrite the file only if it has not changed since you uploaded it.

To cancel the upload instead of completing it, send the abort-upload action with the upload_id:

HTTP
POST /api/3.0/fs/files/Volumes/c/s/v/file?action=abort-upload&upload_id=dvOR6FE39mNb

200 OK
On this page