Dbfs

View as Markdown

DBFS API makes it simple to interact with various data sources without having to include a users credentials every time to read a file.

GetGET /api/2.0/dbfs/get-statusListGET /api/2.0/dbfs/listCreatePOST /api/2.0/dbfs/createDeletePOST /api/2.0/dbfs/deleteAdd BlockPOST /api/2.0/dbfs/add-blockClosePOST /api/2.0/dbfs/closeMkdirsPOST /api/2.0/dbfs/mkdirsMovePOST /api/2.0/dbfs/movePutPOST /api/2.0/dbfs/putReadGET /api/2.0/dbfs/read

FileInfo object

Stores the attributes of a file or directory.

file_sizeint64

The length of the file in bytes. Set to 0 for directories.

is_dirboolean

True if the path is a directory.

modification_timeint64

Last modification time of given file in milliseconds since epoch.

pathstring

The absolute path of the file or directory.

View as Markdown

The FileInfo object

{
  "file_size": 0,
  "is_dir": true,
  "modification_time": 0,
  "path": "string"
}

Get GA

GET /api/2.0/dbfs/get-status

Gets the file information for a file or directory. If the file or directory does not exist, this call throws an exception with RESOURCE_DOES_NOT_EXIST.

API scopes: files

Query parameters

pathstringRequiredquery

The path of the file or directory. The path should be the absolute DBFS path.

Example: path=/mnt/foo

Response

file_sizeint64

The length of the file in bytes. Set to 0 for directories.

is_dirboolean

True if the path is a directory.

modification_timeint64

Last modification time of given file in milliseconds since epoch.

pathstring

The absolute path of the file or directory.

Responses

200

Request completed successfully.

View as Markdown

Response

{
  "file_size": 0,
  "is_dir": true,
  "modification_time": 0,
  "path": "string"
}

List GA

GET /api/2.0/dbfs/list

List the contents of a directory, or details of the file. If the file or directory does not exist, this call throws an exception with RESOURCE_DOES_NOT_EXIST.

When calling list on a large directory, the list operation will time out after approximately 60 seconds. We strongly recommend using list only on directories containing less than 10K files and discourage using the DBFS REST API for operations that list more than 10K files. Instead, we recommend that you perform such operations in the context of a cluster, using the File system utility (dbutils.fs), which provides the same functionality without timing out.

API scopes: files

Query parameters

pathstringRequiredquery

The path of the file or directory. The path should be the absolute DBFS path.

Example: path=/mnt/foo

Response

filesarray of object

A list of FileInfo's that describe contents of directory or file. See example above.

Responses

200

Request completed successfully.

View as Markdown

Response

{
  "files": [
    {
      "file_size": 0,
      "is_dir": true,
      "modification_time": 0,
      "path": "string"
    }
  ]
}

Create GA

POST /api/2.0/dbfs/create

Opens a stream to write to a file and returns a handle to this stream. There is a 10 minute idle timeout on this handle. If a file or directory already exists on the given path and overwrite is set to false, this call will throw an exception with RESOURCE_ALREADY_EXISTS.

A typical workflow for file upload would be:

1. Issue a create call and get a handle.
2. Issue one or more add-block calls with the handle you have.
3. Issue a close call with the handle you have.

API scopes: files

Request body

overwriteboolean

The flag that specifies whether to overwrite existing file/files.

Default: false

pathstringRequired

The path of the new file. The path should be the absolute DBFS path.

Example: /mnt/foo

Response

handleint64

Handle which should subsequently be passed into the AddBlock and Close calls when writing to a file through a stream.

Responses

200

Request completed successfully.

View as Markdown

Request

JSONTerraform

{
  "overwrite": false,
  "path": "/mnt/foo"
}

Response

{
  "handle": 0
}

Delete GA

POST /api/2.0/dbfs/delete

Delete the file or directory (optionally recursively delete all files in the directory). This call throws an exception with IO_ERROR if the path is a non-empty directory and recursive is set to false or on other similar errors.

When you delete a large number of files, the delete operation is done in increments. The call returns a response after approximately 45 seconds with an error message (503 Service Unavailable) asking you to re-invoke the delete operation until the directory structure is fully deleted.

For operations that delete more than 10K files, we discourage using the DBFS REST API, but advise you to perform such operations in the context of a cluster, using the File system utility (dbutils.fs). dbutils.fs covers the functional scope of the DBFS REST API, but from notebooks. Running such operations using notebooks provides better control and manageability, such as selective deletes, and the possibility to automate periodic delete jobs.

API scopes: files

Request body

pathstringRequired

The path of the file or directory to delete. The path should be the absolute DBFS path.

Example: /mnt/foo

recursiveboolean

Whether or not to recursively delete the directory's contents. Deleting empty directories can be done without providing the recursive flag.

Default: false

Responses

200

Request completed successfully.

View as Markdown

Request

{
  "path": "/mnt/foo",
  "recursive": false
}

Response

{}

Add Block GA

POST /api/2.0/dbfs/add-block

Appends a block of data to the stream specified by the input handle. If the handle does not exist, this call will throw an exception with RESOURCE_DOES_NOT_EXIST.

If the block of data exceeds 1 MB, this call will throw an exception with MAX_BLOCK_SIZE_EXCEEDED.

API scopes: files

Request body

datastringRequired

The base64-encoded data to append to the stream. This has a limit of 1 MB.

handleint64Required

The handle on an open stream.

Responses

200

Request completed successfully.

View as Markdown

Request

{
  "data": "string",
  "handle": 0
}

Response

{}

Close GA

POST /api/2.0/dbfs/close

Closes the stream specified by the input handle. If the handle does not exist, this call throws an exception with RESOURCE_DOES_NOT_EXIST.

API scopes: files

Request body

handleint64Required

The handle on an open stream.

Responses

200

Request completed successfully.

View as Markdown

Request

{
  "handle": 0
}

Response

{}

Mkdirs GA

POST /api/2.0/dbfs/mkdirs

Creates the given directory and necessary parent directories if they do not exist. If a file (not a directory) exists at any prefix of the input path, this call throws an exception with RESOURCE_ALREADY_EXISTS. Note: If this operation fails, it might have succeeded in creating some of the necessary parent directories.

API scopes: files

Request body

pathstringRequired

The path of the new directory. The path should be the absolute DBFS path.

Example: /mnt/foo

Responses

200

Request completed successfully.

View as Markdown

Request

{
  "path": "/mnt/foo"
}

Response

{}

Move GA

POST /api/2.0/dbfs/move

Moves a file from one location to another location within DBFS. If the source file does not exist, this call throws an exception with RESOURCE_DOES_NOT_EXIST. If a file already exists in the destination path, this call throws an exception with RESOURCE_ALREADY_EXISTS. If the given source path is a directory, this call always recursively moves all files.

API scopes: files

Request body

destination_pathstringRequired

The destination path of the file or directory. The path should be the absolute DBFS path.

Example: /mnt/bar

source_pathstringRequired

The source path of the file or directory. The path should be the absolute DBFS path.

Example: /mnt/foo

Responses

200

Request completed successfully.

View as Markdown

Request

{
  "destination_path": "/mnt/bar",
  "source_path": "/mnt/foo"
}

Response

{}

Put GA

POST /api/2.0/dbfs/put

Uploads a file through the use of multipart form post. It is mainly used for streaming uploads, but can also be used as a convenient single call for data upload.

Alternatively you can pass contents as base64 string.

The amount of data that can be passed (when not streaming) using the contents parameter is limited to 1 MB. MAX_BLOCK_SIZE_EXCEEDED will be thrown if this limit is exceeded.

If you want to upload large files, use the streaming upload. For details, see dbfs/create, dbfs/addBlock, dbfs/close.

API scopes: files

Request body

contentsstring

This parameter might be absent, and instead a posted file will be used.

overwriteboolean

The flag that specifies whether to overwrite existing file/files.

Default: false

pathstringRequired

The path of the new file. The path should be the absolute DBFS path.

Example: /mnt/foo

Responses

200

Request completed successfully.

View as Markdown

Request

{
  "contents": "string",
  "overwrite": false,
  "path": "/mnt/foo"
}

Response

{}

Read GA

GET /api/2.0/dbfs/read

Returns the contents of a file. If the file does not exist, this call throws an exception with RESOURCE_DOES_NOT_EXIST. If the path is a directory, the read length is negative, or if the offset is negative, this call throws an exception with INVALID_PARAMETER_VALUE. If the read length exceeds 1 MB, this call throws an exception with MAX_READ_SIZE_EXCEEDED.

If offset + length exceeds the number of bytes in a file, it reads the contents until the end of file.

API scopes: files

Query parameters

pathstringRequiredquery

The path of the file to read. The path should be the absolute DBFS path.

Example: path=/mnt/foo

offsetint64query

The offset to read from in bytes.

Default: 0

lengthint64query

The number of bytes to read starting from the offset. This has a limit of 1 MB, and a default value of 0.5 MB.

Default: 524288

Response

bytes_readint64

The number of bytes read (could be less than length if we hit end of file). This refers to number of bytes read in unencoded version (response data is base64-encoded).

datastring

The base64-encoded contents of the file read.

Responses

200

Request completed successfully.

View as Markdown

Response

{
  "bytes_read": 0,
  "data": "string"
}