fs command group

Note

This information applies to Databricks CLI versions 0.205 and above, which are in Public Preview. To find your version of the Databricks CLI, run databricks -v.

The fs command group within the Databricks CLI enables you to automate Databricks File System (DBFS) objects.

You run fs commands by appending them to databricks fs. To display help for the fs command, run databricks fs -h.

fs commands require directory and file paths in DBFS to begin with dbfs:/. Otherwise, the following error is returned: “Error: expected dbfs path (with the dbfs:/ prefix).”

Important

Before you use the Databricks CLI, be sure to set up the Databricks CLI and set up authentication for the Databricks CLI.

List the contents of a directory

To list the contents of a directory in DBFS, use the ls command. Specify the path to the directory in DBFS. The following example lists the names of the objects within the DBFS root:

databricks fs ls dbfs:/

To display full information including object types, sizes, modification times since Epoch in milliseconds, as well as the objects’ names, use the --long or -l option. The following example lists the full information of the objects in a tmp directory within the DBFS root:

databricks fs ls dbfs:/tmp -l

To display full object paths, use the --absolute option. The following example lists the full information of the objects, and the objects’ full paths, in a tmp directory within the DBFS root:

databricks fs ls dbfs:/tmp -l --absolute

Print the contents of a file

To print the contents of a file, use the cat command. Specify the path to the file in DBFS. The following example prints the contents of the file named babynames.csv in a tmp directory within the DBFS root:

databricks fs cat dbfs:/tmp/babynames.csv

The following example returns “Error: not a file: </path/to/file>”, as it tries to print the contents of a directory instead of a file:

databricks fs cat dbfs:/tmp

Create a directory

To create a directory, use the mkdir command. Specify the path to the directory to be created in DBFS. If the directory already exists, nothing happens. The following example creates a directory named squirrel-data in a tmp directory within the DBFS root:

databricks fs mkdir dbfs:/tmp/squirrel-data

Copy a directory or a file

To copy a directory or a file, use the cp command. Specify the paths to the source directory or file to copy and its destination. You can copy directories and files between the local filesystem and DBFS, and you can copy directories and files between DBFS paths. To overwrite existing files, use the --overwrite option. To recursively copy files within a directory, use the --recursive or -r option.

The cp command assumes file:/, if file:/ is omitted.

The following example copies a directory named squirrel-data and its contents, from a local filesystem path to a squirrels directory within the DBFS root.

databricks fs cp /Users/<username>/squirrel-data dbfs:/squirrels -r

The following example copies a file named squirrels.csv from a local filesystem path to a DBFS directory named squirrel-data within the DBFS root. If the file already exists in the destination, it is overwritten.

databricks fs cp /Users/<username>/squirrels.csv dbfs:/squirrel-data --overwrite

Remove a directory

To remove a directory, use the rm command. Specify the path to the existing directory in DBFS. The following example removes a directory named squirrel-data in a tmp directory within the DBFS root:

databricks fs rm dbfs:/tmp/squirrel-data

If the directory does not exist, nothing happens, and no error is returned.

If the directory exists but is not empty, the following error is returned: “Error: </path/to/directory>: Directory is not empty.” To remove a non-empty directory and all of its contents, use the --recursive or -r option. The following example removes a non-empty directory named squirrel-data in a tmp directory within the DBFS root:

databricks fs rm dbfs:/tmp/squirrel-data -r