fs command group
Note
This information applies to Databricks CLI versions 0.205 and above, which are in Public Preview. To find your version of the Databricks CLI, run databricks -v
.
The fs
command group within the Databricks CLI enables you to automate Databricks File System (DBFS) objects.
You run fs
commands by appending them to databricks fs
. To display help for the fs
command, run databricks fs -h
.
fs
commands require directory and file paths in DBFS to begin with dbfs:/
. Otherwise, the following error is returned: “Error: expected dbfs path (with the dbfs:/ prefix).”
Important
Before you use the Databricks CLI, be sure to set up the Databricks CLI and set up authentication for the Databricks CLI.
List the contents of a directory
To list the contents of a directory in DBFS, use the ls
command. Specify the path to the directory in DBFS. The following example lists the names of the objects within the DBFS root:
databricks fs ls dbfs:/
To display full information including object types, sizes, modification times since Epoch in milliseconds, as well as the objects’ names, use the --long
or -l
option. The following example lists the full information of the objects in a tmp
directory within the DBFS root:
databricks fs ls dbfs:/tmp -l
To display full object paths, use the --absolute
option. The following example lists the full information of the objects, and the objects’ full paths, in a tmp
directory within the DBFS root:
databricks fs ls dbfs:/tmp -l --absolute
Print the contents of a file
To print the contents of a file, use the cat
command. Specify the path to the file in DBFS. The following example prints the contents of the file named babynames.csv
in a tmp
directory within the DBFS root:
databricks fs cat dbfs:/tmp/babynames.csv
The following example returns “Error: not a file: </path/to/file>
”, as it tries to print the contents of a directory instead of a file:
databricks fs cat dbfs:/tmp
Create a directory
To create a directory, use the mkdir
command. Specify the path to the directory to be created in DBFS. If the directory already exists, nothing happens. The following example creates a directory named squirrel-data
in a tmp
directory within the DBFS root:
databricks fs mkdir dbfs:/tmp/squirrel-data
Copy a directory or a file
To copy a directory or a file, use the cp
command. Specify the paths to the source directory or file to copy and its destination. You can copy directories and files between the local filesystem and DBFS, and you can copy directories and files between DBFS paths. To overwrite existing files, use the --overwrite
option. To recursively copy files within a directory, use the --recursive
or -r
option.
The cp
command assumes file:/
, if file:/
is omitted.
The following example copies a directory named squirrel-data
and its contents, from a local filesystem path to a squirrels
directory within the DBFS root.
databricks fs cp /Users/<username>/squirrel-data dbfs:/squirrels -r
The following example copies a file named squirrels.csv
from a local filesystem path to a DBFS directory named squirrel-data
within the DBFS root. If the file already exists in the destination, it is overwritten.
databricks fs cp /Users/<username>/squirrels.csv dbfs:/squirrel-data --overwrite
Remove a directory
To remove a directory, use the rm
command. Specify the path to the existing directory in DBFS. The following example removes a directory named squirrel-data
in a tmp
directory within the DBFS root:
databricks fs rm dbfs:/tmp/squirrel-data
If the directory does not exist, nothing happens, and no error is returned.
If the directory exists but is not empty, the following error is returned: “Error: </path/to/directory>
: Directory is not empty.” To remove a non-empty directory and all of its contents, use the --recursive
or -r
option. The following example removes a non-empty directory named squirrel-data
in a tmp
directory within the DBFS root:
databricks fs rm dbfs:/tmp/squirrel-data -r