DBFS CLI

You run Databricks DBFS CLI commands appending them to databricks fs (or the alias dbfs), prefixing all DBFS paths with dbfs:/.

databricks fs -h
Usage: databricks fs [OPTIONS] COMMAND [ARGS]...

  Utility to interact with DBFS. DBFS paths are all prefixed
  with dbfs:/. Local paths can be absolute or local.

Options:
  -v, --version
  -h, --help     Show this message and exit.

Commands:
  cat        Shows the contents of a file. Does not work for directories.
  configure
  cp         Copies files to and from DBFS.
    Options:
      -r, --recursive
      --overwrite     Overwrites files that exist already.
  ls         Lists files in DBFS.
    Options:
      --absolute      Displays absolute paths.
      -l              Displays full information including size and file type.
  mkdirs     Makes directories in DBFS.
  mv         Moves a file between two DBFS paths.
  rm         Removes files from DBFS.
    Options:
      -r, --recursive

For operations that list, move, or delete more than 10k files, we strongly discourage using the DBFS CLI.

  • The list operation (databricks fs ls) will time out after approximately 60s.
  • The move operation (databricks fs mv) will time out after approximately 60s, potentially resulting in partially moved data.
  • The delete operation (databricks fs rm) will incrementally delete batches of files.

We recommend that you perform such operations in the context of a cluster, using File system utilities. dbutils.fs covers the functional scope of the DBFS REST API, but from notebooks. Running such operations using notebooks provides better control, such as selective deletes, manageability, and the possibility to automate periodic jobs.

Copy a file to DBFS

dbfs cp test.txt dbfs:/test.txt
# Or recursively
dbfs cp -r test-dir dbfs:/test-dir

Copy a file from DBFS

dbfs cp dbfs:/test.txt ./test.txt
# Or recursively
dbfs cp -r dbfs:/test-dir ./test-dir