Skip to main content

sync command group

note

This information applies to Databricks CLI versions 0.205 and above. The Databricks CLI is in Public Preview.

Databricks CLI use is subject to the Databricks License and Databricks Privacy Notice, including any Usage Data provisions.

The sync command group within the Databricks CLI enables one-way synchronization of file changes within a local filesystem directory, to a directory within a remote Databricks workspace.

note
  • sync commands cannot synchronize file changes from a directory within a remote Databricks workspace, back to a directory within a local filesystem.
  • sync commands can synchronize file changes from a local development machine only to workspace user (/Users) files in your Databricks workspace. It cannot synchronize to DBFS (dbfs:/) files. To synchronize file changes from a local development machine to DBFS (dbfs:/) in your Databricks workspace, use the dbx sync utility.

You run sync commands by appending them to databricks sync. To display help for the sync command, run databricks sync -h.

Incrementally sync local file changes to a remote directory​

To perform a single, incremental, one-way synchronization of file changes within a local filesystem directory, to a directory within a remote Databricks workspace, run the sync command, as follows:

Bash
databricks sync <local-directory-path> <remote-directory-path>

For example, to do a one-time, one-way, incremental synchronization of all file changes in the folder named my-local-folder within the local current working directory, to the folder my-workspace-folder within the remote workspace, run the following command:

Bash
databricks sync ./my-local-folder /Users/someone@example.com/my-workspace-folder

In this example, only file changes since the last run of the sync command are synchronized to /Users/someone@example.com/my-workspace-folder. By default, the workspace URL within the caller's DEFAULT profile is used to determine the remote workspace to sync to.

Only sync specific files​

To include or exclude specific files to sync based on specified patterns, use the --include, --include-from, --exclude, or --exclude-from options.

Bash
databricks sync ./my-local-folder /Users/someone@example.com/my-workspace-folder --exclude-from .gitignore

If you want to sync files in a bundle, use the sync configuration mapping. See sync.

Fully sync local file changes to a remote directory​

To perform a single, full, one-way synchronization of file changes within a local filesystem directory to a directory within a remote Databricks workspace, regardless of when the last sync command was run, use the --full option, for example:

Bash
databricks sync ./my-local-folder /Users/someone@example.com/my-workspace-folder --full

Continuously sync local file changes to a remote directory​

To turn on continuous, one-way synchronization of file changes within a local filesystem directory, to a directory within a remote Databricks workspace, use the --watch option, for example:

Bash
databricks sync ./my-local-folder /Users/someone@example.com/my-workspace-folder --watch

One-way synchronization continues until the command is stopped from the terminal, typically by pressing Ctrl + c or Ctrl + z.

Polling for possible synchronization events happens once per second by default. To change this interval, use the --interval option along with the number of seconds to poll followed by the character s, for example for five seconds:

Bash
databricks sync ./my-local-folder /Users/someone@example.com/my-workspace-folder --watch --interval 5s

Change the sync progress output format​

Sync progress information is output to the terminal in text format by default. To specify the sync progress output format, use the --output option, specifying either text (the default, if --output is not otherwise specified) or json, for example:

Bash
databricks sync ./my-local-folder /Users/someone@example.com/my-workspace-folder --output json

Preview file operations for a sync​

To preview a sync without actually performing the file sync operations, use the --dry-run option, for example:

Bash
databricks sync ./my-local-folder /Users/someone@example.com/my-workspace-folder --dry-run
Output
Warn: Running in dry-run mode. No actual changes will be made.
Action: PUT: test.txt
Uploaded test.txt
Initial Sync Complete
Last updated on