Skip to main content
Last updated on

Push images with the Databricks CLI

The Databricks CLI provides the helper command databricks air images push to combine Docker authentication, image tagging, and pushing to Artifact Registry. It detects the registry region automatically. You can use this helper instead of the manual steps in the Artifact Registry guide. The helper does not build images or submit workloads.

Prerequisites​

Configure workspace authentication​

Create or refresh a workspace OAuth profile for the workspace where you want to store the image:

Shell
databricks auth login --profile my-databricks-profile

The Docker credential helper requires this workspace OAuth profile. For supported authentication methods, see Set up Docker authentication.

Push a local image​

For example, push a local image named my-training-image:v1 to main.ml.training:v1:

Shell
databricks air images push \
  --profile my-databricks-profile \
  --source my-training-image:v1 \
  --catalog main \
  --schema ml \
  --artifact training:v1

The command prints the Unity Catalog image name to use as environment.unity_catalog_image in your workload YAML. For this example, use main.ml.training:v1, as shown in Use a Docker image in a workload.

The helper reuses a local source image when one exists. Otherwise, it pulls the source image for linux/amd64. Add --pull to refresh a source image from its registry even if it is available locally. Images must use linux/amd64. Rebuild an incompatible local image for that platform before pushing it. If the source image is in a private registry, authenticate Docker to that registry before running the helper.

To enter the image details interactively, run databricks air images push --profile my-databricks-profile without the image flags. Run databricks air images push --help for all options.

Push an image from another registry​

You can also use the CLI helper to copy an existing image from another container registry to Artifact Registry.

The following example uses a private Amazon Elastic Container Registry (ECR) repository as the source. For this example, install and configure the AWS CLI with credentials that can authenticate to ECR and pull the source image. These AWS credentials are separate from your Databricks profile.

First, authenticate Docker to the source ECR registry. Replace the example AWS account ID, region, repository, and image tag with your own values. The AWS region must match the source ECR registry, not the destination Databricks workspace. See Private registry authentication in the AWS documentation.

Shell
aws ecr get-login-password --region us-west-2 | \
  docker login --username AWS --password-stdin \
  123456789012.dkr.ecr.us-west-2.amazonaws.com

Then pass the full ECR image URI as --source:

Shell
databricks air images push \
  --profile my-databricks-profile \
  --source 123456789012.dkr.ecr.us-west-2.amazonaws.com/my-training-image:v1 \
  --catalog main \
  --schema ml \
  --artifact training:ecr-v1 \
  --pull

The --pull flag fetches the linux/amd64 image from ECR even if it is already cached locally. The helper uses your local Docker installation to pull, tag, and push the image. The source image in ECR is unchanged. After the push succeeds, set environment.unity_catalog_image to main.ml.training:ecr-v1 in your workload YAML.

On this page