Define artifact settings dynamically in Databricks Asset Bundles

This article describes how to override the settings for artifacts in Databricks Asset Bundles. See What are Databricks Asset Bundles?

In Databricks bundle configuration files, you can join the artifact settings in a top-level artifacts mapping with the artifact settings in an targets mapping, for example (ellipses indicate omitted content, for brevity):

# ...
artifacts:
  <some-unique-programmatic-identifier-for-this-artifact>:
    # Artifact settings.

targets:
  <some-unique-programmatic-identifier-for-this-target>:
    resources:
      artifacts:
        <the-matching-programmatic-identifier-for-this-artifact>:
          # Any more artifact settings to join with the settings from the
          # matching top-level artifacts mapping.

If any artifact setting is defined both in the top-level artifacts mapping and the targets mapping for the same artifact, then the setting in the targets mapping takes precedence over the setting in the top-level artifacts mapping.

Example 1: Artifact settings defined only in the top-level artifacts mapping

To demonstrate how this works in practice, in the following example, path is defined in the top-level artifacts mapping, which defines all of the settings for the artifact (ellipses indicate omitted content, for brevity):

# ...
artifacts:
  my-artifact:
    type: whl
    path: ./my_package
# ...

When you run databricks bundle validate for this example, the resulting graph is (ellipses indicate omitted content, for brevity):

{
  "...": "...",
  "artifacts": {
    "my-artifact": {
      "type": "whl",
      "path": "./my_package",
      "...": "..."
    }
  },
  "...": "..."
}

Example 2: Conflicting artifact settings defined in multiple artifact mappings

In this example, path is defined both in the top-level artifacts mapping and in the artifacts mapping in targets. In this example, path in the artifacts mapping in targets takes precedence over path in the top-level artifacts mapping, to define the settings for the artifact (ellipses indicate omitted content, for brevity):

# ...
artifacts:
  my-artifact:
    type: whl
    path: ./my_package

targets:
  dev:
    artifacts:
      my-artifact:
        path: ./my_other_package
    # ...

When you run databricks bundle validate for this example, the resulting graph is (ellipses indicate omitted content, for brevity):

{
  "...": "...",
  "artifacts": {
    "my-artifact": {
      "type": "whl",
      "path": "./my_other_package",
      "...": "..."
    }
  },
  "...": "..."
}