Aller au contenu principal
Dernière mise à jour le

Spark Declarative Pipelines sur Lakeflow CLI (hérité)

important

Cette documentation a été retirée et pourrait ne pas être mise à jour.

Ces informations s'appliquent aux anciennes versions de la CLI Databricks 0.18 et antérieures. Databricks vous recommande d'utiliser plutôt la version 0.205 ou ultérieure de Databricks CLI. Voir CLI Databricks. Pour trouver votre version de la CLI Databricks, exécutez databricks -v.

Pour migrer de Databricks CLI version 0.18 ou inférieure vers Databricks CLI version 0.205 ou supérieure, consultez la migration de Databricks CLI.

Vous exécutez Databricks Spark Declarative Pipelines sur les sous-commandes Lakeflow CLI en les ajoutant à databricks pipelines. Ces sous-commandes appellent l'API de pipelines.

Bash
databricks pipelines -h
Usage: databricks pipelines [OPTIONS] COMMAND [ARGS]...

  Utility to interact with Databricks :re[LDP].

Options:
  -v, --version   0.18.0
  --debug         Debug Mode. Shows full stack trace on error.
  --profile TEXT  CLI connection profile to use. The default profile is
                  "DEFAULT".

  -h, --help      Show this message and exit.

Commands:
  create  Creates a pipeline.
  delete  Deletes the pipeline and cancels any active updates.
  deploy  [Deprecated] This command is deprecated, use create and edit
          commands instead.  Creates or edits a pipeline specified by the
          pipeline settings.

  edit    Edits a pipeline.
  get     Gets a pipeline's current settings and status.
  list    Lists all pipelines and their statuses.
  reset   [Deprecated] Use the "start --full-refresh" command instead. Resets
          a pipeline so that data can be reprocessed from the beginning.

  run     [Deprecated] Use the "start" command instead. Starts a pipeline
          update.

  start   Starts a pipeline update.
  stop    Stops the pipeline by cancelling any active update.

Créer un pipeline​

Pour afficher la documentation d'utilisation, exécutez databricks pipelines create --help.

Utilisation​

Bash
databricks pipelines create --settings create-pipeline.json

create-pipeline.json:

JSON
{
  "name": "Wikipedia pipeline (SQL)",
  "storage": "/Users/username/data",
  "clusters": [
    {
      "label": "default",
      "autoscale": {
        "min_workers": 1,
        "max_workers": 5
      }
    }
  ],
  "libraries": [
    {
      "notebook": {
        "path": "/Users/username/DLT quickstart (SQL)"
      }
    }
  ],
  "target": "wikipedia_analysis",
  "continuous": false
}
Console
Successfully created pipeline: https://<databricks-instance>/#joblist/pipelines/4ee6b5c2-7017-11ec-90d6-0242ac120003 with ID: 4ee6b5c2-7017-11ec-90d6-0242ac120003.

Modifier un pipeline​

Pour afficher la documentation d'utilisation, exécutez databricks pipelines edit --help.

Utilisation​

Bash
databricks pipelines edit --settings edit-pipeline.json

edit-pipeline.json:

JSON
{
  "id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
  "name": "Wikipedia pipeline (SQL)",
  "storage": "/Users/username/data",
  "clusters": [
    {
      "label": "default",
      "autoscale": {
        "min_workers": 1,
        "max_workers": 3
      }
    }
  ],
  "libraries": [
    {
      "notebook": {
        "path": "/Users/username/DLT quickstart (SQL)"
      }
    }
  ],
  "target": "wikipedia_quickstart_data",
  "continuous": false
}
Console
Successfully edited pipeline settings: https://<databricks-instance>/#joblist/pipelines/4ee6b5c2-7017-11ec-90d6-0242ac120003.

Supprimer un pipeline​

Pour afficher la documentation d'utilisation, exécutez databricks pipelines delete --help.

Utilisation​

Bash
databricks pipelines delete --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Console
Pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003 deleted

Obtenir des informations sur un pipeline​

Pour afficher la documentation d'utilisation, exécutez databricks pipelines get --help.

Utilisation​

Bash
databricks pipelines get --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
JSON
{
  "pipeline_id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
  "spec": {
    "id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
    "name": "Wikipedia pipeline (SQL)",
    "storage": "/Users/username/data",
    "clusters": [
      {
        "label": "default",
        "autoscale": {
          "min_workers": 1,
          "max_workers": 5
        }
      }
    ],
    "libraries": [
      {
        "notebook": {
          "path": "/Users/username/DLT quickstart (SQL)"
        }
      }
    ],
    "target": "wikipedia_analysis",
    "continuous": false
  },
  "state": "IDLE",
  "name": "Wikipedia pipeline (SQL)",
  "creator_user_name": "username@databricks.com",
  "last_modified": 1641600056265
}

Répertorier tous les pipelines et les informations sur leur état​

Pour afficher la documentation d'utilisation, exécutez databricks pipelines list --help.

Utilisation​

Bash
databricks pipelines list
JSON
[
  {
    "pipeline_id": "8a93e1b4-7024-11ec-90d6-0242ac120003",
    "state": "IDLE",
    "name": "Retail sales pipeline (SQL)",
    "latest_updates": [
      {
        "update_id": "9df03924-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-29T16:18:57.982Z"
      },
      {
        "update_id": "a57e601c-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-28T18:19:30.371Z"
      }
    ],
    "creator_user_name": "user@databricks.com"
  },
  {
    "pipeline_id": "b46e2670-7024-11ec-90d6-0242ac120003",
    "state": "IDLE",
    "name": ":re[LDP] quickstart (Python)",
    "latest_updates": [
      {
        "update_id": "bcd8fa2e-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-12-16T18:19:25.827Z"
      },
      {
        "update_id": "c2c7a2c8-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-29T22:22:32.586Z"
      },
      {
        "update_id": "c88d2818-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-29T22:16:18.258Z"
      }
    ],
    "creator_user_name": "user@databricks.com"
  }
]

start une mise à jour du pipeline​

Pour afficher la documentation d'utilisation, exécutez databricks pipelines start --help.

Bash
databricks pipelines start --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Console
Started an update c6462e2e-beb7-4c24-a9d3-962eaa19781b for pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.

start a pipeline update avec un refresh complet​

Pour afficher la documentation d'utilisation, exécutez databricks pipelines start --help.

Utilisation​

Bash
databricks pipelines start --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003 --full-refresh
Console
Started an update ed885d3d-4fc0-44fd-b5dd-59471c789327 with full refresh for pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.

Arrêter une mise à jour de pipeline​

Pour afficher la documentation d'utilisation, exécutez databricks pipelines stop --help.

Utilisation​

Bash
databricks pipelines stop --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Console
Stopped pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.
Sur cette page