CLI de pipeline declarativoLakeFlow Spark (legado)
Essa documentação foi descontinuada e pode não estar atualizada.
Esta informação se aplica ao legado Databricks CLI versões 0.18 e abaixo. Databricks recomenda que o senhor use a versão mais recente do Databricks CLI 0.205 ou o acima. Consulte O que é a CLI do Databricks? Para encontrar sua versão do site Databricks CLI, execute databricks -v.
Para migrar de Databricks CLI versão 0.18 ou abaixo para Databricks CLI versão 0.205 ou acima, consulte Databricks CLI migration.
Você executa subcomandos da CLI do pipeline declarativo Databricks LakeFlow Spark adicionando-os a databricks pipelines. Esses subcomandos chamam a APIdo pipeline.
databricks pipelines -h
Usage: databricks pipelines [OPTIONS] COMMAND [ARGS]...
Utility to interact with Databricks :re[LDP].
Options:
-v, --version 0.18.0
--debug Debug Mode. Shows full stack trace on error.
--profile TEXT CLI connection profile to use. The default profile is
"DEFAULT".
-h, --help Show this message and exit.
Commands:
create Creates a pipeline.
delete Deletes the pipeline and cancels any active updates.
deploy [Deprecated] This command is deprecated, use create and edit
commands instead. Creates or edits a pipeline specified by the
pipeline settings.
edit Edits a pipeline.
get Gets a pipeline's current settings and status.
list Lists all pipelines and their statuses.
reset [Deprecated] Use the "start --full-refresh" command instead. Resets
a pipeline so that data can be reprocessed from the beginning.
run [Deprecated] Use the "start" command instead. Starts a pipeline
update.
start Starts a pipeline update.
stop Stops the pipeline by cancelling any active update.
Criar um pipeline
Para exibir a documentação de uso, execute databricks pipelines create --help.
Uso
databricks pipelines create --settings create-pipeline.json
create-pipeline.json:
{
"name": "Wikipedia pipeline (SQL)",
"storage": "/Users/username/data",
"clusters": [
{
"label": "default",
"autoscale": {
"min_workers": 1,
"max_workers": 5
}
}
],
"libraries": [
{
"notebook": {
"path": "/Users/username/DLT quickstart (SQL)"
}
}
],
"target": "wikipedia_analysis",
"continuous": false
}
Successfully created pipeline: https://<databricks-instance>/#joblist/pipelines/4ee6b5c2-7017-11ec-90d6-0242ac120003 with ID: 4ee6b5c2-7017-11ec-90d6-0242ac120003.
Editar um pipeline
Para exibir a documentação de uso, execute databricks pipelines edit --help.
Uso
databricks pipelines edit --settings edit-pipeline.json
edit-pipeline.json:
{
"id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
"name": "Wikipedia pipeline (SQL)",
"storage": "/Users/username/data",
"clusters": [
{
"label": "default",
"autoscale": {
"min_workers": 1,
"max_workers": 3
}
}
],
"libraries": [
{
"notebook": {
"path": "/Users/username/DLT quickstart (SQL)"
}
}
],
"target": "wikipedia_quickstart_data",
"continuous": false
}
Successfully edited pipeline settings: https://<databricks-instance>/#joblist/pipelines/4ee6b5c2-7017-11ec-90d6-0242ac120003.
Excluir um pipeline
Para exibir a documentação de uso, execute databricks pipelines delete --help.
Uso
databricks pipelines delete --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003 deleted
Obter informações sobre um pipeline
Para exibir a documentação de uso, execute databricks pipelines get --help.
Uso
databricks pipelines get --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
{
"pipeline_id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
"spec": {
"id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
"name": "Wikipedia pipeline (SQL)",
"storage": "/Users/username/data",
"clusters": [
{
"label": "default",
"autoscale": {
"min_workers": 1,
"max_workers": 5
}
}
],
"libraries": [
{
"notebook": {
"path": "/Users/username/DLT quickstart (SQL)"
}
}
],
"target": "wikipedia_analysis",
"continuous": false
},
"state": "IDLE",
"name": "Wikipedia pipeline (SQL)",
"creator_user_name": "username@databricks.com",
"last_modified": 1641600056265
}
Listar todos os pipelines e informações sobre seu status
Para exibir a documentação de uso, execute databricks pipelines list --help.
Uso
databricks pipelines list
[
{
"pipeline_id": "8a93e1b4-7024-11ec-90d6-0242ac120003",
"state": "IDLE",
"name": "Retail sales pipeline (SQL)",
"latest_updates": [
{
"update_id": "9df03924-7024-11ec-90d6-0242ac120003",
"state": "COMPLETED",
"creation_time": "2021-10-29T16:18:57.982Z"
},
{
"update_id": "a57e601c-7024-11ec-90d6-0242ac120003",
"state": "COMPLETED",
"creation_time": "2021-10-28T18:19:30.371Z"
}
],
"creator_user_name": "user@databricks.com"
},
{
"pipeline_id": "b46e2670-7024-11ec-90d6-0242ac120003",
"state": "IDLE",
"name": ":re[LDP] quickstart (Python)",
"latest_updates": [
{
"update_id": "bcd8fa2e-7024-11ec-90d6-0242ac120003",
"state": "COMPLETED",
"creation_time": "2021-12-16T18:19:25.827Z"
},
{
"update_id": "c2c7a2c8-7024-11ec-90d6-0242ac120003",
"state": "COMPLETED",
"creation_time": "2021-10-29T22:22:32.586Z"
},
{
"update_id": "c88d2818-7024-11ec-90d6-0242ac120003",
"state": "COMPLETED",
"creation_time": "2021-10-29T22:16:18.258Z"
}
],
"creator_user_name": "user@databricks.com"
}
]
O senhor pode começar a pipeline update
Para exibir a documentação de uso, execute databricks pipelines start --help.
databricks pipelines start --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Started an update c6462e2e-beb7-4c24-a9d3-962eaa19781b for pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.
começar a pipeline update with full refresh
Para exibir a documentação de uso, execute databricks pipelines start --help.
Uso
databricks pipelines start --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003 --full-refresh
Started an update ed885d3d-4fc0-44fd-b5dd-59471c789327 with full refresh for pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.
Interromper uma atualização do pipeline
Para exibir a documentação de uso, execute databricks pipelines stop --help.
Uso
databricks pipelines stop --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Stopped pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.