Delta Live Tables CLI (legacy)

October 09, 2024

Important

This documentation has been retired and might not be updated.

This information applies to legacy Databricks CLI versions 0.18 and below. Databricks recommends that you use newer Databricks CLI version 0.205 or above instead. See What is the Databricks CLI?. To find your version of the Databricks CLI, run databricks -v.

To migrate from Databricks CLI version 0.18 or below to Databricks CLI version 0.205 or above, see Databricks CLI migration.

You run Databricks Delta Live Tables CLI subcommands by appending them to databricks pipelines. These subcommands call the Delta Live Tables API.

Bash
databricks pipelines -h

Usage: databricks pipelines [OPTIONS] COMMAND [ARGS]...

  Utility to interact with Databricks Delta Live Tables Pipelines.

Options:
  -v, --version   0.18.0
  --debug         Debug Mode. Shows full stack trace on error.
  --profile TEXT  CLI connection profile to use. The default profile is
                  "DEFAULT".

  -h, --help      Show this message and exit.

Commands:
  create  Creates a pipeline.
  delete  Deletes the pipeline and cancels any active updates.
  deploy  [Deprecated] This command is deprecated, use create and edit
          commands instead.  Creates or edits a pipeline specified by the
          pipeline settings.

  edit    Edits a pipeline.
  get     Gets a pipeline's current settings and status.
  list    Lists all pipelines and their statuses.
  reset   [Deprecated] Use the "start --full-refresh" command instead. Resets
          a pipeline so that data can be reprocessed from the beginning.

  run     [Deprecated] Use the "start" command instead. Starts a pipeline
          update.

  start   Starts a pipeline update.
  stop    Stops the pipeline by cancelling any active update.
この記事の内容:

Create a pipeline

To display usage documentation, run databricks pipelines create --help.

Usage

Bash
databricks pipelines create --settings create-pipeline.json

create-pipeline.json:

JSON
{
    "name": "Wikipedia pipeline (SQL)",
    "storage": "/Users/username/data",
    "clusters": [
      {
        "label": "default",
        "autoscale": {
          "min_workers": 1,
          "max_workers": 5
        }
      }
    ],
    "libraries": [
      {
        "notebook": {
          "path": "/Users/username/Delta Live Tables quickstart (SQL)"
        }
      }
    ],
    "target": "wikipedia_analysis",
    "continuous": false
}
Console
Successfully created pipeline: https://<databricks-instance>/#joblist/pipelines/4ee6b5c2-7017-11ec-90d6-0242ac120003 with ID: 4ee6b5c2-7017-11ec-90d6-0242ac120003.

Edit a pipeline

To display usage documentation, run databricks pipelines edit --help.

Usage

Bash
databricks pipelines edit --settings edit-pipeline.json

edit-pipeline.json:

JSON
{
    "id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
    "name": "Wikipedia pipeline (SQL)",
    "storage": "/Users/username/data",
    "clusters": [
        {
            "label": "default",
            "autoscale": {
                "min_workers": 1,
                "max_workers": 3
            }
        }
    ],
    "libraries": [
        {
            "notebook": {
                "path": "/Users/username/Delta Live Tables quickstart (SQL)"
            }
        }
    ],
    "target": "wikipedia_quickstart_data",
    "continuous": false
}
Console
Successfully edited pipeline settings: https://<databricks-instance>/#joblist/pipelines/4ee6b5c2-7017-11ec-90d6-0242ac120003.

Delete a pipeline

To display usage documentation, run databricks pipelines delete --help.

Usage

Bash
databricks pipelines delete --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Console
Pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003 deleted

Get information about a pipeline

To display usage documentation, run databricks pipelines get --help.

Usage

Bash
databricks pipelines get --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
JSON
{
  "pipeline_id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
  "spec": {
    "id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
    "name": "Wikipedia pipeline (SQL)",
    "storage": "/Users/username/data",
    "clusters": [
      {
        "label": "default",
        "autoscale": {
          "min_workers": 1,
          "max_workers": 5
        }
      }
    ],
    "libraries": [
      {
        "notebook": {
          "path": "/Users/username/Delta Live Tables quickstart (SQL)"
        }
      }
    ],
    "target": "wikipedia_analysis",
    "continuous": false
  },
  "state": "IDLE",
  "name": "Wikipedia pipeline (SQL)",
  "creator_user_name": "username@databricks.com",
  "last_modified": 1641600056265
}

List all pipelines and information on their status

To display usage documentation, run databricks pipelines list --help.

Usage

Bash
databricks pipelines list
JSON
[
  {
    "pipeline_id": "8a93e1b4-7024-11ec-90d6-0242ac120003",
    "state": "IDLE",
    "name": "Retail sales pipeline (SQL)",
    "latest_updates": [
      {
        "update_id": "9df03924-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-29T16:18:57.982Z"
      },
      {
        "update_id": "a57e601c-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-28T18:19:30.371Z"
      }
    ],
    "creator_user_name": "user@databricks.com"
  },
  {
    "pipeline_id": "b46e2670-7024-11ec-90d6-0242ac120003",
    "state": "IDLE",
    "name": "DLT quickstart (Python)",
    "latest_updates": [
      {
        "update_id": "bcd8fa2e-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-12-16T18:19:25.827Z"
      },
      {
        "update_id": "c2c7a2c8-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-29T22:22:32.586Z"
      },
      {
        "update_id": "c88d2818-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-29T22:16:18.258Z"
      }
    ],
    "creator_user_name": "user@databricks.com"
  }
]

Start a pipeline update

To display usage documentation, run databricks pipelines start --help.

Bash
databricks pipelines start --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Console
Started an update c6462e2e-beb7-4c24-a9d3-962eaa19781b for pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.

Start a pipeline update with full refresh

To display usage documentation, run databricks pipelines start --help.

Usage

Bash
databricks pipelines start --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003 --full-refresh
Console
Started an update ed885d3d-4fc0-44fd-b5dd-59471c789327 with full refresh for pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.

Stop a pipeline update

To display usage documentation, run databricks pipelines stop --help.

Usage

Bash
databricks pipelines stop --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Console
Stopped pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.