Skip to main content

Install a sample ETL pipeline with dbdemos

Install a complete extract, transform, and load (ETL) pipeline, built with Lakeflow pipelines, in your workspace, and explore how it ingests, cleans, and aggregates data before you build your own. This page uses the pipeline-bike demo from dbdemos, a Python library from Databricks that installs demos into your workspace. The demo processes bike rental data through bronze, silver, and gold layers, and installs the following resources:

  • A pipeline named dbdemos_pipeline_bike_<catalog>_<schema>, with streaming tables that ingest raw files from a volume, data quality expectations that drop or flag invalid records, and materialized views that aggregate the cleaned data.
  • Source files that define the pipeline in both SQL and Python, and notebooks that explore the data and the pipeline event log. The pipeline runs the SQL source files.
  • A job that generates the sample data and then runs the pipeline.
  • AI/BI dashboards for business insights, operational insights, and data quality monitoring.

See Databricks Demo Center.

Requirements​

Confirm the following before you install the demo:

  • A workspace that is enabled for Unity Catalog.
  • The USE CATALOG and CREATE SCHEMA privileges on the catalog where you install the demo. If the schema already exists, you must have the USE SCHEMA, CREATE TABLE, CREATE MATERIALIZED VIEW, and CREATE VOLUME privileges on it instead.
  • Compute that can install Python packages from the Python Package Index (PyPI). The installer detects whether the notebook runs on serverless compute, and creates the demo's resources to match. To use serverless compute for the demo, attach the notebook to serverless compute.

Install the demo​

A notebook runs the installer, which creates the demo's pipeline, job, and dashboards.

Install the dbdemos library​

Install the library in a notebook, and then restart Python so that the notebook can import it:

  1. In your workspace, create a Python notebook, and attach it to compute.

  2. Run the following code in the first cell:

    Python
    %pip install dbdemos
  3. Run the following code in the next cell to restart Python so that the notebook can import the library:

    Python
    dbutils.library.restartPython()

Install the pipeline demo​

Run the following code in a new cell. Replace <catalog> and <schema> with the catalog and schema where you want the demo to create its tables. If the schema doesn't exist, the installer creates it.

Python
import dbdemos

dbdemos.install('pipeline-bike', catalog='<catalog>', schema='<schema>')

The installer creates the demo resources in a pipeline-bike folder next to your notebook, and then starts the job that generates the sample data and runs the pipeline. When the installation completes, the cell output lists links to the notebooks, the pipeline, the job, and the dashboards.

note

The first pipeline run starts after the data generation finishes, so the tables and dashboards can take several minutes to populate.

If the installation fails, see Troubleshooting.

Explore the pipeline​

Open the installed resources to see how the pipeline moves data through each layer:

  1. In your workspace, click Workflows icon. Jobs & Pipelines, and confirm that the dbdemos_job_bike_init_<user> job run succeeds.
  2. Click the dbdemos_pipeline_bike_<catalog>_<schema> pipeline. The pipeline graph shows the bronze streaming tables that ingest the raw files, the silver streaming tables that clean them, and the gold materialized views that aggregate them.
  3. Click a silver table, such as rides, to view the number of records that its expectations dropped.
  4. In the pipeline-bike folder, open 1-sdp-sql/transformations, and read the 01-bronze.sql, 02-silver.sql, and 03-gold.sql source files to see how each layer is defined.
  5. Open the dashboards from the links in the installer output to view the business metrics and data quality results.

Troubleshooting​

Four errors can stop the demo installation. Each entry gives the fix.

ModuleNotFoundError: No module named 'dbdemos'

The notebook imported dbdemos before the library was installed, or before Python restarted. Run %pip install dbdemos, run dbutils.library.restartPython(), and then run the installation cell again.

The installer fails because the folder already exists

A previous installation created the pipeline-bike folder. Replacing the folder deletes its contents. To replace it, run dbdemos.install('pipeline-bike', catalog='<catalog>', schema='<schema>', overwrite=True). To keep the existing folder, run the notebook from a different folder instead.

The installer can't create the schema or tables

You don't have the required privileges on the catalog or schema. Choose a catalog and schema where you have the privileges in Requirements, or ask the catalog owner or a metastore admin to grant them.

%pip install fails

The compute can't reach PyPI. Use compute that has outbound access to PyPI, or ask your workspace admin to allow it.

Next steps​