Create pipelines with sdp-meta
The sdp-meta project from Databricks Labs provides tools to generate pipelines from metadata that you maintain.
The open source sdp-meta project, like all projects in the databrickslabs GitHub account, exists for exploration purposes only. Databricks does not support it or provide service-level agreements (SLAs) for it. Do not submit Databricks support tickets for issues related to this project. Instead, file a GitHub issue, which is reviewed as time permits.
What is sdp-meta?
Lakeflow pipelines allow you to declaratively specify a table, and generate a flow in a pipeline that both creates the table and keeps it up to date as the source data changes. However, if your organization has hundreds of tables, generating and managing these pipelines is time consuming, and can lead to inconsistent practices.
The sdp-meta project is a metadata-driven metaprogramming framework designed to work with Lakeflow pipelines. This framework enables the automation of bronze and silver data pipelines by leveraging metadata recorded in a set of JSON and YAML files. At runtime, generic pipelines read your metadata and dynamically build the flows described in it. Bronze and silver processing can run in separate pipelines or together in a combined pipeline. You generate the metadata about your pipelines, and sdp-meta generates your pipelines.
With your logic centralized in one place (the metadata), your system is faster, reusable, and easier to maintain.
The sdp-meta project was formerly named dlt-meta, after the older Delta Live Tables feature in Databricks. Delta Live Tables has been replaced by Lakeflow pipelines, and sdp-meta works with Lakeflow pipelines. If you are upgrading an existing dlt-meta installation, see the migration guide in the sdp-meta documentation.
Benefits of sdp-meta
There are two main use cases for sdp-meta:
- Ingest and clean a large number of tables simply.
- Enforce data engineering standards across multiple pipelines and users.
The benefits of using a metadata-driven approach include:
- Maintaining metadata can be done without knowledge of Python or SQL code.
- Maintaining metadata, rather than the code, requires less overhead, and reduces errors.
- The code is generated by sdp-meta, so it stays consistent and has less custom code across pipelines and published tables.
- You can easily group tables into pipelines within the metadata, generating the number of pipelines needed to most efficiently update your data.
How sdp-meta works
The following image shows an overview of the sdp-meta system:
- You create the metadata files as input to sdp-meta, to specify your source files and outputs, quality rules, and required processing. These onboarding files can be written in JSON or YAML.
- You run the sdp-meta onboarding job. The engine compiles the onboarding files into a data flow specification, called DataflowSpec, and stores it in Delta tables (
bronze_dataflowspecandsilver_dataflowspec) for later use. - At runtime, a generic pipeline reads the DataflowSpec and dynamically builds the bronze processing graph. It reads your source data (files, streams, or CDC) and applies the correct data expectations to match your quality rules, generating your bronze tables and quarantining records that fail those rules.
- Silver processing can run in a separate generic pipeline, which is the default for Declarative Automation Bundles, or in the same pipeline when you use combined mode. The pipeline uses the DataflowSpec to apply the appropriate transformations and other processing, generating silver tables that are cleaned, enriched, and analytics-ready.
You run the pipelines generated by sdp-meta to keep the output current as your source data is updated.
Get started
To use sdp-meta, you must:
- Deploy and configure the sdp-meta solution.
- Prepare the metadata for your bronze and silver layer tables.
- Create a job to onboard the metadata.
- Use the metadata to create pipelines for your tables.
The sdp-meta project supports several deployment interfaces: bundles (recommended), an interactive CLI, a Databricks App with a browser-based GUI, an MCP server for AI-assisted setup, and a portable Agent Skill for skill-aware AI agents.
The sdp-meta documentation on GitHub has a tutorial to help you get started with this process. For more information, see getting started with sdp-meta on GitHub.