Data engineering with Databricks

Databricks provides a rich set of data engineering features that include a robust environment for collaboration among data scientists, engineers, and analysts. Through the center of it are the pipelines that ingest and transform the data. To enable easy engineering of the most efficient pipelines, Databricks offers Structured Streaming and Delta Live Tables. See Streaming on Databricks and What is Delta Live Tables?.

For an overview of why incremental stream processing of data provided by Structured Streaming and Delta Live Tables is the best option for data pipelines, see Why incremental stream processing?.

Additional data engineering resources

Databricks provides features for data engineering tasks for many additional scenarios:

• Databricks notebooks are a popular tool for collaboration.

• Schedule and orchestrate your workflows using Databricks Jobs.

• See Databricks Mosaic AI if you are architecting machine learning solutions.

Note

If you primarily work with SQL queries and BI tools, see Databricks SQL.

For getting started tutorials and introductory informaton, see Get started with Databricks and What is Databricks?.

• Delta Live Tables

  Learn how to build data pipelines for ingestion and transformation with Databricks Delta Live Tables.

• Structured Streaming

  Learn about streaming, incremental, and real-time workloads powered by Structured Streaming on Databricks.