Delta Lake

Delta Lake is an open source storage layer that brings reliability to data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. Delta Lake runs on top of your existing data lake and is fully compatible with Apache Spark APIs.

Delta Lake on Databricks allows you to configure Delta Lake based on your workload patterns and provides optimized layouts and indexes for fast interactive queries.

This section covers Delta Lake on Databricks.

• Introduction to Delta Lake
  • Quickstart
  • Resources
• Delta Lake quickstart
  • Create a table
  • Modify a table
  • Read a table
  • Optimize a table
  • Clean up snapshots
• Introductory notebooks
  • Delta Lake Quickstart Python notebook
  • Delta Lake Quickstart Scala notebook
  • Delta Lake Quickstart SQL notebook
• Ingest data into Delta Lake
  • Partner integrations
  • COPY INTO SQL command
  • Auto Loader
• Table batch reads and writes
  • Create a table
  • Read a table
  • Write to a table
  • Schema validation
  • Update table schema
  • Replace table schema
  • Views on tables
  • Table properties
  • Table metadata
  • Notebook
• Table streaming reads and writes
  • Delta table as a stream source
  • Delta table as a sink
• Table deletes, updates, and merges
  • Delete from a table
  • Update a table
  • Upsert into a table using merge
  • Merge examples
• Table utility commands
  • Vacuum
  • History
  • Generate
  • Convert to Delta
  • Convert Delta table to a Parquet table
• Delta Lake API reference
• Concurrency control
  • Optimistic concurrency control
  • Write conflicts
  • Avoid conflicts using partitioning and disjoint command conditions
  • Conflict exceptions
• Integrations
  • Presto and Athena to Delta Lake integration
  • Redshift Spectrum to Delta Lake integration
  • Snowflake to Delta Lake integration
• Migrate workloads to Delta Lake
  • Example
• Optimizations
  • Optimize performance with file management
  • Auto Optimize
  • Optimize performance with caching
  • Isolation levels
  • Replicate MySQL tables to Delta tables
  • Optimize join performance
  • Optimized data transformation
  • Table versioning
• Best practices
  • Provide data location hints
  • Choose the right partition column
  • Compact files
• Frequently asked questions (FAQ)
  • What is Delta Lake?
  • How is Delta Lake related to Apache Spark?
  • What format does Delta Lake use to store data?
  • How can I read and write data with Delta Lake?
  • Where does Delta Lake store the data?
  • Why is Delta Lake data I deleted still stored in S3?
  • Why does a table show old data after I delete Delta Lake files with rm -rf and create a new table in the same location?
  • Can I stream data directly into and from Delta tables?
  • Does Delta Lake support writes or reads using the Spark Streaming DStream API?
  • When I use Delta Lake, will I be able to port my code to other Spark platforms easily?
  • How do Delta tables compare to Hive SerDe tables?
  • What DDL and DML features does Delta Lake not support?
  • Does Delta Lake support multi-table transactions?
  • How can I change the type of a column?
  • What does it mean that Delta Lake supports multi-cluster writes?
  • What are the limitations of multi-cluster writes?
  • Can I access Delta tables outside of Databricks Runtime?
• Resources
  • Blog posts
  • Talks
  • Examples
  • Delta Lake transaction log specification