Delta Engine

Delta Engine is a high performance, Apache Spark compatible query engine that provides an efficient way to process data in data lakes including data stored in open source Delta Lake. Delta Engine optimizations accelerate data lake operations, supporting a variety of workloads ranging from large-scale ETL processing to ad-hoc, interactive queries. Many of these optimizations take place automatically; you get the benefits of these Delta Engine capabilities just by using Databricks for your data lakes.

• Optimize performance with file management
  • Compaction (bin-packing)
  • Data skipping
  • Z-Ordering (multi-dimensional clustering)
  • Tune file size
  • Notebooks
  • Improve interactive query performance
  • Frequently asked questions (FAQ)
• Auto Optimize
  • How Auto Optimize works
  • Enable Auto Optimize
  • When to opt in and opt out
  • Example workflow: Streaming ingest with concurrent deletes or updates
  • Frequently asked questions (FAQ)
• Optimize performance with caching
  • Delta and Apache Spark caching
  • Delta cache consistency
  • Use Delta caching
  • Cache a subset of the data
  • Monitor the Delta cache
  • Configure the Delta cache
• Dynamic file pruning
• Isolation levels
  • Set the isolation level
• Bloom filter indexes
  • Configuration
  • Create a Bloom filter index
  • Drop a Bloom filter index
  • Display the list of Bloom filter indexes
  • Notebook
• Optimize join performance
  • Range join optimization
  • Skew join optimization
• Optimized data transformation
  • Higher-order functions
  • Transform complex data types

Additional resources

• Cost-based optimizer