Optimizations

Databricks provides optimizations for Delta Lake that accelerate data lake operations, supporting a variety of workloads ranging from large-scale ETL processing to ad-hoc, interactive queries. Many of these optimizations take place automatically; you get their benefits simply by using Databricks for your data lakes.

• Optimize performance with file management
  • Compaction (bin-packing)
  • Data skipping
  • Z-Ordering (multi-dimensional clustering)
  • Tune file size
  • Notebooks
  • Improve interactive query performance
  • Frequently asked questions (FAQ)
• Auto Optimize
  • How Auto Optimize works
  • Enable Auto Optimize
  • When to opt in and opt out
  • Example workflow: Streaming ingest with concurrent deletes or updates
  • Frequently asked questions (FAQ)
• Optimize performance with caching
  • Delta and Apache Spark caching
  • Delta cache consistency
  • Use Delta caching
  • Cache a subset of the data
  • Monitor the Delta cache
  • Configure the Delta cache
• Dynamic file pruning
• Isolation levels
  • Set the isolation level
• Bloom filter indexes
  • How Bloom filter indexes work
  • Configuration
  • Create a Bloom filter index
  • Drop a Bloom filter index
  • Display the list of Bloom filter indexes
  • Notebook
• Low Shuffle Merge
  • Optimized performance
  • Optimized data layout
  • Availability
• Optimize join performance
  • Range join optimization
  • Skew join optimization
• Optimized data transformation
  • Higher-order functions
  • Transform complex data types

Additional resources

• Cost-based optimizer