Optimizations

This section contains information on optimizations available with Delta Lake on Databricks.

• Optimizing Performance with File Management
  • Compaction (bin-packing)
  • Data skipping
  • Z-Ordering (multi-dimensional clustering)
  • Garbage collection
  • Improving performance for interactive queries
  • Frequently asked questions (FAQ)
• Auto Optimize
  • How Auto Optimize works
  • Requirements
  • Usage
  • When to opt-in and opt-out
  • Example workflow: Streaming ingest with concurrent deletes or updates
  • Frequently asked questions (FAQ)
• Optimizing Performance with Caching
  • Delta and Apache Spark caching comparison
  • Delta cache consistency
  • Use Delta caching
  • Cache a subset of the data
  • Monitor the Delta cache
  • Configure the Delta cache
• Isolation Levels
  • Set the isolation level
  • Write conflicts
  • Avoid conflicts using partitioning and disjoint command conditions
  • Conflict exceptions
• Replicating MySQL Tables to Delta Lake Tables
  • MySQL configuration
  • Set up a Structured Streaming job in Databricks
  • Advanced configuration
  • Limitations
• Optimize Join Performance
  • Range Join Optimization
  • Skew Join Optimization
• Optimized Data Transformation
  • Higher-Order Functions
  • Transform Complex Data Types
• Table Versioning