Why use AI Runtime instead of classic compute
AI Runtime is serverless GPU compute for training, fine-tuning, and running batch inference on custom deep learning models. If you run these workloads on classic compute with Databricks Runtime ML, Databricks recommends moving them to AI Runtime for lower total cost of ownership, a more flexible developer experience, and multinode scale. The following sections compare the two for ML engineers and platform teams, and describe how Genie Code can help you migrate.
Compare AI Runtime with classic compute
The following table compares AI Runtime with Databricks Runtime ML on classic compute:
Capability | AI Runtime | Classic compute with Databricks Runtime ML |
|---|---|---|
Total cost of ownership (TCO) | Lower TCO through managed serverless GPU infrastructure, high-throughput performance, optimized data loading, and Spark access through Spark Connect. You pay only for the compute your workloads use. Databricks manages clusters, drivers, and GPU health for you. | Higher TCO from cluster management overhead, slow startup, and unoptimized data loading and checkpointing. You manage cluster configuration and GPU health, which raises the cost to acquire and maintain GPU clusters. |
Developer experience | Develop and deploy from the Databricks CLI, IDEs, and notebooks. Native Ray support. Choose a minimal Standard environment for full control over dependencies, the Databricks AI environment with common AI and ML packages, or a custom Docker image. | Develop in notebooks or connect from an IDE over the SSH tunnel. No dedicated CLI for GPU training workloads. Ray runs on Spark. A large set of preinstalled libraries tied to the runtime version, including packages unrelated to AI and ML. |
Scale and performance | Multinode training and batch inference. Optimized data loading and checkpointing. High GPU reliability for long training runs. | Not optimized for multinode training because you configure nodes and networking yourself. You optimize data loading and checkpointing and validate GPU health manually. |
Total cost of ownership
AI Runtime reduces total cost of ownership with managed, high-performance serverless GPUs:
- Faster startup: Serverless GPU infrastructure is available on demand, so you don't wait for a GPU cluster to start. Classic GPU clusters can take several minutes or longer to start, depending on cluster size and GPU availability.
- Pay-as-you-go pricing: You pay only for the serverless GPU compute your workloads use, without managing or paying for idle clusters.
- Higher GPU utilization: Flexible compute options, fast data loading, efficient checkpointing, and multinode training reduce idle GPU time.
- Reduced administrative burden: AI Runtime handles compute deployment, cluster networking, GPU health management, and CUDA driver upgrades. With classic compute, your team handles node administration, cluster configuration, and ongoing infrastructure maintenance.
Developer experience
AI Runtime offers a flexible developer experience for AI and deep learning workloads:
- Connectivity: Work in notebooks, connect from an IDE over an SSH tunnel (Beta), or submit workloads with the Databricks CLI. To automate runs, schedule jobs or use Declarative Automation Bundles. Classic compute has no dedicated CLI for submitting GPU training workloads.
- Dedicated deep learning runtime: AI Runtime natively supports frameworks such as Ray and PyTorch. Choose the Standard environment for full control over your dependency stack, the Databricks AI environment with common AI and ML packages, or a custom Docker image. Classic runtimes include a large set of preinstalled libraries tied to the runtime version.
- Debugging and observability: Track experiments, monitor compute utilization, manage models, and troubleshoot issues with the built-in MLflow integration. With classic compute, logs and compute utilization metrics are not automatically stored in the MLflow run.
- Secure data access: Access structured and unstructured data governed by Unity Catalog through Spark Connect, without running an Apache Spark cluster.
Scale and performance
AI Runtime helps you scale deep learning workloads efficiently:
- Optimized data loading and checkpointing: High-throughput data loading and asynchronous distributed checkpointing accelerate training pipelines and minimize idle GPU time. Classic compute lacks native high-throughput data loaders and distributed checkpointing. See Improve training performance and resiliency on AI Runtime.
- Built-in GPU resiliency: Continuous automated health checks help detect and isolate failing GPU nodes, which reduces disruptions to long training runs. With classic compute, you maintain GPU node health yourself, and isolating a bad node requires interrupting active workloads. See How we keep GPUs reliable across Databricks AI from the Databricks engineering blog.
- Multinode training: Scale from a single GPU to multinode distributed training with high-speed networking and automatic environment configuration. For the accelerators that support multinode training, see hardware options.
- Custom batch inference: Run large-scale custom batch inference across distributed GPUs without managing GPU clusters.
Migrate from classic compute with Genie Code
Genie Code can help automate code refactoring and environment setup in notebooks connected to AI Runtime when you move a workload from classic compute. For the full list of migration steps, see Migrate classic GPU workloads to serverless.
What Genie Code automates
Genie Code can help with the following migration tasks:
- Refactoring: Converts Databricks Runtime ML notebooks, such as notebooks that use
TorchDistributor, to AI Runtime notebooks that use the@distributeddecorator fromdatabricks.air. - Dependency resolution: Evaluates notebook imports, identifies missing or incompatible packages, and configures a serverless GPU environment.
- Multi-rank error diagnosis: Analyzes output logs across all nodes to isolate root causes such as GPU out-of-memory (OOM) errors or communication hangs.
Example Genie Code migration prompts
The following are example prompts you can give Genie Code:
- Code conversion: "Migrate this notebook from TorchDistributor to the databricks.air @distributed decorator."
- Environment setup: "Inspect imported packages, recommend either the Databricks AI or Standard environment, and generate the required %pip install commands."
- Debugging: "Analyze the distributed training logs across all ranks to determine why the job failed."