Guides for AI Runtime

Migrate existing workloads, improve training performance, track GPU usage, and troubleshoot common errors.

• • Why use AI Runtime
  • Compare AI Runtime with Databricks Runtime ML classic compute for cost, developer experience, and scale.
• • Migrate classic GPU workloads
  • Move an existing deep learning workload from a classic Databricks cluster to AI Runtime.
• • Migrate from Slurm
  • Map your Slurm batch scripts, distributed launchers, and shared storage onto the Databricks CLI.
• • Optimize your workload performance
  • Improve training throughput and make long-running jobs more resilient to failures.

Track usage and costs

You can monitor your AI Runtime GPU spend by querying the billable usage system table (system.billing.usage). The following query returns total usage for serverless GPU workloads:

SQL

SELECT
  SUM(usage_quantity)
FROM
  system.billing.usage
WHERE
  product_features.serverless_gpu IS NOT NULL

For more information about the billable usage table schema, see Billable usage system table reference.

For current GPU pricing, see AI Runtime pricing.

Examples

Explore notebook tutorials and runnable Databricks CLI recipes organized by task. See AI Runtime examples.

Troubleshooting

Genie Code can help resolve environment and dependency issues and investigate GPU and distributed workload failures in notebooks connected to AI Runtime. See Use Genie Code with AI Runtime.

To debug interactively on the compute, use the web terminal to run shell commands, inspect GPU usage with nvidia-smi, and manage files. The web terminal is available when connected to AI Runtime environment version 5 or above. See Run shell commands in Databricks web terminal.

ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

The error typically arises when there is a mismatch in the NumPy versions used during the compilation of a dependent package and the NumPy version currently installed in the runtime environment. This incompatibility often occurs due to changes in NumPy's C API and is particularly noticeable from NumPy 1.x to 2.x. This error indicates that the Python package installed in the notebook may have changed the NumPy version.

Recommended solution:

Check the NumPy version in the runtime and ensure it is compatible with your packages. See the AI Runtime environment release notes for environment 6, environment 5, and environment 4 for information on preinstalled Python libraries. If you have a dependency on a different version of NumPy, add that dependency to your compute environment.

PyTorch cannot find libcudnn when installing torch

When you install a different version of torch, you might see the error: ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory. This is because torch only searches for the cuDNN library in the local path.

Recommended solution:

Reinstall the dependencies by adding --force-reinstall when installing torch:

Python

%pip install torch --force-reinstall