Machine learning on Databricks

Build, deploy, and manage machine learning applications on Databricks. The integrated platform unifies the entire ML lifecycle from data preparation to production monitoring.

Looking for generative AI and AI agents? See Build AI agents on Databricks.

Get started

Try a quickstart, prepare your data, or build a low-code model.

• • Get started: Build your first machine learning model on Databricks
  • Build a simple classification model with scikit-learn end-to-end.
• • AutoML
  • Automatically build high-quality models with minimal code using automated feature engineering and hyperparameter tuning.
• • Load data for machine learning and deep learning
  • Load and prepare data for ML and deep learning workflows.
• • Train recommender models
  • Train a recommender model with the two-tower or DLRM architecture.

Train classic machine learning models

Create machine learning models with automated tools and collaborative development environments.

• • Databricks Runtime for ML
  • Pre-configured clusters with scikit-learn, XGBoost, MLflow, and other ML libraries, plus support for deep learning frameworks.
• • MLflow tracking
  • Track experiments, compare model performance, and manage the complete model development lifecycle.
• • Feature engineering
  • Create, manage, and serve features with automated data pipelines and feature discovery.
• • Databricks notebooks
  • Collaborative development environment with support for Python, R, Scala, and SQL for ML workflows.

Train deep learning models

Use managed compute and built-in frameworks to develop deep learning models.

• • Distributed training
  • Examples of distributed deep learning using Ray, TorchDistributor, and DeepSpeed.
• • AI Runtime
  • Serverless GPU compute for custom deep learning training and inference workloads.
• • DL best practices
  • Guidance for framework choice, data loading, distributed scaling, and managing the deep learning model lifecycle.
• • PyTorch
  • Single-node and distributed training using PyTorch.

Deploy and serve models

Deploy models to production with scalable endpoints, real-time inference, and enterprise-grade monitoring.

• • Model Serving
  • Deploy custom models and LLMs as scalable REST endpoints with automatic scaling and GPU support.
• • AI Gateway
  • Govern and monitor access to models served on Databricks with usage tracking, payload logging, and security controls.
• • External models
  • Integrate third-party models hosted outside Databricks with unified governance and monitoring.
• • Foundation model APIs
  • Access and query state-of-the-art open models hosted by Databricks.

Monitor and govern ML systems

Ensure model quality, data integrity, and compliance with comprehensive monitoring and governance tools.

• • Unity Catalog
  • Govern data, features, models, and functions with unified access control, lineage tracking, and discovery.
• • Data profiling
  • Monitor data quality, model performance, and prediction drift with automated alerts and root cause analysis.
• • Anomaly detection
  • Monitor data freshness and completeness at the catalog level.
• • MLflow for Models
  • Track experiments, manage models in Unity Catalog, deploy, and evaluate machine learning models throughout the development lifecycle.

Productionize ML workflows

Scale machine learning operations with automated workflows, CI/CD integration, and production-ready pipelines.

• • Models in Unity Catalog
  • Use the model registry in Unity Catalog for centralized governance and to manage the model lifecycle, including deployments.
• • Lakeflow Jobs
  • Build automated workflows and production-ready ETL pipelines for ML data processing.
• • Ray on Databricks
  • Scale ML workloads with distributed computing for large-scale model training and inference.
• • MLOps workflows
  • Implement end-to-end MLOps with automated training, testing, and deployment pipelines.