MLOps Stacks automates the creation of infrastructure for an ML project workflow. It sets up the elements required to implement and operate ML for continuous deployment across development, staging, and production environments, including:
ML pipelines for model training, deployment, and inference.
Release pipeline for production.
MLOps Stacks is fully integrated into the Databricks CLI and Databricks Asset Bundles, providing a single toolchain for developing, testing, and deploying both data and ML assets on Databricks. The environment created by MLOps Stacks implements the MLOps workflow recommended by Databricks. You can customize the code to create stacks to match your organization’s processes or requirements.
This article explains how MLOps Stacks works and describes the structure of the project created by MLOps.
A “stack” refers to the set of tools used in a development process. The default MLOps Stack takes advantage of the unified Databricks platform and uses the following tools:
You use the Databricks CLI to create an MLOps Stack. For step-by-step instructions, see Databricks Asset Bundles for MLOps Stacks.
When you initiate an MLOps Stacks project, the software steps you through entering the configuration details and then creates a directory containing the files that compose your project. This directory, or stack, implements the production MLOps workflow recommended by Databricks. The components shown in the diagram are created for you, and you need only edit the files to add your custom code.
Your organization can use the default stack, or customize it as needed to add, remove, or revise components to fit your organization’s practices. See the GitHub repository readme for details.
MLOps Stacks is designed with a modular structure to allow the different ML teams to work independently on a project while following software engineering best practices and maintaining production-grade CI/CD. Production engineers configure ML infrastructure that allows data scientists to develop, test, and deploy ML pipelines and models to production.
As shown in the green boxes in the diagram, the three Databricks code components of an MLOps Stack are the following:
ML code. A data scientist can create code in a Databricks notebook or in a local IDE. You can use GitHub or Azure DevOps for source control. As shown in the diagram, when the project is created, it is in a runnable state with example code. You edit or replace this code with your own code.
Resource configurations. These .yml files define individual workflows that comprise the project, such as training and batch inference jobs. They are configured and deployed using Databricks CLI bundles. By defining these resources in .yml files, you can govern, audit, and deploy changes using pull requests instead of untrackable changes made using the UI.
CI/CD workflows. Implemented using GitHub Actions or Azure DevOps in conjunction with Databricks Workflows, these workflows test and deploy the ML code (for model training, batch inference, and so on) and the Databricks ML resource configurations across your development, staging, and production workspaces. As with the resource files, these workflows automate all production changes and ensure that only tested code is deployed to production.
An MLOps Stack uses Databricks Asset Bundles – a collection of source files that serves as the end-to-end definition of a project. These source files include information about how they are to be tested and deployed. Collecting the files as a bundle makes it easy to co-version changes and use software engineering best practices such as source control, code review, testing, and CI/CD.
The diagram shows the files created for the default MLOps Stack. For details about the files included in the stack, see the documentation on the GitHub repository or Databricks Asset Bundles for MLOps Stacks.
To get started, see Databricks Asset Bundles for MLOps Stacks.