MLflow is an open source platform for managing the end-to-end machine learning lifecycle. It has three primary components: Tracking, Models, and Projects:
- Tracking: Allows you to track experiments to record and compare parameters and results.
- Models: Allow you to manage and deploy models from a variety of ML libraries to a variety of model serving and inference platforms.
- Projects: Allow you to package ML code in a reusable, reproducible form to share with other data scientists or transfer to production.
Databricks provides a fully managed and hosted version of MLflow integrated with enterprise security features, high availability, and other Databricks workspace features such as experiment and run management and notebook revision capture. MLflow on Databricks offers an integrated experience for tracking and securing machine learning model training runs and running machine learning projects.
If you’re just getting started with Databricks, consider using MLflow on Databricks Community Edition, which provides a simple managed MLflow experience for lightweight experimentation. Remote execution of MLflow projects is not supported on Community Edition. We plan to impose moderate limits on the number of experiments and runs. For the initial launch of MLflow on CE no limits are imposed.
The first article provides a quick start that demonstrates the basic MLflow tracking APIs. The subsequent articles introduce each MLflow component and describe how these components are hosted within Databricks. They include numerous Databricks notebooks and examples that illustrate how to use each MLflow component.