Model training examples
This section includes examples showing how to train machine learning models on Databricks using many popular open-source libraries.
You can also use AutoML, which automatically prepares a dataset for model training, performs a set of trials using open-source libraries such as scikit-learn and XGBoost, and creates a Python notebook with the source code for each trial run so you can review, reproduce, and modify the code.
Machine learning examples
Package | Notebook(s) | Features |
|---|---|---|
scikit-learn | Unity Catalog, classification model, MLflow, automated hyperparameter tuning with Hyperopt and MLflow | |
scikit-learn | Unity Catalog, classification model, MLflow, automated hyperparameter tuning with Hyperopt and MLflow, XGBoost | |
MLlib | Binary classification, decision trees, GBT regression, Structured Streaming, custom transformer | |
xgboost | Python, PySpark, and Scala, single node workloads and distributed training |
Hyperparameter tuning examples
For general information about hyperparameter tuning in Databricks, see Hyperparameter tuning.
The open-source version of Hyperopt is no longer being maintained.
Hyperopt is not included in Databricks Runtime for Machine Learning after 16.4 LTS ML. Databricks recommends using either Optuna, which can run parallel studies on Spark, or Ray Tune for a similar experience to the deprecated Hyperopt distributed hyperparameter tuning functionality. Learn more about using Ray Tune on Databricks.
Package | Notebook | Features |
|---|---|---|
Optuna | Optuna, distributed Optuna, scikit-learn, MLflow | |
Hyperopt | Distributed hyperopt, scikit-learn, MLflow | |
Hyperopt | Use distributed hyperopt to search hyperparameter space for different model types simultaneously | |
Hyperopt | Hyperopt, MLlib | |
Hyperopt | Best practices for datasets of different sizes |