deploy-mlflow-pyfunc-model-serving(Python)

Import Notebook

Deploy an MLflow pyfunc model with Model Serving

In this notebook, learn how to deploy MLflow pyfunc to a serving endpoint. MLflow pyfunc offers greater flexibility and customization to your deployment. You can run any custom model, add preprocessing or post-processing logic, or execute any arbitrary Python code. While using the MLflow built-in flavor is recommended for optimal performance, you can use MLflow pyfunc where more customization is required. The following demonstrates how to deploy a GPT-2 model with pyfunc to a GPU endpoint, but the workflow outlined below also works with various other types of models to either CPU or GPU endpoints.

Install and import libraries

Collecting git+https://github.com/mlflow/mlflow@master Cloning https://github.com/mlflow/mlflow (to revision master) to /tmp/pip-req-build-jfuqo93o Running command git clone --filter=blob:none --quiet https://github.com/mlflow/mlflow /tmp/pip-req-build-jfuqo93o Resolved https://github.com/mlflow/mlflow to commit cead93801101ce3965f0f1c908565f72599a3a80 Running command git submodule update --init --recursive -q Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Requirement already satisfied: requests<3,>=2.17.3 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (2.28.1) Collecting docker<7,>=4.0.0 Downloading docker-6.1.3-py3-none-any.whl (148 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 148.1/148.1 kB 3.0 MB/s eta 0:00:00a 0:00:01 Requirement already satisfied: numpy<2 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (1.21.5) Collecting alembic!=1.10.0,<2 Downloading alembic-1.12.0-py3-none-any.whl (226 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 226.0/226.0 kB 6.0 MB/s eta 0:00:0000:01 Requirement already satisfied: gunicorn<22 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (20.1.0) Requirement already satisfied: Flask<3 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (1.1.2+db1) Requirement already satisfied: sqlalchemy<3,>=1.4.0 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (1.4.39) Requirement already satisfied: pyyaml<7,>=5.1 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (6.0) Requirement already satisfied: packaging<24 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (21.3) Requirement already satisfied: scipy<2 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (1.9.1) Requirement already satisfied: click<9,>=7.0 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (8.0.4) Requirement already satisfied: matplotlib<4 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (3.5.2) Requirement already satisfied: sqlparse<1,>=0.4.0 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (0.4.2) Collecting querystring-parser<2 Downloading querystring_parser-1.2.4-py2.py3-none-any.whl (7.9 kB) Requirement already satisfied: importlib-metadata!=4.7.0,<7,>=3.7.0 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (4.11.3) Requirement already satisfied: markdown<4,>=3.3 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (3.3.4) Requirement already satisfied: databricks-cli<1,>=0.8.7 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (0.17.7) Requirement already satisfied: protobuf<5,>=3.12.0 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (3.19.4) Requirement already satisfied: cloudpickle<3 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (2.0.0) Requirement already satisfied: pyarrow<13,>=4.0.0 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (8.0.0) Requirement already satisfied: scikit-learn<2 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (1.1.1) Requirement already satisfied: Jinja2<4,>=2.11 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (2.11.3) Requirement already satisfied: gitpython<4,>=2.1.0 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (3.1.27) Requirement already satisfied: pytz<2024 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (2022.1) Requirement already satisfied: entrypoints<1 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (0.4) Requirement already satisfied: pandas<3 in /databricks/python3/lib/python3.10/site-packages (from mlflow==2.6.1.dev0) (1.4.4) Requirement already satisfied: Mako in /databricks/python3/lib/python3.10/site-packages (from alembic!=1.10.0,<2->mlflow==2.6.1.dev0) (1.2.0) Requirement already satisfied: typing-extensions>=4 in /databricks/python3/lib/python3.10/site-packages (from alembic!=1.10.0,<2->mlflow==2.6.1.dev0) (4.3.0) Requirement already satisfied: pyjwt>=1.7.0 in /usr/lib/python3/dist-packages (from databricks-cli<1,>=0.8.7->mlflow==2.6.1.dev0) (2.3.0) Requirement already satisfied: tabulate>=0.7.7 in /databricks/python3/lib/python3.10/site-packages (from databricks-cli<1,>=0.8.7->mlflow==2.6.1.dev0) (0.8.10) Requirement already satisfied: six>=1.10.0 in /usr/lib/python3/dist-packages (from databricks-cli<1,>=0.8.7->mlflow==2.6.1.dev0) (1.16.0) Requirement already satisfied: oauthlib>=3.1.0 in /usr/lib/python3/dist-packages (from databricks-cli<1,>=0.8.7->mlflow==2.6.1.dev0) (3.2.0) Requirement already satisfied: urllib3<2.0.0,>=1.26.7 in /databricks/python3/lib/python3.10/site-packages (from databricks-cli<1,>=0.8.7->mlflow==2.6.1.dev0) (1.26.11) Requirement already satisfied: websocket-client>=0.32.0 in /databricks/python3/lib/python3.10/site-packages (from docker<7,>=4.0.0->mlflow==2.6.1.dev0) (0.58.0) Requirement already satisfied: Werkzeug>=0.15 in /databricks/python3/lib/python3.10/site-packages (from Flask<3->mlflow==2.6.1.dev0) (2.0.3) Requirement already satisfied: itsdangerous>=0.24 in /databricks/python3/lib/python3.10/site-packages (from Flask<3->mlflow==2.6.1.dev0) (2.0.1) Requirement already satisfied: gitdb<5,>=4.0.1 in /databricks/python3/lib/python3.10/site-packages (from gitpython<4,>=2.1.0->mlflow==2.6.1.dev0) (4.0.10) Requirement already satisfied: setuptools>=3.0 in /databricks/python3/lib/python3.10/site-packages (from gunicorn<22->mlflow==2.6.1.dev0) (63.4.1) Requirement already satisfied: zipp>=0.5 in /databricks/python3/lib/python3.10/site-packages (from importlib-metadata!=4.7.0,<7,>=3.7.0->mlflow==2.6.1.dev0) (3.8.0) Requirement already satisfied: MarkupSafe>=0.23 in /databricks/python3/lib/python3.10/site-packages (from Jinja2<4,>=2.11->mlflow==2.6.1.dev0) (2.0.1) Requirement already satisfied: pyparsing>=2.2.1 in /databricks/python3/lib/python3.10/site-packages (from matplotlib<4->mlflow==2.6.1.dev0) (3.0.9) Requirement already satisfied: cycler>=0.10 in /databricks/python3/lib/python3.10/site-packages (from matplotlib<4->mlflow==2.6.1.dev0) (0.11.0) Requirement already satisfied: kiwisolver>=1.0.1 in /databricks/python3/lib/python3.10/site-packages (from matplotlib<4->mlflow==2.6.1.dev0) (1.4.2) Requirement already satisfied: fonttools>=4.22.0 in /databricks/python3/lib/python3.10/site-packages (from matplotlib<4->mlflow==2.6.1.dev0) (4.25.0) Requirement already satisfied: pillow>=6.2.0 in /databricks/python3/lib/python3.10/site-packages (from matplotlib<4->mlflow==2.6.1.dev0) (9.2.0) Requirement already satisfied: python-dateutil>=2.7 in /databricks/python3/lib/python3.10/site-packages (from matplotlib<4->mlflow==2.6.1.dev0) (2.8.2) Requirement already satisfied: certifi>=2017.4.17 in /databricks/python3/lib/python3.10/site-packages (from requests<3,>=2.17.3->mlflow==2.6.1.dev0) (2022.9.14) Requirement already satisfied: idna<4,>=2.5 in /databricks/python3/lib/python3.10/site-packages (from requests<3,>=2.17.3->mlflow==2.6.1.dev0) (3.3) Requirement already satisfied: charset-normalizer<3,>=2 in /databricks/python3/lib/python3.10/site-packages (from requests<3,>=2.17.3->mlflow==2.6.1.dev0) (2.0.4) Requirement already satisfied: joblib>=1.0.0 in /databricks/python3/lib/python3.10/site-packages (from scikit-learn<2->mlflow==2.6.1.dev0) (1.2.0) Requirement already satisfied: threadpoolctl>=2.0.0 in /databricks/python3/lib/python3.10/site-packages (from scikit-learn<2->mlflow==2.6.1.dev0) (2.2.0) Requirement already satisfied: greenlet!=0.4.17 in /databricks/python3/lib/python3.10/site-packages (from sqlalchemy<3,>=1.4.0->mlflow==2.6.1.dev0) (1.1.1) Requirement already satisfied: smmap<6,>=3.0.1 in /databricks/python3/lib/python3.10/site-packages (from gitdb<5,>=4.0.1->gitpython<4,>=2.1.0->mlflow==2.6.1.dev0) (5.0.0) Building wheels for collected packages: mlflow Building wheel for mlflow (pyproject.toml) ... done Created wheel for mlflow: filename=mlflow-2.6.1.dev0-py3-none-any.whl size=4449531 sha256=c646ea88b7f649e0ce1baa1115749001b4d6cdaac9ed57b01120f6fb936745b6 Stored in directory: /tmp/pip-ephem-wheel-cache-arcpqbmt/wheels/2e/80/4f/9a5f3814be89392f8b115cda11d1c33cd31f4211d3ec1b9c52 Successfully built mlflow Installing collected packages: querystring-parser, docker, alembic, mlflow Successfully installed alembic-1.12.0 docker-6.1.3 mlflow-2.6.1.dev0 querystring-parser-1.2.4 [notice] A new release of pip available: 22.2.2 -> 23.2.1 [notice] To update, run: pip install --upgrade pip Requirement already satisfied: transformers in /local_disk0/.ephemeral_nfs/envs/pythonEnv-7474d9db-15e1-4347-ac73-b9f5f396da63/lib/python3.10/site-packages (4.33.1) Requirement already satisfied: regex!=2019.12.17 in /databricks/python3/lib/python3.10/site-packages (from transformers) (2022.7.9) Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /databricks/python3/lib/python3.10/site-packages (from transformers) (0.13.3) Requirement already satisfied: requests in /databricks/python3/lib/python3.10/site-packages (from transformers) (2.28.1) Requirement already satisfied: numpy>=1.17 in /databricks/python3/lib/python3.10/site-packages (from transformers) (1.21.5) Requirement already satisfied: huggingface-hub<1.0,>=0.15.1 in /databricks/python3/lib/python3.10/site-packages (from transformers) (0.16.4) Requirement already satisfied: pyyaml>=5.1 in /databricks/python3/lib/python3.10/site-packages (from transformers) (6.0) Requirement already satisfied: packaging>=20.0 in /databricks/python3/lib/python3.10/site-packages (from transformers) (21.3) Requirement already satisfied: filelock in /databricks/python3/lib/python3.10/site-packages (from transformers) (3.6.0) Requirement already satisfied: tqdm>=4.27 in /databricks/python3/lib/python3.10/site-packages (from transformers) (4.64.1) Requirement already satisfied: safetensors>=0.3.1 in /databricks/python3/lib/python3.10/site-packages (from transformers) (0.3.2) Requirement already satisfied: typing-extensions>=3.7.4.3 in /databricks/python3/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.15.1->transformers) (4.3.0) Requirement already satisfied: fsspec in /databricks/python3/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.15.1->transformers) (2022.7.1) Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /databricks/python3/lib/python3.10/site-packages (from packaging>=20.0->transformers) (3.0.9) Requirement already satisfied: charset-normalizer<3,>=2 in /databricks/python3/lib/python3.10/site-packages (from requests->transformers) (2.0.4) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /databricks/python3/lib/python3.10/site-packages (from requests->transformers) (1.26.11) Requirement already satisfied: certifi>=2017.4.17 in /databricks/python3/lib/python3.10/site-packages (from requests->transformers) (2022.9.14) Requirement already satisfied: idna<4,>=2.5 in /databricks/python3/lib/python3.10/site-packages (from requests->transformers) (3.3) [notice] A new release of pip available: 22.2.2 -> 23.2.1 [notice] To update, run: pip install --upgrade pip Requirement already satisfied: accelerate in /local_disk0/.ephemeral_nfs/envs/pythonEnv-7474d9db-15e1-4347-ac73-b9f5f396da63/lib/python3.10/site-packages (0.22.0) Requirement already satisfied: psutil in /databricks/python3/lib/python3.10/site-packages (from accelerate) (5.9.0) Requirement already satisfied: torch>=1.10.0 in /databricks/python3/lib/python3.10/site-packages (from accelerate) (1.13.1+cu117) Requirement already satisfied: pyyaml in /databricks/python3/lib/python3.10/site-packages (from accelerate) (6.0) Requirement already satisfied: packaging>=20.0 in /databricks/python3/lib/python3.10/site-packages (from accelerate) (21.3) Requirement already satisfied: numpy>=1.17 in /databricks/python3/lib/python3.10/site-packages (from accelerate) (1.21.5) Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /databricks/python3/lib/python3.10/site-packages (from packaging>=20.0->accelerate) (3.0.9) Requirement already satisfied: typing-extensions in /databricks/python3/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (4.3.0) [notice] A new release of pip available: 22.2.2 -> 23.2.1 [notice] To update, run: pip install --upgrade pip

Initialize and configure your model

Download your model and package it with the model container.

Log your model using MLflow

The following code defines the inference parameters to pass to the model at the time of inference and the model schema.

Log the model with the MLflow pyfunc flavor.

Test your model in a notebook

Configure and create your model serving endpoint

The following variables set the values for configuring the model serving endpoint, such as the endpoint name, compute type, and which model to serve with the endpoint. After you call the create endpoint API, the logged model is deployed to the endpoint.

The follwoing sends the POST request to create the serving endpoint.

View your endpoint

To see more information about your endpoint, go to the Serving UI and search for your endpoint name.

Query your endpoint

Once your endpoint is ready, you can query it by making an API request. Depending on the model size and complexity, it can take 30 minutes or more for the endpoint to get ready.