provisioned-throughput-gte-serving(Python)

Import Notebook

Provisioned Throughput GTE serving example

Provisioned Throughput provides optimized inference for Foundation Models with performance guarantees for production workloads.

This example walks through:

  • Downloading the model from Hugging Face transformers
  • Logging the model in a provisioned throughput supported format into the Databricks Unity Catalog or Workspace Registry
  • Enabling optimized serving on the model

Step 1: Log the model for serving

3

WARNING: The index url "12" seems invalid, please provide a scheme. WARNING: The index url "34" seems invalid, please provide a scheme. WARNING: The index url "56\78k;" seems invalid, please provide a scheme. Looking in indexes: https://pypi.org/simple, 12, 34, 56\78k; Requirement already satisfied: mlflow in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (2.16.0) WARNING: Location '12/mlflow/' is ignored: it is either a non-existing path or lacks a specific scheme. WARNING: Location '34/mlflow/' is ignored: it is either a non-existing path or lacks a specific scheme. WARNING: Location '56\78k;/mlflow/' is ignored: it is either a non-existing path or lacks a specific scheme. Requirement already satisfied: scipy<2 in /databricks/python3/lib/python3.10/site-packages (from mlflow) (1.10.0) Requirement already satisfied: pandas<3 in /databricks/python3/lib/python3.10/site-packages (from mlflow) (1.5.3) Requirement already satisfied: gunicorn<24 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow) (23.0.0) Requirement already satisfied: matplotlib<4 in /databricks/python3/lib/python3.10/site-packages (from mlflow) (3.7.0) Requirement already satisfied: alembic!=1.10.0,<2 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow) (1.13.2) Requirement already satisfied: numpy<3 in /databricks/python3/lib/python3.10/site-packages (from mlflow) (1.23.5) Requirement already satisfied: markdown<4,>=3.3 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow) (3.7) Requirement already satisfied: graphene<4 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow) (3.3) Requirement already satisfied: docker<8,>=4.0.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow) (7.1.0) Requirement already satisfied: Flask<4 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow) (3.0.3) Requirement already satisfied: scikit-learn<2 in /databricks/python3/lib/python3.10/site-packages (from mlflow) (1.1.1) Requirement already satisfied: sqlalchemy<3,>=1.4.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow) (2.0.34) Requirement already satisfied: mlflow-skinny==2.16.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow) (2.16.0) Requirement already satisfied: Jinja2<4,>=2.11 in /databricks/python3/lib/python3.10/site-packages (from mlflow) (3.1.2) Requirement already satisfied: pyarrow<18,>=4.0.0 in /databricks/python3/lib/python3.10/site-packages (from mlflow) (8.0.0) Requirement already satisfied: packaging<25 in /databricks/python3/lib/python3.10/site-packages (from mlflow-skinny==2.16.0->mlflow) (23.2) Requirement already satisfied: cachetools<6,>=5.0.0 in /databricks/python3/lib/python3.10/site-packages (from mlflow-skinny==2.16.0->mlflow) (5.3.2) Requirement already satisfied: databricks-sdk<1,>=0.20.0 in /databricks/python3/lib/python3.10/site-packages (from mlflow-skinny==2.16.0->mlflow) (0.20.0) Requirement already satisfied: cloudpickle<4 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow-skinny==2.16.0->mlflow) (3.0.0) Requirement already satisfied: importlib-metadata!=4.7.0,<9,>=3.7.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow-skinny==2.16.0->mlflow) (8.4.0) Requirement already satisfied: gitpython<4,>=3.1.9 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow-skinny==2.16.0->mlflow) (3.1.43) Requirement already satisfied: opentelemetry-api<3,>=1.9.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow-skinny==2.16.0->mlflow) (1.27.0) Requirement already satisfied: sqlparse<1,>=0.4.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow-skinny==2.16.0->mlflow) (0.5.1) Requirement already satisfied: pyyaml<7,>=5.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow-skinny==2.16.0->mlflow) (6.0.2) Requirement already satisfied: protobuf<6,>=3.12.0 in /databricks/python3/lib/python3.10/site-packages (from mlflow-skinny==2.16.0->mlflow) (4.25.3) Requirement already satisfied: opentelemetry-sdk<3,>=1.9.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow-skinny==2.16.0->mlflow) (1.27.0) Requirement already satisfied: requests<3,>=2.17.3 in /databricks/python3/lib/python3.10/site-packages (from mlflow-skinny==2.16.0->mlflow) (2.28.1) Requirement already satisfied: click<9,>=7.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from mlflow-skinny==2.16.0->mlflow) (8.1.7) Requirement already satisfied: typing-extensions>=4 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from alembic!=1.10.0,<2->mlflow) (4.12.2) Requirement already satisfied: Mako in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from alembic!=1.10.0,<2->mlflow) (1.3.5) Requirement already satisfied: urllib3>=1.26.0 in /databricks/python3/lib/python3.10/site-packages (from docker<8,>=4.0.0->mlflow) (1.26.14) Requirement already satisfied: Werkzeug>=3.0.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from Flask<4->mlflow) (3.0.4) Requirement already satisfied: blinker>=1.6.2 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from Flask<4->mlflow) (1.8.2) Requirement already satisfied: itsdangerous>=2.1.2 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from Flask<4->mlflow) (2.2.0) Requirement already satisfied: graphql-relay<3.3,>=3.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from graphene<4->mlflow) (3.2.0) Requirement already satisfied: aniso8601<10,>=8 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from graphene<4->mlflow) (9.0.1) Requirement already satisfied: graphql-core<3.3,>=3.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from graphene<4->mlflow) (3.2.4) Requirement already satisfied: MarkupSafe>=2.0 in /databricks/python3/lib/python3.10/site-packages (from Jinja2<4,>=2.11->mlflow) (2.1.1) Requirement already satisfied: pillow>=6.2.0 in /databricks/python3/lib/python3.10/site-packages (from matplotlib<4->mlflow) (9.4.0) Requirement already satisfied: kiwisolver>=1.0.1 in /databricks/python3/lib/python3.10/site-packages (from matplotlib<4->mlflow) (1.4.4) Requirement already satisfied: pyparsing>=2.3.1 in /databricks/python3/lib/python3.10/site-packages (from matplotlib<4->mlflow) (3.0.9) Requirement already satisfied: contourpy>=1.0.1 in /databricks/python3/lib/python3.10/site-packages (from matplotlib<4->mlflow) (1.0.5) Requirement already satisfied: python-dateutil>=2.7 in /databricks/python3/lib/python3.10/site-packages (from matplotlib<4->mlflow) (2.8.2) Requirement already satisfied: fonttools>=4.22.0 in /databricks/python3/lib/python3.10/site-packages (from matplotlib<4->mlflow) (4.25.0) Requirement already satisfied: cycler>=0.10 in /databricks/python3/lib/python3.10/site-packages (from matplotlib<4->mlflow) (0.11.0) Requirement already satisfied: pytz>=2020.1 in /databricks/python3/lib/python3.10/site-packages (from pandas<3->mlflow) (2022.7) Requirement already satisfied: threadpoolctl>=2.0.0 in /databricks/python3/lib/python3.10/site-packages (from scikit-learn<2->mlflow) (2.2.0) Requirement already satisfied: joblib>=1.0.0 in /databricks/python3/lib/python3.10/site-packages (from scikit-learn<2->mlflow) (1.2.0) Requirement already satisfied: greenlet!=0.4.17 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from sqlalchemy<3,>=1.4.0->mlflow) (3.1.0) Requirement already satisfied: google-auth~=2.0 in /databricks/python3/lib/python3.10/site-packages (from databricks-sdk<1,>=0.20.0->mlflow-skinny==2.16.0->mlflow) (2.28.1) Requirement already satisfied: gitdb<5,>=4.0.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from gitpython<4,>=3.1.9->mlflow-skinny==2.16.0->mlflow) (4.0.11) Requirement already satisfied: zipp>=0.5 in /usr/lib/python3/dist-packages (from importlib-metadata!=4.7.0,<9,>=3.7.0->mlflow-skinny==2.16.0->mlflow) (1.0.0) Requirement already satisfied: deprecated>=1.2.6 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from opentelemetry-api<3,>=1.9.0->mlflow-skinny==2.16.0->mlflow) (1.2.14) Requirement already satisfied: opentelemetry-semantic-conventions==0.48b0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from opentelemetry-sdk<3,>=1.9.0->mlflow-skinny==2.16.0->mlflow) (0.48b0) Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.7->matplotlib<4->mlflow) (1.16.0) Requirement already satisfied: certifi>=2017.4.17 in /databricks/python3/lib/python3.10/site-packages (from requests<3,>=2.17.3->mlflow-skinny==2.16.0->mlflow) (2022.12.7) Requirement already satisfied: charset-normalizer<3,>=2 in /databricks/python3/lib/python3.10/site-packages (from requests<3,>=2.17.3->mlflow-skinny==2.16.0->mlflow) (2.0.4) Requirement already satisfied: idna<4,>=2.5 in /databricks/python3/lib/python3.10/site-packages (from requests<3,>=2.17.3->mlflow-skinny==2.16.0->mlflow) (3.4) Requirement already satisfied: wrapt<2,>=1.10 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from deprecated>=1.2.6->opentelemetry-api<3,>=1.9.0->mlflow-skinny==2.16.0->mlflow) (1.16.0) Requirement already satisfied: smmap<6,>=3.0.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from gitdb<5,>=4.0.1->gitpython<4,>=3.1.9->mlflow-skinny==2.16.0->mlflow) (5.0.1) Requirement already satisfied: rsa<5,>=3.1.4 in /databricks/python3/lib/python3.10/site-packages (from google-auth~=2.0->databricks-sdk<1,>=0.20.0->mlflow-skinny==2.16.0->mlflow) (4.9) Requirement already satisfied: pyasn1-modules>=0.2.1 in /databricks/python3/lib/python3.10/site-packages (from google-auth~=2.0->databricks-sdk<1,>=0.20.0->mlflow-skinny==2.16.0->mlflow) (0.3.0) Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /databricks/python3/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth~=2.0->databricks-sdk<1,>=0.20.0->mlflow-skinny==2.16.0->mlflow) (0.5.1) Note: you may need to restart the kernel using %restart_python or dbutils.library.restartPython() to use updated packages. WARNING: The index url "12" seems invalid, please provide a scheme. WARNING: The index url "34" seems invalid, please provide a scheme. WARNING: The index url "56\78k;" seems invalid, please provide a scheme. Looking in indexes: https://pypi.org/simple, 12, 34, 56\78k; Requirement already satisfied: transformers in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (4.44.2) WARNING: Location '12/transformers/' is ignored: it is either a non-existing path or lacks a specific scheme. WARNING: Location '34/transformers/' is ignored: it is either a non-existing path or lacks a specific scheme. WARNING: Location '56\78k;/transformers/' is ignored: it is either a non-existing path or lacks a specific scheme. Requirement already satisfied: regex!=2019.12.17 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from transformers) (2024.9.11) Requirement already satisfied: safetensors>=0.4.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from transformers) (0.4.5) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.14.0) Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from transformers) (0.24.7) Requirement already satisfied: tqdm>=4.27 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from transformers) (4.66.5) Requirement already satisfied: packaging>=20.0 in /databricks/python3/lib/python3.10/site-packages (from transformers) (23.2) Requirement already satisfied: requests in /databricks/python3/lib/python3.10/site-packages (from transformers) (2.28.1) Requirement already satisfied: numpy>=1.17 in /databricks/python3/lib/python3.10/site-packages (from transformers) (1.23.5) Requirement already satisfied: pyyaml>=5.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from transformers) (6.0.2) Requirement already satisfied: tokenizers<0.20,>=0.19 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from transformers) (0.19.1) Requirement already satisfied: fsspec>=2023.5.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.23.2->transformers) (2024.9.0) Requirement already satisfied: typing-extensions>=3.7.4.3 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.23.2->transformers) (4.12.2) Requirement already satisfied: charset-normalizer<3,>=2 in /databricks/python3/lib/python3.10/site-packages (from requests->transformers) (2.0.4) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /databricks/python3/lib/python3.10/site-packages (from requests->transformers) (1.26.14) Requirement already satisfied: certifi>=2017.4.17 in /databricks/python3/lib/python3.10/site-packages (from requests->transformers) (2022.12.7) Requirement already satisfied: idna<4,>=2.5 in /databricks/python3/lib/python3.10/site-packages (from requests->transformers) (3.4) Note: you may need to restart the kernel using %restart_python or dbutils.library.restartPython() to use updated packages. WARNING: The index url "12" seems invalid, please provide a scheme. WARNING: The index url "34" seems invalid, please provide a scheme. WARNING: The index url "56\78k;" seems invalid, please provide a scheme. Looking in indexes: https://pypi.org/simple, 12, 34, 56\78k; Requirement already satisfied: torch in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (2.4.1) WARNING: Location '12/torch/' is ignored: it is either a non-existing path or lacks a specific scheme. WARNING: Location '34/torch/' is ignored: it is either a non-existing path or lacks a specific scheme. WARNING: Location '56\78k;/torch/' is ignored: it is either a non-existing path or lacks a specific scheme. Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (12.1.105) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (12.1.105) Requirement already satisfied: sympy in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (1.13.2) Requirement already satisfied: typing-extensions>=4.8.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (4.12.2) Requirement already satisfied: networkx in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (3.3) Requirement already satisfied: fsspec in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (2024.9.0) Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (12.1.3.1) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (12.1.105) Requirement already satisfied: jinja2 in /databricks/python3/lib/python3.10/site-packages (from torch) (3.1.2) Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (10.3.2.106) Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (2.20.5) Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (11.0.2.54) Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (9.1.0.70) Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (12.1.105) Requirement already satisfied: triton==3.0.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (3.0.0) Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (11.4.5.107) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch) (3.14.0) Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch) (12.1.0.106) Requirement already satisfied: nvidia-nvjitlink-cu12 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch) (12.6.68) Requirement already satisfied: MarkupSafe>=2.0 in /databricks/python3/lib/python3.10/site-packages (from jinja2->torch) (2.1.1) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from sympy->torch) (1.3.0) Note: you may need to restart the kernel using %restart_python or dbutils.library.restartPython() to use updated packages. WARNING: The index url "12" seems invalid, please provide a scheme. WARNING: The index url "34" seems invalid, please provide a scheme. WARNING: The index url "56\78k;" seems invalid, please provide a scheme. Looking in indexes: https://pypi.org/simple, 12, 34, 56\78k; Requirement already satisfied: torchvision in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (0.19.1) WARNING: Location '12/torchvision/' is ignored: it is either a non-existing path or lacks a specific scheme. WARNING: Location '34/torchvision/' is ignored: it is either a non-existing path or lacks a specific scheme. WARNING: Location '56\78k;/torchvision/' is ignored: it is either a non-existing path or lacks a specific scheme. Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /databricks/python3/lib/python3.10/site-packages (from torchvision) (9.4.0) Requirement already satisfied: numpy in /databricks/python3/lib/python3.10/site-packages (from torchvision) (1.23.5) Requirement already satisfied: torch==2.4.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torchvision) (2.4.1) Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (12.1.105) Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (12.1.3.1) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (12.1.105) Requirement already satisfied: triton==3.0.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (3.0.0) Requirement already satisfied: networkx in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (3.3) Requirement already satisfied: fsspec in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (2024.9.0) Requirement already satisfied: jinja2 in /databricks/python3/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (3.1.2) Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (9.1.0.70) Requirement already satisfied: sympy in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (1.13.2) Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (11.4.5.107) Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (12.1.0.106) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch==2.4.1->torchvision) (3.14.0) Requirement already satisfied: typing-extensions>=4.8.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (4.12.2) Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (11.0.2.54) Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (12.1.105) Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (10.3.2.106) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (12.1.105) Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch==2.4.1->torchvision) (2.20.5) Requirement already satisfied: nvidia-nvjitlink-cu12 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch==2.4.1->torchvision) (12.6.68) Requirement already satisfied: MarkupSafe>=2.0 in /databricks/python3/lib/python3.10/site-packages (from jinja2->torch==2.4.1->torchvision) (2.1.1) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from sympy->torch==2.4.1->torchvision) (1.3.0) Note: you may need to restart the kernel using %restart_python or dbutils.library.restartPython() to use updated packages. WARNING: The index url "12" seems invalid, please provide a scheme. WARNING: The index url "34" seems invalid, please provide a scheme. WARNING: The index url "56\78k;" seems invalid, please provide a scheme. Looking in indexes: https://pypi.org/simple, 12, 34, 56\78k; Requirement already satisfied: accelerate in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (0.34.2) WARNING: Location '12/accelerate/' is ignored: it is either a non-existing path or lacks a specific scheme. WARNING: Location '34/accelerate/' is ignored: it is either a non-existing path or lacks a specific scheme. WARNING: Location '56\78k;/accelerate/' is ignored: it is either a non-existing path or lacks a specific scheme. Requirement already satisfied: packaging>=20.0 in /databricks/python3/lib/python3.10/site-packages (from accelerate) (23.2) Requirement already satisfied: huggingface-hub>=0.21.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from accelerate) (0.24.7) Requirement already satisfied: pyyaml in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from accelerate) (6.0.2) Requirement already satisfied: safetensors>=0.4.3 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from accelerate) (0.4.5) Requirement already satisfied: psutil in /databricks/python3/lib/python3.10/site-packages (from accelerate) (5.9.0) Requirement already satisfied: numpy<3.0.0,>=1.17 in /databricks/python3/lib/python3.10/site-packages (from accelerate) (1.23.5) Requirement already satisfied: torch>=1.10.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from accelerate) (2.4.1) Requirement already satisfied: tqdm>=4.42.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from huggingface-hub>=0.21.0->accelerate) (4.66.5) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.21.0->accelerate) (3.14.0) Requirement already satisfied: typing-extensions>=3.7.4.3 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from huggingface-hub>=0.21.0->accelerate) (4.12.2) Requirement already satisfied: requests in /databricks/python3/lib/python3.10/site-packages (from huggingface-hub>=0.21.0->accelerate) (2.28.1) Requirement already satisfied: fsspec>=2023.5.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from huggingface-hub>=0.21.0->accelerate) (2024.9.0) Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (2.20.5) Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (9.1.0.70) Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (11.4.5.107) Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (12.1.105) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (12.1.105) Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (11.0.2.54) Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (12.1.3.1) Requirement already satisfied: jinja2 in /databricks/python3/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (3.1.2) Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (12.1.0.106) Requirement already satisfied: sympy in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (1.13.2) Requirement already satisfied: triton==3.0.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (3.0.0) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (12.1.105) Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (10.3.2.106) Requirement already satisfied: networkx in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (3.3) Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (12.1.105) Requirement already satisfied: nvidia-nvjitlink-cu12 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.10.0->accelerate) (12.6.68) Requirement already satisfied: MarkupSafe>=2.0 in /databricks/python3/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->accelerate) (2.1.1) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /databricks/python3/lib/python3.10/site-packages (from requests->huggingface-hub>=0.21.0->accelerate) (1.26.14) Requirement already satisfied: idna<4,>=2.5 in /databricks/python3/lib/python3.10/site-packages (from requests->huggingface-hub>=0.21.0->accelerate) (3.4) Requirement already satisfied: certifi>=2017.4.17 in /databricks/python3/lib/python3.10/site-packages (from requests->huggingface-hub>=0.21.0->accelerate) (2022.12.7) Requirement already satisfied: charset-normalizer<3,>=2 in /databricks/python3/lib/python3.10/site-packages (from requests->huggingface-hub>=0.21.0->accelerate) (2.0.4) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ce218075-e679-4671-bada-438af6160064/lib/python3.10/site-packages (from sympy->torch>=1.10.0->accelerate) (1.3.0) Note: you may need to restart the kernel using %restart_python or dbutils.library.restartPython() to use updated packages.
4

config.json: 0%| | 0.00/1.35k [00:00<?, ?B/s]
model.safetensors: 0%| | 0.00/1.74G [00:00<?, ?B/s]
tokenizer_config.json: 0%| | 0.00/1.38k [00:00<?, ?B/s]
vocab.txt: 0%| | 0.00/232k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/712k [00:00<?, ?B/s]
special_tokens_map.json: 0%| | 0.00/695 [00:00<?, ?B/s]

To enable optimized serving, when logging the model, include the extra metadata dictionary when calling mlflow.transformers.log_model as shown below:

metadata = {"task": "llm/v1/completions"} This specifies the API signature used for the model serving endpoint.

6

7

README.md: 0%| | 0.00/71.8k [00:00<?, ?B/s]
Uploading artifacts: 0%| | 0/21 [00:00<?, ?it/s]
Successfully registered model 'ml.models.gte-large'.
Uploading artifacts: 0%| | 0/21 [00:00<?, ?it/s]
Created version '1' of model 'ml.models.gte-large'. 2024/09/12 21:57:20 INFO mlflow.tracking._tracking_service.client: 🏃 View run auspicious-auk-762 at: https://oregon.staging.cloud.databricks.com/ml/experiments/774702280974447/runs/7da4877fe96e4e3b86bd9eec328ca0e1. 2024/09/12 21:57:20 INFO mlflow.tracking._tracking_service.client: 🧪 View experiment at: https://oregon.staging.cloud.databricks.com/ml/experiments/774702280974447.

Step 2: View optimization information for your model

Modify the cell below to change the model name. After calling the model optimization information API, you will be able to retrieve throughput chunk size information for your model. This is the number of tokens/second that corresponds to 1 throughput unit for your specific model.

9

{ "optimizable": true, "model_type": "new", "throughput_chunk_size": 19000, "dbus": 24 }

Step 3: Configure and create your model serving GPU endpoint

Modify the cell below to change the endpoint name. After calling the create endpoint API, the logged MPT-7B model is automatically deployed with optimized LLM serving.

11

12

13

{ "name": "gte-large", "creator": "prithu.dasgupta@databricks.com", "creation_timestamp": 1726178249000, "last_updated_timestamp": 1726178249000, "state": { "ready": "NOT_READY", "config_update": "IN_PROGRESS", "suspend": "NOT_SUSPENDED" }, "pending_config": { "start_time": 1726178249000, "served_models": [ { "name": "gte-large-1", "model_name": "ml.models.gte-large", "model_version": "1", "workload_size": "Small", "workload_type": "GPU_MEDIUM", "min_provisioned_throughput": 38000, "max_provisioned_throughput": 38000, "min_dbus": 48.0, "max_dbus": 48.0, "state": { "deployment": "DEPLOYMENT_CREATING", "deployment_state_message": "Creating resources for served entity." }, "creator": "prithu.dasgupta@databricks.com", "creation_timestamp": 1726178249000 } ], "served_entities": [ { "name": "gte-large-1", "entity_name": "ml.models.gte-large", "entity_version": "1", "workload_size": "Small", "workload_type": "GPU_MEDIUM", "min_provisioned_throughput": 38000, "max_provisioned_throughput": 38000, "min_dbus": 48.0, "max_dbus": 48.0, "state": { "deployment": "DEPLOYMENT_CREATING", "deployment_state_message": "Creating resources for served entity." }, "creator": "prithu.dasgupta@databricks.com", "creation_timestamp": 1726178249000 } ], "config_version": 1, "traffic_config": { "routes": [ { "served_model_name": "gte-large-1", "traffic_percentage": 100, "served_entity_name": "gte-large-1" } ] } }, "id": "e1e4355999f949149668ab771833044c", "permission_level": "CAN_MANAGE", "route_optimized": false }

Step 4: Query your endpoint

After your endpoint is ready, you can query it by making an API request. Depending on the model size and complexity, it can take 30 minutes or more for the endpoint to get ready.

15

{"id": "ca57c2c7-5364-48bf-9c19-b5f84d51a74e", "object": "list", "model": "", "data": [{"index": 0, "object": "embedding", "embedding": [0.35498046875, -0.45361328125, 0.640625, 0.60986328125, 0.287353515625, 0.311279296875, -0.07806396484375, 0.308349609375, 0.86962890625, 0.08160400390625, -0.65087890625, -0.4794921875, -0.2135009765625, 0.537109375, -0.80224609375, 0.1463623046875, -0.59716796875, 0.0802001953125, 0.08447265625, 0.288330078125, 0.26025390625, -1.345703125, -0.345458984375, -1.71484375, -0.39208984375, 1.1005859375, -0.5869140625, -0.57177734375, 0.791015625, -0.396728515625, 0.42041015625, 0.61376953125, -0.87548828125, -0.0462646484375, 0.4248046875, -0.3037109375, -0.1922607421875, -0.020416259765625, -0.03619384765625, -0.6875, -0.401611328125, -0.412109375, -0.46044921875, -0.5234375, 1.0302734375, 0.049591064453125, 0.5771484375, 0.2462158203125, -0.541015625, 0.6298828125, -0.89990234375, 0.513671875, 0.1611328125, -0.580078125, -0.0701904296875, 0.677734375, 0.2470703125, 0.11083984375, 0.51318359375, -0.343505859375, -0.30224609375, 0.90283203125, -0.468505859375, -0.83544921875, 0.09039306640625, -0.004558563232421875, -0.58154296875, 0.06610107421875, -1.044921875, 0.68994140625, 0.9150390625, 0.5595703125, 0.2578125, -0.356689453125, 1.5419921875, -0.8251953125, 0.0972900390625, 0.51171875, 0.64111328125, 0.5703125, -0.404296875, 0.30078125, 0.0298919677734375, -0.33740234375, 0.07330322265625, 1.5166015625, 0.97998046875, -0.47314453125, 0.36767578125, 1.3662109375, -0.4658203125, -0.05950927734375, 0.04815673828125, -0.8212890625, -0.442626953125, 0.278076171875, 0.78369140625, -0.66259765625, -0.94482421875, -0.19091796875, -0.55126953125, -0.09228515625, 0.5205078125, 0.2529296875, 0.1871337890625, 0.87646484375, -1.189453125, -0.146484375, 0.8056640625, -0.64794921875, 0.1351318359375, 0.209228515625, 0.09912109375, 0.2066650390625, 0.50341796875, 0.376220703125, 0.333740234375, -1.2734375, 0.64306640625, 0.357421875, 0.41015625, -0.1297607421875, 0.67236328125, 0.08154296875, 0.703125, 0.227783203125, 0.382080078125, 0.31005859375, -0.86474609375, 0.5595703125, 0.072021484375, -0.315185546875, -0.6630859375, 0.3984375, 0.0535888671875, -0.056640625, -0.9951171875, 0.283935546875, 0.75439453125, -0.39599609375, -0.33642578125, -0.1748046875, 0.771484375, -0.35546875, 1.1162109375, 0.24072265625, -0.73779296875, 1.4130859375, 0.250732421875, -1.72265625, -1.7900390625, 0.2666015625, -0.22412109375, -0.0933837890625, 0.8916015625, 0.81689453125, -0.323974609375, -0.83154296875, -0.59619140625, -0.853515625, -0.748046875, -0.9150390625, 0.0693359375, 1.13671875, -0.6962890625, 0.99609375, -0.65771484375, 1.03515625, 0.51318359375, 0.55859375, -1.125, -0.397705078125, 0.348388671875, 0.264404296875, -0.197021484375, -0.267822265625, 0.0007805824279785156, -0.9453125, 0.10284423828125, -0.9599609375, -0.05859375, -0.33203125, -0.18603515625, -0.345703125, -0.66064453125, -0.264404296875, 0.37890625, 0.52783203125, 0.2166748046875, -0.5791015625, -0.10369873046875, 0.63525390625, -0.34228515625, 0.17578125, -0.89453125, -0.392578125, -0.84765625, -0.26220703125, -0.1448974609375, 0.55224609375, -0.3681640625, -0.427490234375, -0.464599609375, -0.2919921875, -0.2509765625, -0.1705322265625, 0.0806884765625, 0.074462890625, -0.038726806640625, -0.497314453125, -1.58984375, -1.341796875, -0.4375, 0.60107421875, -0.78955078125, -0.2459716796875, -0.029632568359375, -0.250244140625, -0.489990234375, 0.2039794921875, 0.302734375, 0.1884765625, 0.1239013671875, 0.038604736328125, 0.8525390625, -0.206787109375, -0.167236328125, 0.465087890625, -0.47021484375, 0.33544921875, -0.362060546875, 0.406005859375, 0.0809326171875, -0.03369140625, 1.197265625, -1.0927734375, -0.82177734375, 0.4384765625, 0.08673095703125, 0.045440673828125, 0.44775390625, -0.57958984375, 0.689453125, -0.201904296875, 0.271240234375, 0.38623046875, 0.6943359375, -0.2080078125, -0.33740234375, 0.10308837890625, 0.1827392578125, 0.0908203125, -0.1739501953125, 0.062744140625, -0.10992431640625, -0.494384765625, 0.11773681640625, -0.434326171875, 0.64013671875, 0.1153564453125, -0.416015625, -0.33251953125, 1.099609375, -1.4619140625, 0.319091796875, 0.6875, 0.213134765625, -0.88720703125, 0.58984375, -0.2344970703125, 0.05755615234375, -0.66162109375, 1.05859375, -0.8916015625, -0.057769775390625, -0.5634765625, -0.4296875, -0.422607421875, 0.1802978515625, 0.282470703125, -0.2939453125, 0.67041015625, -0.53271484375, -0.240478515625, -0.58203125, -0.1663818359375, -0.2298583984375, 0.468505859375, 0.65478515625, 0.359130859375, 0.1600341796875, -0.347900390625, 0.69677734375, -0.70458984375, -0.487548828125, 0.9482421875, 0.70556640625, -0.374755859375, 1.32421875, -0.150634765625, 0.87841796875, 1.0078125, -0.208984375, -1.2421875, 1.3671875, 0.293212890625, 0.382080078125, -0.135986328125, 0.225341796875, 0.032684326171875, -0.6611328125, 1.08984375, 0.66162109375, 0.5439453125, 0.0025005340576171875, 0.6396484375, -1.6455078125, 0.430908203125, -0.31298828125, -0.468017578125, 0.42431640625, -0.78271484375, 0.0557861328125, 1.3212890625, -0.2080078125, 0.578125, 0.4541015625, 0.05682373046875, 0.46728515625, 0.5498046875, 0.275390625, -0.057403564453125, -0.44140625, 0.251953125, 0.1768798828125, -0.47021484375, 0.87939453125, -0.72900390625, -0.49072265625, 0.225341796875, -0.455078125, -0.1806640625, 0.84912109375, 0.3955078125, 1.3642578125, 0.5263671875, -0.8310546875, -0.64697265625, -0.51025390625, -0.07928466796875, 0.37353515625, 0.285400390625, 0.43310546875, -0.265625, 0.56396484375, -0.4619140625, -0.052978515625, 0.44384765625, -0.884765625, 0.56689453125, 0.6484375, -0.8359375, -2.087890625, 0.2802734375, -0.62353515625, -0.3203125, 0.82568359375, 0.0665283203125, 0.1510009765625, 0.37939453125, 0.69921875, -0.0056915283203125, -0.460205078125, -0.451171875, 0.035247802734375, 0.29052734375, -0.60595703125, -0.2252197265625, -0.275390625, 1.0283203125, 1.1123046875, -0.07421875, 0.38671875, -0.1453857421875, -0.281982421875, 0.11431884765625, -1.0810546875, 0.54296875, -0.64990234375, 1.0166015625, 0.2332763671875, -0.83544921875, 0.228515625, -0.18798828125, -0.0870361328125, -0.693359375, -0.047027587890625, 0.20361328125, 0.6591796875, -0.26025390625, -0.09478759765625, 0.791015625, -1.044921875, 0.5615234375, -0.1343994140625, 0.1890869140625, 0.431396484375, -0.73046875, -0.77197265625, -0.62060546875, 0.007282257080078125, -0.75048828125, -0.474365234375, 0.459228515625, -0.2449951171875, -0.327392578125, 0.80029296875, 0.52490234375, 0.54296875, -0.25927734375, -0.476318359375, -0.62890625, 0.371337890625, 0.61279296875, -1.15625, 0.59765625, -0.2386474609375, -0.255859375, 0.33740234375, 0.8896484375, -0.48095703125, -0.241455078125, 0.6923828125, 0.032623291015625, 0.6787109375, -0.857421875, -0.01117706298828125, -0.315673828125, -0.5322265625, 0.467041015625, -0.059967041015625, 0.89599609375, -0.55322265625, 0.9443359375, 0.0302886962890625, 1.1064453125, 0.5361328125, -0.2022705078125, 0.443603515625, 0.1947021484375, 0.08453369140625, -1.318359375, 0.9404296875, 0.74169921875, 0.69873046875, -0.5439453125, -0.078125, -0.42431640625, 0.54638671875, 0.51123046875, 0.254638671875, 0.017669677734375, -0.1453857421875, -0.693359375, 1.189453125, -0.81640625, -0.040802001953125, 0.343017578125, -0.64990234375, 0.1407470703125, 0.64501953125, 0.8623046875, -0.01093292236328125, -0.169921875, 0.1363525390625, 0.69384765625, 0.9658203125, 0.471435546875, 0.180908203125, -1.2138671875, 0.5517578125, -0.9716796875, -0.5126953125, -1.2060546875, -0.5146484375, 0.39697265625, 0.22216796875, -0.98583984375, -0.8564453125, -1.3935546875, -0.11151123046875, -0.02459716796875, -0.94873046875, -0.578125, -1.404296875, 0.13818359375, 0.036529541015625, -0.56103515625, -0.013153076171875, -1.1787109375, -0.52587890625, -0.147705078125, 0.371826171875, 0.41162109375, 0.390625, -0.09075927734375, -0.1181640625, 0.01177215576171875, -0.51123046875, 0.44482421875, -0.417724609375, -0.52685546875, 0.94384765625, 0.05584716796875, 0.55810546875, -0.350830078125, 0.45849609375, -0.42236328125, 0.0011692047119140625, 0.260498046875, -1.1162109375, 0.319091796875, -0.52001953125, 0.55615234375, 0.5595703125, 2.01171875, -0.203857421875, -0.6416015625, -0.434326171875, 0.1356201171875, 0.61474609375, 0.07867431640625, -0.444091796875, 0.478271484375, -0.131103515625, -0.47705078125, 0.0731201171875, -0.587890625, -0.2449951171875, -0.0011682510375976562, 0.63623046875, 0.00390625, -0.79150390625, 0.07513427734375, 0.1314697265625, 0.92919921875, -0.036834716796875, -0.09979248046875, -0.748046875, -0.314208984375, -0.1453857421875, -0.6328125, 0.1090087890625, 0.28271484375, -0.2257080078125, -0.271484375, 0.0175933837890625, 0.343994140625, -0.26025390625, -0.90478515625, -0.2919921875, 0.0255126953125, -0.454833984375, 0.443359375, -0.365478515625, 0.55810546875, 0.71728515625, 0.53466796875, -0.1475830078125, -0.5625, -0.8134765625, 0.1507568359375, -0.42626953125, -0.84716796875, 0.83154296875, 0.1260986328125, 0.391357421875, -0.261962890625, -0.45947265625, 0.0946044921875, -0.289794921875, 0.78857421875, -0.60302734375, -0.57080078125, 0.59716796875, 1.173828125, 0.78955078125, -1.0908203125, -0.433837890625, -0.6669921875, 0.54052734375, -0.927734375, -0.049102783203125, 0.284423828125, -1.345703125, 0.1402587890625, -0.30126953125, -0.529296875, -0.256103515625, -0.17822265625, 0.250732421875, -0.364501953125, -0.5908203125, -0.6123046875, 0.49462890625, -0.036224365234375, 0.0302734375, -0.986328125, -0.39013671875, 0.50537109375, 0.161376953125, 0.947265625, -0.0963134765625, -0.06787109375, 0.6630859375, 0.95556640625, 0.243896484375, -0.29345703125, 0.046112060546875, 0.1761474609375, -0.439208984375, 0.10003662109375, 1.240234375, 0.9111328125, -0.38818359375, 0.384521484375, 0.55859375, 0.10296630859375, 0.071533203125, 0.79638671875, -0.30029296875, -0.1202392578125, 0.5517578125, -0.1881103515625, -0.84130859375, -1.0419921875, 0.58251953125, 0.95947265625, 0.69140625, -1.2197265625, -0.78466796875, -0.42333984375, -0.333984375, -0.480224609375, -0.01715087890625, 0.057281494140625, 0.6953125, 0.1334228515625, -0.1951904296875, 0.0207061767578125, -0.369140625, 0.83740234375, 0.344482421875, 0.089111328125, 0.39794921875, 1.1513671875, 0.2099609375, -0.66259765625, -0.0013713836669921875, 0.271728515625, -1.439453125, 2.150390625, 0.69091796875, 0.3310546875, 0.271240234375, 0.1839599609375, -0.287109375, -0.89697265625, -0.60498046875, 0.09478759765625, -0.38671875, 0.404296875, 0.390380859375, 0.87158203125, 0.2265625, 0.5, -0.1533203125, -0.22607421875, -1.9130859375, -0.1221923828125, 0.419921875, 0.2493896484375, 0.205810546875, -0.6806640625, -0.89453125, -0.669921875, 0.0029850006103515625, -0.153076171875, -0.403564453125, -0.1517333984375, -0.5439453125, 0.04180908203125, 0.84228515625, 0.05572509765625, -0.037322998046875, -1.1494140625, -0.1666259765625, -0.1243896484375, -0.3330078125, -0.09033203125, -0.79541015625, -0.85302734375, -0.4189453125, -0.4375, 0.64697265625, -0.11236572265625, -0.82373046875, 0.19921875, 0.2403564453125, 0.763671875, 0.5888671875, -0.630859375, -0.52197265625, 0.048858642578125, 1.7451171875, 0.300048828125, -0.4609375, -0.1280517578125, -0.275146484375, -0.41845703125, -0.5615234375, -0.71923828125, -0.03216552734375, -0.3291015625, -0.454345703125, -0.06097412109375, -0.1773681640625, 0.1717529296875, 0.55908203125, 0.483154296875, 0.06951904296875, -0.53466796875, 0.042144775390625, -0.53173828125, 0.8798828125, -0.099853515625, -0.869140625, 0.385009765625, -0.708984375, -0.032470703125, -0.350830078125, 0.2369384765625, -0.307861328125, 0.80029296875, -0.298828125, -0.267822265625, 1.171875, 0.14794921875, -1.1884765625, -0.70703125, 0.82275390625, 0.109375, -0.070068359375, -0.619140625, 0.1572265625, -0.1697998046875, -1.29296875, -0.58056640625, -0.8251953125, -0.4482421875, 0.1915283203125, 0.740234375, 0.615234375, -0.50927734375, 0.6767578125, -0.62060546875, -0.68115234375, -0.032440185546875, -0.16455078125, -1.0478515625, -0.15087890625, 0.183349609375, -0.471435546875, -0.343017578125, -0.2386474609375, -0.50146484375, 0.411865234375, 0.12347412109375, 0.05633544921875, -0.892578125, 0.400390625, -0.74365234375, -0.50537109375, -0.373291015625, 0.271240234375, -0.035614013671875, 0.05926513671875, 0.0592041015625, -0.50146484375, -1.0859375, -0.52734375, 0.4462890625, 0.73291015625, 0.8134765625, 0.34912109375, -0.9931640625, 0.4052734375, -0.2861328125, -0.29541015625, 0.61181640625, 1.0087890625, 0.53955078125, 1.0712890625, 0.5009765625, 0.6875, 0.06951904296875, 0.3603515625, -0.74462890625, 0.422607421875, 0.26611328125, -0.595703125, -0.3291015625, -0.42333984375, -0.0192413330078125, 0.23681640625, 0.00885009765625, 0.66796875, 0.8212890625, 0.366455078125, -0.50927734375, -0.15234375, -0.55419921875, 0.33056640625, 0.12176513671875, 0.44677734375, 0.11279296875, -0.490234375, -0.1429443359375, -0.2379150390625, 0.195556640625, 0.51220703125, 0.163330078125, -1.22265625, 0.55126953125, 0.0809326171875, -1.333984375, -0.326904296875, 0.0513916015625, 0.07452392578125, 0.131103515625, 0.142578125, -0.016021728515625, -0.053253173828125, -1.1962890625, -0.05877685546875, -0.79443359375, -0.09515380859375, -0.261474609375, -0.42236328125, -0.1185302734375, 0.849609375, 0.360107421875, -0.1510009765625, 0.250244140625, 0.50244140625, -0.50244140625, 0.060211181640625, -0.607421875, -0.1339111328125, -0.301513671875, 0.7548828125, -0.6416015625, -0.128173828125, 1.0107421875, -0.34912109375, 0.5771484375, -1.0859375, -0.047637939453125, -0.6064453125, 0.70166015625, -0.314453125, -0.37451171875, 1.1943359375, -0.83984375, -0.529296875, -0.5087890625, 0.1719970703125, -0.486328125, -0.79931640625, 1.08984375, 0.77490234375, -1.1533203125, 0.092041015625, -0.68798828125, 1.0498046875, -0.56103515625, 0.00974273681640625, -1.1708984375, -0.1502685546875, 1.171875, -0.01342010498046875, -0.65673828125, 0.40576171875, 0.41162109375, -0.890625, 0.431396484375, -0.0809326171875, 0.441162109375, -0.0595703125, 0.0401611328125, 0.3251953125, -0.673828125, 0.312255859375, 0.8271484375, -0.495849609375, 0.96533203125, 0.467529296875, -0.11346435546875, 1.296875, -0.430419921875, -0.607421875, -0.72314453125, 0.262451171875, -0.86572265625, -0.10406494140625, 0.31591796875, 0.407958984375, 0.5107421875, 0.29150390625, -0.30517578125, 0.143798828125, 0.01654052734375, 0.11480712890625, -0.45166015625, 0.093017578125, 0.286865234375, -0.0594482421875, -0.5458984375, -0.73291015625, 0.68408203125, 0.29833984375, 0.68798828125, 0.96875, -0.7529296875, 0.6982421875, -0.2685546875, -0.8310546875, -0.0161285400390625, 0.22802734375, 0.83935546875, 0.5390625, -0.94287109375, -14.3203125, 0.27392578125, 0.7763671875, -0.489013671875, -0.19580078125, -0.2119140625, -0.11358642578125, 0.208984375, -0.7294921875, -0.431396484375, -0.85400390625, 0.41259765625, -0.3994140625, 0.13232421875, -0.2025146484375, -0.54443359375, -0.87255859375, -1.240234375, 0.241455078125, -0.66943359375, 0.305908203125, 0.7890625, -0.6708984375, -0.48583984375, -0.12066650390625, 1.619140625, 0.0179290771484375, -0.30615234375, -0.869140625, 1.349609375, -0.349365234375, 0.681640625, -1.0712890625, 0.2244873046875, -0.509765625, 0.02130126953125, 0.15576171875, 0.59521484375, -0.7822265625, -0.5693359375, 0.8779296875, 0.47802734375, -0.43310546875, 0.7470703125, 0.34326171875, 0.8720703125, -0.97412109375, -0.41015625, -0.443359375, 0.1256103515625, 0.372314453125, 0.218505859375, -0.01543426513671875, 0.765625, -0.1585693359375, -0.05224609375, -0.4521484375, -0.443359375, -0.467041015625, -0.0157928466796875, 0.1348876953125, 1.4580078125, -0.37646484375, -0.468994140625, 0.31103515625, -0.2177734375, -0.6259765625, -0.08538818359375, 0.509765625, -0.81201171875, 0.3623046875, -1.54296875, -0.03546142578125, -0.485595703125, 0.56640625, 0.7080078125, 0.276611328125, 0.1353759765625, 0.126220703125, 0.29541015625, -0.3779296875, -0.6943359375]}], "usage": {"prompt_tokens": 8, "total_tokens": 8}}