chat-batch-inference-udf(Python)

Loading...

Chat model batch inference tasks using PySpark Pandas UDF

This notebook is the partner notebook to the Perform batch inference on a provisioned throughput endpoint notebook. This notebook and the Perform batch inference on a provisioned throughput endpoint notebook must be in the same directory of your workspace for the batch inference workflow to perform successfully.

The following tasks are accomplished in this notebook:

  1. Read data from the input table and input column
  2. Construct the requests and send the requests to a Foundation Model APIs endpoint with some kind of concurrency
  3. Persist input row together with the response data to the output table

Requirements

To run this notebook you need:

The Databricks Runtime for Machine Learning version 14.3 LTS or above

Set up configuration parameters

UDF for batch inference