Deploy models for batch inference and prediction

This article describes what Databricks recommends for batch and streaming inference.

For real-time model serving on Databricks, see Model serving with Databricks.

Use ai_query for batch inference

Preview

This feature is in Public Preview.

Databricks recommends using ai_query with Model Serving for batch inference. ai_query is a built-in Databricks SQL function that allows you to query existing model serving endpoints using SQL. It has been verified to reliably and consistently process datasets in the range of billions of tokens. See ai_query function for more detail about this AI function.

For quick experimentation, ai_query can be used with pay-per-token endpoints since these endpoints are pre-configured on your workspace.

When you are ready to run batch inference on large or production data, Databricks recommends using provisioned throughput endpoints for faster performance. See Perform batch LLM inference using ai_query.