Query LLMs and agents on Databricks

Databricks provides multiple ways to query large language models (LLMs), foundation models, and deployed agents. Choose an interactive UI, SQL, REST APIs, or client libraries depending on your workflow.

AI Playground

The AI Playground is a no-code chat environment in your Databricks workspace for prompting and comparing LLMs. Use it to experiment with prompts, tune parameters like temperature and max tokens, and prototype tool-calling agents and question-answering bots side by side before moving to code.

Foundation models

Unity AI Gateway serves Databricks-managed, pay-per-token foundation models behind a unified API, ready to use with no infrastructure commitment. Choose the type of model service that fits your needs:

• System-provided model services in system.ai — pre-configured model services in Unity Catalog. Good for getting started.
• Custom model services — create your own model services to distribute load across multiple models or set up fallbacks. Best for advanced governance and scale.

Query agents

After you build and deploy an agent, query the agent from your application. Agents can be hosted on Databricks Apps or Mosaic AI Model Serving endpoints. Databricks supports three query methods:

• Databricks OpenAI Client — recommended for new applications, with native streaming and full feature support.
• OpenAI-compatible REST API — language-agnostic, works with any platform that already speaks the OpenAI API.
• ai_query — query legacy agents hosted on Model Serving endpoints from SQL.

Enrich data using AI Functions

AI Functions are built-in SQL functions that apply LLMs and other models to data stored in Databricks. Run them from Databricks SQL, notebooks, Lakeflow Spark Declarative Pipelines, or Workflows to classify support tickets, extract entities from documents, summarize content, or translate text at batch scale.

Choose between two styles of functions:

• Task-specific functions like ai_classify, ai_extract, and ai_parse_document are optimized for a single task and use Databricks-managed, research-backed systems.
• ai_query is the general-purpose function — provide your own prompt and choose any supported foundation model. See Use ai_query.

For an end-to-end example, see Analyze customer reviews using AI Functions.