Skip to main content
Last updated on

Query LLMs and agents on Databricks

Databricks provides multiple ways to query large language models (LLMs), foundation models, and deployed agents. Choose an interactive UI, SQL, REST APIs, or client libraries depending on your workflow.

AI Playground​

The AI Playground is a no-code chat environment in your Databricks workspace for prompting and comparing LLMs. Use it to experiment with prompts, tune parameters like temperature and max tokens, and prototype tool-calling agents and question-answering bots side by side before moving to code.

Enrich data using AI Functions​

AI Functions are built-in SQL functions that apply LLMs and other models to data stored in Databricks. Run them from Databricks SQL, notebooks, Lakeflow Spark Declarative Pipelines, or Workflows to classify support tickets, extract entities from documents, summarize content, or translate text at batch scale.

Choose between two styles of functions:

  • Task-specific functions like ai_classify, ai_extract, and ai_parse_document are optimized for a single task and use Databricks-managed, research-backed systems.
  • ai_query is the general-purpose function — provide your own prompt and choose any supported foundation model. See Use ai_query.

For an end-to-end example, see Analyze customer reviews using AI Functions.

Query agents​

After you build and deploy an agent, query the agent from your application. Agents can be hosted on Databricks Apps or Mosaic AI Model Serving endpoints. Databricks supports three query methods:

  • Databricks OpenAI Client — recommended for new applications, with native streaming and full feature support.
  • OpenAI-compatible REST API — language-agnostic, works with any platform that already speaks the OpenAI API.
  • ai_query — query legacy agents hosted on Model Serving endpoints from SQL.

Foundation models​

Mosaic AI Model Serving hosts open, Databricks-managed, and external foundation models behind a unified API. Choose the deployment option that fits your workload:

  • Pay-per-token — query pre-configured endpoints in your workspace with no infrastructure commitment. Good for experimentation.
  • Provisioned throughput — deploy fine-tuned models with optimized serving and performance guarantees for production.
  • External models — route requests to providers like OpenAI or Anthropic through Databricks governance.
On this page