Use Agent Bricks: Knowledge Assistant to create a high-quality chatbot over your documents

Beta

This feature is in Beta.

This page describes how to use Agent Bricks: Knowledge Assistant to create a question-and-answer chatbot over your documents and improve its quality based on natural language feedback from your subject matter experts.

Agent Bricks provides a simple approach to build and optimize domain-specific, high-quality AI agent systems for common AI use cases.

What is Agent Bricks: Knowledge Assistant?

Use Agent Bricks: Knowledge Assistant to create a chatbot with which you can ask questions on your documents and receive high-quality responses with citations. Knowledge Assistant uses advanced AI and follows a retrieval-augmented generation (RAG) approach to deliver accurate, reliable answers based on the domain-specialized knowledge you provide it.

Agent Bricks: Knowledge Assistant is ideal for supporting the following use cases:

• Answer user questions based on product documentation.
• Answer employee questions related to HR policies.
• Answer customer inquiries based on support knowledge bases.

Knowledge Assistant enables you to improve the chat agent's quality and adjust its behavior based on natural language feedback from your subject matter experts. Provide questions for a labeling session and send it to experts to review in the Review App. Their responses provide labeled data that helps optimize the agent's performance.

Agent Bricks: Knowledge Assistant creates an end-to-end RAG agent endpoint that you can use downstream for your applications. For example, the image below shows how you can interact with the endpoint by chatting with it in AI Playground. Ask the agent questions related to your documents, and the agent will answer with citations.

Agent Bricks uses default storage to store temporary data transformations, model checkpoints, and internal metadata that power each agent. On agent deletion, all data associated with the agent is removed from default storage.

Requirements

• A workspace that includes the following:
  • Mosaic AI Agent Bricks Preview (Beta) enabled. See Manage Databricks previews.
  • Production monitoring for MLflow (Beta) enabled. This is required for tracing to work. See Manage Databricks previews.
  • Serverless compute enabled. See Serverless compute requirements.
  • Unity Catalog enabled. See Enable a workspace for Unity Catalog.
  • Access to Mosaic AI Model Serving.
  • Access to foundation models in Unity Catalog through the system.ai schema.
  • Access to a serverless budget policy with a nonzero budget.
• A workspace in one of the supported regions: us-east-1 or us-west-2.
• The databricks-gte-large-en embedding model endpoint must have AI Guardrails and rate limits disabled. See Configure AI Gateway on model serving endpoints.
• You must have input data ready to use. You can choose to provide either:
  • Files in a Unity Catalog volume or volume directory. Supported file types are txt, pdf, md, ppt/pptx, and docx.
  • A vector search index that uses databricks-gte-large-en as its embedding model. See Create a vector search index.

Create a knowledge assistant agent

Go to Agents in the left navigation pane of your workspace. From the Knowledge Assistant tile, click Build.

Step 1: Configure your agent

On the Configure tab, configure your agent and provide knowledge sources for it to use to answer questions.

1. In the Name field, enter a name for your agent.

2. In the Description field, describe what your agent can do.

3. In the Knowledge source panel, add your knowledge source. You can choose to provide either Unity Catalog files or a vector search index.

   UC Files

   For UC files, the following file types are supported: txt, pdf, md, ppt/pptx, and docx. Files larger than 50 MB are automatically skipped during ingestion and will not be included in the knowledge base.

   

   1. Under Type, select UC Files.
   2. In the Source field, select the Unity Catalog volume or volume directory that contains your files.
   3. In the Name field, enter a name for your knowledge source.
   4. Under Describe the content, describe what content the knowledge source contains to help the agent understand when to use this data source.

   Vector Search Index

   

   note

   Vector search indexes are only supported if the index uses databricks-gte-large-en as its embedding model. When creating your vector search index, ensure you select this embedding model. For more information, see Create a vector search index.

   1. Under Type, select Vector Search Index.
   2. In the Source field, select the vector search index you want to provide the agent.
   3. In the Doc URI Column, select the column with a link or reference to where the information came from. The agent will use this in its citations.
   4. In the Text Column field, specify the column that contains the raw text you want the agent to retrieve.
   5. In the Name field, enter a name for your knowledge source.
   6. Under Describe the content, describe what content the knowledge source contains to help the agent understand when to use this data source.

4. (Optional) If you would like to add more knowledge sources, click Add knowledge source. You can provide up to 10 knowledge sources.

5. (Optional) In the Instructions field, specify guidelines for how the agent should respond.

   

6. Click Create Agent.

It can take up to a few hours to create your agent and sync the knowledge sources you provided. The right side panel will update with links to the deployed agent, experiment, and synced knowledge sources.

important

If you update or add files to your knowledge sources, you need to click Sync for the agent to pick up the changes. Syncing is done incrementally. For example, if you add a new file to a previously synced Unity Catalog volume, syncing will only process the newly added file.

Only the creator of the knowledge assistant can sync knowledge sources.

Step 2: Test your agent

After your agent has finished building, test it out by chatting with it. The agent should respond with citations for questions related to its knowledge sources.

1. Under Test your agent, start chatting with your agent.

2. (Optional) You can also click Open in Playground to chat with it in AI Playground. If you have AI assistive features enabled, you can enable AI Judge and Synthetic question generation to help you evaluate your agent.

3. Enter a question for your agent.

4. Evaluate its response:

   1. Click View thoughts to see how your agent approached responding to the question.
   2. Click View sources to see what files the agent is citing. This opens up side panel with a list of sources for you to review.
   3. Click View trace to see the full trace. You can add labels to traces in the UI to track quality assessments during the development process.

If you're satisfied with your agent's performance, continue using the agent as-is. By default, Agent Bricks endpoints scale to zero after 3 days of inactivity, so you'll only be billed for the uptime.

Step 3: Improve quality

Agent Bricks: Knowledge Assistant can adjust the agent's behavior based on natural language feedback. Gather human feedback through a labeling session to improve your agent's quality. Collecting labeled data for your agent can improve its quality. Agent Bricks will retrain and optimize the agent from the new data. To learn more about collecting feedback, see Domain expert feedback.

In the Improve Quality tab, add questions and start a labeling session. Alternatively, you can also import labeled data directly from a Unity Catalog table.

1. Add questions to include in your labeling session:

   1. Click + Add to add a question.
   2. In the Add a question modal, enter your question.
   3. Click Add. The question should appear in the UI.
   4. Repeat until you’ve added all the questions you want to evaluate.
   5. To delete a question, click the kebab menu, then Delete.

   

2. After you’ve finished adding your questions, send the questions to experts for review to help you build a high-quality labeled dataset. On the right, click Start labeling session.

   When your labeling session is ready, the UI will update as shown below.

   

3. Share the review app with experts to gather feedback.

   To learn more about the Review App and labeling sessions, see Collect feedback and expectations by labeling existing traces and Create and manage Labeling Sessions.

   note

   In order for experts to access the labeling session, you need to grant them the following permissions:

   • CAN QUERY permission to the endpoint
   • EDIT permission to the experiment
   • USE CATALOG, USE SCHEMA, and SELECT permissions to the schema

4. To label the data yourself, click Open labeling session.

   This opens the review app in a new tab. As a reviewer:

   1. Click Start review. For each question, the reviewer will see the question and the agent's response.

   2. On the left side, review the question and answer. You can click View thoughts to see how the agent is thinking about the question.

   3. On the right side, under Expectations, review any existing guidelines and add more as you see fit.

      1. To add a guideline, click + Add input.
      2. Enter the guideline in the text box that appears.
      3. Click Save.

   4. When you’re done reviewing a question, click Next unreviewed > in the top right to move onto the next one.

   5. When you’re done reviewing all questions, simply exit the review app.

      

5. When your reviewers are done with their labeling sessions, return to your agent’s Improve Quality tab.

6. Click Merge to merge feedback from the experts to your labeled dataset. The table of questions on the right side will update with the merged feedback.

   

7. Test the agent again in AI Playground to see its improved performance. If needed, start another labeling session to gather more labeled data.

(Optional) Import and export labeling session data

To import new questions and feedback directly from a Unity Catalog table:

1. Click Import.

2. In the Source field, select the Unity Catalog table containing the labeled data.

   The table must have the following schema:

   • eval_id: string
   • request: string
   • guidelines: array
     • items: string
   • metadata: string
   • tags: string

3. Click Import.

New questions and guidelines are merged into the labeled data table on the right.

To export feedback data from the labeling session as a Unity Catalog table:

1. Click Export.
2. In the Schema field, select the Unity Catalog schema location to save the data to.
3. In the Table name field, enter a name for the table.
4. Click Export.

A new table is created with the feedback data from the labeling session.

Manage permissions

By default, only Agent Bricks authors and workspace admins have permissions to the agent. To allow other users to edit or query your agent, you need to explicitly grant them permission.

To manage permissions on your agent:

1. Open your agent in Agent Bricks.
2. At the top, click the kebab menu.
3. Click Manage permissions.
4. In the Permission Settings window, select the user, group, or service principal.
5. Select the permission to grant:
   • Can Manage: Allows managing the Agent Bricks, including setting permissions, editing the agent configuration, and improving its quality.
   • Can Query: Allows querying the Agent Bricks endpoint in AI Playground and through the API. Users with only this permission cannot view or edit the agent in Agent Bricks.
6. Click Add.
7. Click Save.

note

For agent endpoints created before September 16, 2025, you can grant Can Query permissions to the endpoint from the Serving endpoints page.

important

Only the creator of the knowledge assistant can sync knowledge sources.

Query the agent endpoint

There are multiple ways to query the created knowledge assistant endpoint. Use the code examples provided in AI Playground as a starting point.

1. On the Configure tab, click Open in playground.
2. From Playground, click Get code.
3. Choose how you want to use the endpoint:
   • Select Apply on data to create a SQL query that applies the agent to a specific table column.
   • Select Curl API for a code example to query the endpoint using curl.
   • Select Python API for a code example to interact with the endpoint using Python.

Evaluate your knowledge assistant

This notebook demonstrates how to evaluate a Databricks Knowledge Assistant using curated evaluation datasets and custom scorers.

Notebook

Open notebook in new tab

Limitations

• .doc files are not supported.

• Files larger than 50 MB are automatically skipped during ingestion and are not included in the knowledge base.

• Workspaces that have Enhanced Security and Compliance enabled are not supported.

• Unity Catalog tables are not supported.

• Only vector search indexes that use databricks-gte-large-en as the embedding model are supported.

• AI Guardrails and rate limits must be disabled on the databricks-gte-large-en model endpoint. See Configure AI Gateway on model serving endpoints.

• In order for tracing to work, you must enable production monitoring for MLflow (Beta). See Manage Databricks previews.