Use Agent Bricks: Knowledge Assistant to create a high-quality chatbot over your documents

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

This page describes how to use Agent Bricks: Knowledge Assistant to create a question-and-answer chatbot over your documents and improve its quality based on natural language feedback from your subject matter experts.

Agent Bricks provides a simple approach to build and optimize domain-specific, high-quality AI agent systems for common AI use cases.

What is Agent Bricks: Knowledge Assistant?

Use Agent Bricks: Knowledge Assistant to create a chatbot that can answer questions about your documents and provide high-quality responses with citations. Knowledge Assistant uses advanced AI and follows an Instructed Retriever approach to address limitations of the traditional retrieval-augmented generation (RAG) approach and deliver the highest quality answers based on the documentation you provide.

Agent Bricks: Knowledge Assistant is ideal for supporting the following use cases:

• Answer user questions based on product documentation.
• Answer employee questions related to HR policies.
• Answer customer inquiries based on support knowledge bases.

Knowledge Assistant enables you to improve the chat agent's quality and adjust its behavior based on natural language feedback from your subject matter experts. Provide questions and guidelines directly in the experience, and share your agent to let others collaborate and improve the agent's performance.

Agent Bricks: Knowledge Assistant creates an agent endpoint that you can use downstream for your applications. For example, the image below shows how you can interact with the endpoint by chatting with it in AI Playground. Ask the agent questions related to your documents, and the agent will answer with citations.

Agent Bricks uses default storage to store temporary data transformations, model checkpoints, and internal metadata that power each agent. On agent deletion, all data associated with the agent is removed from default storage.

Requirements

• A workspace that includes the following:
  • Mosaic AI Agent Bricks Preview (Beta) enabled. See Manage Databricks previews.
  • Production monitoring for MLflow (Beta) enabled. This is required for tracing to work. See Manage Databricks previews.
  • Serverless compute enabled. See Serverless compute requirements.
  • Unity Catalog enabled. See Enable a workspace for Unity Catalog.
  • Access to Mosaic AI Model Serving.
  • Access to foundation models in Unity Catalog through the system.ai schema.
  • Access to a serverless budget policy with a nonzero budget.
• A workspace in one of the supported regions: us-east-1 or us-west-2.
• The databricks-gte-large-en embedding model endpoint must have AI Guardrails and rate limits disabled. See Configure AI Gateway on model serving endpoints.
• You must have input data ready to use. You can choose to provide either:
  • Files in a Unity Catalog volume or volume directory. Supported file types are txt, pdf, md, ppt/pptx, and doc/docx.
  • A vector search index that uses databricks-gte-large-en as its embedding model. See Create a vector search index.

Create a knowledge assistant agent

Go to Agents in the left navigation pane of your workspace. From the Knowledge Assistant tile, click Build.

Step 1: Configure your agent

On the Build tab, configure your agent and provide knowledge sources for it to use to answer questions.

1. In the Name field, enter a name for your agent.

2. In the Description field, describe what your agent can do.

3. In the Knowledge source panel, add your knowledge source. You can choose to provide either Unity Catalog files or a vector search index.

   UC Files

   For UC files, the following file types are supported: txt, pdf, md, ppt/pptx, and doc/docx. Files larger than 50 MB are automatically skipped during ingestion and will not be included in the knowledge base.

   

   1. Under Type, select UC Files.
   2. In the Source field, select the Unity Catalog volume or volume directory that contains your files.
   3. In the Name field, enter a name for your knowledge source.
   4. Under Describe the content, describe what content the knowledge source contains to help the agent understand when to use this data source.

   Vector Search Index

   

   note

   Vector search indexes are only supported if the index uses databricks-gte-large-en as its embedding model. When creating your vector search index, ensure you select this embedding model. For more information, see Create a vector search index.

   1. Under Type, select Vector Search Index.
   2. In the Source field, select the vector search index you want to provide the agent.
   3. In the Doc URI Column, select the column with a link or reference to where the information came from. The agent will use this in its citations.
   4. In the Text Column field, specify the column that contains the raw text you want the agent to retrieve.
   5. In the Name field, enter a name for your knowledge source.
   6. Under Describe the content, describe what content the knowledge source contains to help the agent understand when to use this data source.

4. (Optional) If you would like to add more knowledge sources, click Add knowledge source. You can provide up to 10 knowledge sources.

5. (Optional) In the Instructions field, specify guidelines for how the agent should respond.

   

6. Click Create Agent.

It can take up to a few hours to create your agent and sync the knowledge sources you provided. The right side panel will update with links to the deployed agent, experiment, and synced knowledge sources.

important

If you update or add files to your knowledge sources, you need to click Sync for the agent to pick up the changes. Syncing is done incrementally. For example, if you add a new file to a previously synced Unity Catalog volume, syncing will only process the newly added file.

Only the creator of the knowledge assistant can sync knowledge sources.

Step 2: Test your agent

After your agent has finished building, test it out by chatting with it. The agent should respond with citations for questions related to its knowledge sources.

1. Under Test your agent, start chatting with your agent.

2. (Optional) You can also click Open in Playground to chat with it in AI Playground. If you have AI assistive features enabled, you can enable AI Judge and Synthetic question generation to help you evaluate your agent.

3. Enter a question for your agent.

4. Evaluate its response:

   1. Click View thoughts to see how your agent approached responding to the question.
   2. Click View sources to see what files the agent is citing. This opens up side panel with a list of sources for you to review.
   3. Click View trace to see the full trace. You can add labels to traces in the UI to track quality assessments during the development process.

If you're satisfied with your agent's performance, continue using the agent as-is. By default, Agent Bricks endpoints scale to zero after 3 days of inactivity, so you'll only be billed for the uptime.

Step 3: Improve quality

Agent Bricks: Knowledge Assistant can adjust the agent's behavior based on natural language feedback. Gather feedback from experts through the configuration page to improve your agent's quality. Collecting labeled data for your agent can improve its quality and behavior.

In the Examples tab, add questions your users will ask or questions your agent has previously gotten wrong. Alternatively, you can also import labeled data directly from a Unity Catalog table.

1. Add questions to label:

   1. Click + Add to add a question.
   2. In the Add a question modal, enter your question.
   3. Click Add. The question should appear in the UI.
   4. Repeat until you’ve added all the questions you want to evaluate.
   5. To delete a question, click the kebab menu, then Delete.

2. After you've finished adding your questions, you can share the agent with others to review to help you build a high-quality labeled dataset. In the upper righthand corner, click the kebab menu to manage permissions.

   In order for experts to access and provide feedback, you need to grant them the following permissions:

   • CAN_MANAGE permission from the agent configuration page

3. Share a link to your Agent Bricks: Knowledge Assistant configuration page to gather feedback from experts. Collected guidelines apply right after they are saved.

4. To label data yourself, click on a question and add Guidelines in the panel that appears. Guidelines apply right after they are saved.

5. Test the agent again in the configuration page or AI Playground to see its improved performance. If needed, add more questions and guidelines to continue improving the behavior.

(Optional) Import and export labeled data

To import new questions and feedback directly from a Unity Catalog table:

1. Click Import.

2. In the Source field, select the Unity Catalog table containing the labeled data.

   The table must have the following schema:

   • eval_id: string
   • request: string
   • guidelines: array
     • items: string
   • metadata: string
   • tags: string

3. Click Import.

New questions and guidelines are merged into the labeled data table on the right.

To export labeled data as a Unity Catalog table:

1. Click Export.
2. In the Schema field, select the Unity Catalog schema location to save the data to.
3. In the Table name field, enter a name for the table.
4. Click Export.

A new table is created with the labeled data.

Manage permissions

By default, only Agent Bricks authors and workspace admins have permissions to the agent. To allow other users to edit or query your agent, you need to explicitly grant them permission.

To manage permissions on your agent:

1. Open your agent in Agent Bricks.
2. At the top, click the kebab menu.
3. Click Manage permissions.
4. In the Permission Settings window, select the user, group, or service principal.
5. Select the permission to grant:
   • Can Manage: Allows managing the Agent Bricks, including setting permissions, editing the agent configuration, and improving its quality.
   • Can Query: Allows querying the Agent Bricks endpoint in AI Playground and through the API. Users with only this permission cannot view or edit the agent in Agent Bricks.
6. Click Add.
7. Click Save.

important

Only the creator of the knowledge assistant can sync knowledge sources.

Query the agent endpoint

On the agent page, click See Agent status in the upper-right to get your deployed agent endpoint and see endpoint details.

There are multiple ways to query the created knowledge assistant endpoint. Use the code examples provided in AI Playground as a starting point:

1. On the Build tab, click Open in playground.
2. From Playground, click Get code.
3. Choose how you want to use the endpoint:
   • Select Curl API for a code example to query the endpoint using curl.
   • Select Python API for a code example to interact with the endpoint using Python.

Evaluate your knowledge assistant

This notebook demonstrates how to evaluate a Databricks Knowledge Assistant using curated evaluation datasets and custom scorers.

Notebook

Open notebook in new tab

Limitations

• Files larger than 50 MB are automatically skipped during ingestion and are not included in the knowledge base.

• Files with names that start with an underscore (_) or period (.) are automatically skipped during ingestion and are not included in the knowledge base.

• Workspaces that have Enhanced Security and Compliance enabled are not supported.

• Unity Catalog tables are not supported.

• Only vector search indexes that use databricks-gte-large-en as the embedding model are supported.

• AI Guardrails and rate limits must be disabled on the databricks-gte-large-en model endpoint. See Configure AI Gateway on model serving endpoints.

• In order for tracing to work, you must enable production monitoring for MLflow (Beta). See Manage Databricks previews.