Skip to main content
Last updated on

Analyze files in volumes with a Genie Agent

A Genie Agent can analyze the unstructured files stored in Unity Catalog volumes, such as PDFs, Word documents, slide decks, and images. An agent author attaches volumes to the agent, and business users ask questions about their content alongside the agent's structured data in a single conversation.

Preview

This feature is in Public Preview. Workspace admins can control access to Analyze Files in Volumes with Genie Agents from the Previews page. See Manage Databricks previews.

note

Genie Agents were formerly known as Genie Spaces.

What you can ask a Genie Agent about your files​

By attaching volumes to an agent, business users can combine data from documents such as support tickets, sales call logs, and policy files with data from the agent's structured tables. Users can then answer questions such as:

  • Which support issues are taking the longest to resolve, and what are customers saying in escalation requests?
  • Why are late-stage sales opportunities stalling, and what objections appear most often in sales call logs?
  • Which regions are underperforming, and what internal escalation policies apply?

When to use file analysis in a Genie Agent​

Attach volumes to a Genie Agent when you want a specialized agent for a specific domain. Choose a different approach when your needs are broader:

  • Use Genie One as a unified entry point for all questions about your structured data. Genie One can call a Genie Agent to answer questions about files.
  • Build a custom agent in code with the Agent Bricks CLI only when you need fine-grained control over agent behavior, such as model selection, chunking strategy, or orchestrating multiple sub-agents.
  • Use file upload to blend a local CSV or Excel file into a single conversation instead of analyzing files stored in volumes.

How a Genie Agent analyzes files​

When a user asks a question that requires context from a volume, the Genie Agent retrieves the file content that is most relevant to the question and adds it to the agent's context. The agent then reasons over that content together with any structured data to generate an answer with citations. These citations can point to the specific page of the source file.

File analysis runs only in Agent mode, which uses multi-step reasoning to retrieve and analyze the relevant files. See Agent mode.

The Genie Agent always uses the permissions of the user who asks the question. A user who does not have at least READ VOLUME permission on an attached volume receives an error. Users can ask questions across a specific volume, multiple volumes, and structured data in the same conversation.

Requirements​

The following requirements apply to the agent and its users:

Attach volumes to a Genie Agent​

Before you begin, review the Requirements.

To attach volumes to a Genie Agent:

  1. Open the Genie Agent that you want to add files to.
  2. Open the Sources tab, then click Add.
  3. Select one or more volumes. The Genie Agent processes the entire volume. You cannot attach a subset of the folders or files in a volume.
  4. Add a clear description in the Description field. See Optimize your agent for file questions.

You can attach a volume as a source even when its catalog isn't accessible from this workspace by default, such as a catalog that is bound to specific workspaces.

When you attach a volume, Databricks automatically enables content search.

How content search works​

Content search parses and indexes your files so the Genie Agent can reason over a larger set of files, draw on specific passages across many files, and answer questions with lower latency. For file and volume limits, see Limitations.

Content search stays in sync with your volume automatically. When you add, change, or remove files, Databricks updates the index to reflect your changes, so the Genie Agent reasons over current content without any manual step.

Optimize your agent for file questions​

After you attach your volumes, follow these recommendations to help the Genie Agent use their content:

  • Add a clear description: Describe what content the volume contains, how it is organized, and how the agent should use it. The agent uses this description to retrieve relevant files.
  • Avoid duplicate content: Attaching multiple volumes that contain overlapping information makes it harder for the agent to retrieve relevant documents. The same applies to individual files within a volume.
  • Avoid irrelevant files: Include only files that are relevant to the agent's domain. Irrelevant files can confuse the agent.
  • Use clear file names: Use descriptive file names so the agent can distinguish between files.

Ask questions about files​

To ask questions about files in a Genie Agent:

  1. Open a Genie Agent that has volumes attached.
  2. Turn on Agent mode in the chat box. You must use Agent mode to answer questions about files.
  3. Enter a question and send your prompt. Agent mode retrieves context from the relevant volume, and then generates an answer after the analysis is complete.

Content search supported regions​

Content search is available in the regions listed in this section.

In some regions, content search runs only if an account admin enables cross-Geo processing, which allows Databricks to process your files outside your workspace Geo. See Enable cross-Geo processing.

Regions that don't require cross-Geo processing​

Content search runs in your workspace region for the following regions:

  • Asia Pacific (Sydney)
  • Asia Pacific (Tokyo)
  • EU (Frankfurt)
  • EU (Ireland)
  • EU (London)
  • US East (Northern Virginia)
  • US East (Ohio)
  • US West (Oregon)

Regions that require cross-Geo processing​

For the following regions, an account admin must enable cross-Geo processing before content search runs:

  • Asia Pacific (Mumbai)
  • Asia Pacific (Singapore)
  • Canada (Central)
  • South America (São Paulo)

Limitations​

The following limitations apply:

  • This feature is not available on AWS GovCloud.
  • Only Unity Catalog managed volumes support content search. External volumes and My Files volumes aren't supported. See Store files in My Files.
  • Each workspace can have up to 250 volumes attached to Genie Agents.
  • Each indexed volume can contain up to 10,000 files. To use a larger volume, enable file events. See Use volumes with more than 10,000 files.
  • Content search skips files larger than 100 MB and files with more than 500 pages.
  • Content search supports PDF, JPG, JPEG, PNG, TIFF, TIF, DOC, DOCX, PPT, PPTX, MD, or TXT files. It skips all other file types.

Use volumes with more than 10,000 files​

Without file events, content search supports Unity Catalog managed volumes with up to 10,000 files. To use a managed volume with more than 10,000 files, enable file events on the external location that contains the volume. File events let content search detect new, changed, and deleted files through cloud storage notifications instead of repeatedly scanning your storage. To learn what file events are and how they work, see File events FAQ.

The setup has the following steps:

  1. Find the external location that contains your volume.
  2. Turn on file events for that external location and test the connection.
  3. If the test fails, configure the cloud permissions that file events need. See Set up file events for an external location.

You can skip this setup in the following cases:

  • Your volume uses default storage. To check, open the volume in Catalog Explorer and go to the Details tab. If the Storage location is Default Storage, you can skip this setup.
  • File events are already enabled and working on the external location that contains your volume. To find the external location and check its status, see Step 1: Find the external location that contains your volume.

Step 1: Find the external location that contains your volume​

Permissions required:

  • USE CATALOG on the catalog that contains the volume.
  • USE SCHEMA on the schema that contains the volume.
  • READ METADATA on the volume. READ VOLUME, MANAGE, or ownership of the volume also lets you view its metadata.
  • READ METADATA on each external location that you need to inspect. MANAGE, ownership of the external location, or metastore admin access also qualifies.

  1. In the sidebar, click Data icon. Catalog.

  2. Open the volume that your Genie Agent uses. On the Details tab, find Storage location and copy the cloud path. You use it to find the matching external location. For example:

    s3://bucket-1/6c55cf12…/volumes/de8e8205…

  3. Click Connect > External Locations.

  4. Find the external location whose URL is the beginning of the volume's storage path.

    For example, an external location at s3://bucket-1/6c55cf12… covers a managed volume stored at s3://bucket-1/6c55cf12…/volumes/de8e8205….

  5. Open the matching external location's Overview tab and check Enable file events. If it shows Enabled, click Test connection to confirm that file events are working. If the test passes, you don't need Steps 2 and 3. If it fails, skip to Step 3.

If no external location matches, you might not have access to it. The list shows only external locations that you can access. Ask your metastore admin or the owner of the external location to check. If the volume uses metastore root storage without an external location, you can't enable file events until one exists. Removing metastore-level storage creates one, but after that, every new catalog must have its own storage location. See Remove metastore-level storage.

On the external location's Overview tab in Catalog Explorer, note the storage credential name in the Credential row. You need this name for the permissions in Step 2 and the cloud configuration in Step 3.

Step 2: Turn on file events and test the connection​

Use the external location and storage credential that you found in Step 1.

Permissions required: Metastore admins can complete this step. Otherwise, you need both of the following:

  • MANAGE on the external location, or ownership of the external location.
  • MANAGE on the storage credential, ownership of the credential, or CREATE EXTERNAL LOCATION on both the credential and the metastore that contains the external location.
  1. Go to the external location in Catalog Explorer. Click the Kebab menu icon. kebab menu next to Test connection and select Edit.
  2. Expand Advanced Options. Turn on Enable file events and set File event type to Automatic.
  3. Click Update. If validation fails and you're asked to force the update, cancel and continue with Step 3. A forced update leaves file events enabled but not working.
  4. Wait a few seconds, and then click Test connection to confirm that file events are working. If the test fails and file events were off before this step, turn off Enable file events again so that file arrival triggers on this location keep working. Then continue with Step 3.

Step 3: Configure cloud permissions​

Complete this step if Test connection failed in Step 1 or Step 2, or if file events validation failed when you clicked Update in Step 2.

Follow Set up file events for an external location for the complete cloud configuration instructions. The following notes and examples help you identify the permissions and cloud resources needed for your external location.

Permissions required: Permission to grant access to the cloud identity that your storage credential uses, and to change the storage settings that the guide requires. If you don't have this access, ask your cloud administrator.

To find that cloud identity, open the external location's Overview tab in Catalog Explorer and click the name in the Credential row to view the storage credential's details.

If a cloud administrator makes the change, provide them with the external location's URL and its storage credential details so that they can identify the correct cloud resources and identity.

For example, for an external location on S3 with the URL s3://bucket-1/6c55cf12…, the bucket name is bucket-1.

On the storage credential's Overview tab, copy the IAM role (ARN). For example, arn:aws:iam::123456789123:role/<role-name>.

In the setup guide, copy the three statements in the Automatic policy: ManagedFileEventsSetupStatement, ManagedFileEventsListStatement, and ManagedFileEventsTeardownStatement. Add them in full to the AWS Identity and Access Management (IAM) role's policy.

In ManagedFileEventsSetupStatement, replace <BUCKET> with the bucket name only, without s3:// or the folder path:

JSON
"Resource": [
  "arn:aws:s3:::bucket-1",
  "arn:aws:sqs:*:*:csms-*",
  "arn:aws:sns:*:*:csms-*"
]

After completing the guide's cloud configuration for automatic setup, return to Step 2 to enable file events and test the connection.

Frequently asked questions​

The following questions cover common topics for analyzing files in volumes.

How is this feature billed?​

Databricks charges for the following:

  • Preparing files: When Databricks enables content search or syncs a volume, it parses your files with the ai_parse_document AI function, which is billed under AI Functions pricing.
  • Answering questions: The agent's reasoning over your files is billed as part of standard Genie usage. See Genie pricing.

Can I analyze files in volumes through the API?​

Yes. You can analyze files in volumes through the Agent mode APIs. See the Agent mode API reference.

How can I use volumes with more than 10,000 files?​

To use a Unity Catalog managed volume with more than 10,000 files, enable file events on the external location that contains the volume. See Use volumes with more than 10,000 files.

Additional resources​

See the following resources for related information:

On this page