Skip to main content
Last updated on

Content search for Unity Catalog volumes

Content search prepares and indexes the documents in a Unity Catalog volume. Searches and agent questions then run against the index, so users and agents retrieve the most relevant information faster and more accurately. Because the index covers the content within each file, an agent can draw on specific passages across many files instead of retrieving a handful of whole files.

Content search is especially useful for volumes attached to a Genie Agent. See Analyze files in volumes with a Genie Agent.

Beta

This feature is in Beta. To use it, a workspace admin must turn on Analyze Files in Volumes with Genie Agents from the Previews page. See Manage Databricks previews.

Requirements​

Before you enable content search for a Unity Catalog volume, confirm that you meet the following requirements:

Regional availability​

Content search is available in the regions listed in this section. It isn't available in regions that aren't listed. In some regions, it runs only if an account admin enables cross-Geo processing, which allows Databricks to process your files outside your workspace Geo. See Enable cross-Geo processing.

Regions that don't require cross-Geo processing​

Content search runs in your workspace region for the following regions:

  • us-central1 (Iowa)
  • us-east1 (South Carolina)
  • us-east4 (Virginia)
  • us-west1 (Oregon)

Regions that require cross-Geo processing​

For the following regions, an account admin must enable cross-Geo processing before you can enable content search:

  • asia-northeast1 (Tokyo)
  • asia-south1 (Mumbai)
  • asia-southeast1 (Singapore)
  • australia-southeast1 (Sydney)
  • europe-west1 (Belgium)
  • europe-west2 (London)
  • europe-west3 (Frankfurt)
  • northamerica-northeast1 (Montreal)
  • southamerica-east1 (Sao Paulo)
  • us-west4 (Nevada)

Enable content search​

You enable and disable content search from the volume's page in Catalog Explorer.

To enable content search:

  1. In your Databricks workspace, click Data icon. Catalog.
  2. Search or browse for the volume that contains the files you want to index and select it.
  3. On the Overview tab, find the Content search section in the right sidebar.
  4. Click Enable.

After you enable content search, Databricks runs an initial sync to parse and index the supported files in the volume. The Content search section on the Overview tab shows the ingestion status, the result of the latest sync, and the time through which the volume is indexed.

Content search doesn't automatically parse and index changes to your volume, such as newly added files or edits to existing files. To refresh content search so that it reflects your changes, click Sync now in the Content search section. You need CAN MANAGE on the volume to trigger a sync.

To disable content search, click Disable in the Content search section. Disabling content search deletes the generated index and the processed content. If you enable it again, Databricks reindexes the volume from scratch and you incur the ingestion cost again.

Databricks recommends enabling content search for the volumes that you attach to a Genie Agent. See Analyze files in volumes with a Genie Agent.

How content search is billed​

Content search has two cost components:

  • Ingestion: When you enable content search or run a sync, Databricks uses the ai_parse_document AI function to parse the files in the volume before it builds the index. Databricks bills this processing under AI Functions pricing.
  • Query serving: Lakebase stores and serves the resulting indexes. Databricks bills compute and storage based on the compute units (CUs) that Lakebase provisions. Query serving scales from 2 to 12 CUs based on index size and query concurrency. After three days without traffic, the endpoint scales to zero and stops incurring query-serving compute costs. For current rates, see the Lakebase pricing page.

When a Genie Agent queries an indexed volume, the query can also incur Genie Agent costs. See How is this feature billed?.

Limitations​

The following limitations apply to content search during the Beta:

  • Only Unity Catalog managed volumes support content search. External volumes, My Files volumes, and volumes in catalogs with workspace bindings aren't supported. See Store files in My Files.
  • You enable content search for an entire volume. You can't index a subset of the folders or files in a volume.
  • Each workspace can have up to 100 volumes with content search enabled.
  • Each indexed volume can contain up to 10,000 files.
  • Content search skips files larger than 50 MB.
  • Content search supports PDF, JPG, JPEG, PNG, TIFF, TIF, DOC, DOCX, PPT, or PPTX files. It skips all other file types.
  • Content search isn't available on AWS GovCloud, in Azure Government regions, or in workspaces with FedRAMP compliance.

Additional resources​

See the following resources for related information:

On this page