Skip to main content

Information Extraction

note

This page covers the new version of Information Extraction. For information about the previous version, see Use Information Extraction (legacy)

Information Extraction transforms unstructured documents and text into key, structured insights using a defined schema. This lets you use information embedded in unstructured text, PDFs, images, or tables directly for analysis, reporting, or downstream agents and applications.

Examples of Information Extraction include:

  • Extracting legal parties and terms from contracts.
  • Extracting line items and payment terms from invoices.
  • Pulling key details from medical records and notes.

Information Extraction uses default storage to store temporary data transformations, model checkpoints, and internal metadata that power each agent. When you delete an agent, Databricks removes all data associated with the agent from default storage.

Information extraction is available through the Agent Bricks UI, in SQL, with the REST API, and with the Asynchronous API (Beta).

Create an information extraction agent in the UI​

Requirements​

Go to Agents icon. Agents in the left navigation pane of your workspace. Click Create Agent > Information Extraction.

Step 1. Select the data to extract information from​

  1. On the Start with your data page, select the files or data you want to extract information from. You can do any of the following:

    • Drag and drop one or more files into the upload area, or click to browse for files to upload.
    • Click Select volume to select a Unity Catalog volume with supported file types.
    • Click Select table to select a Unity Catalog table that contains text data.
  2. If you select a table, select the column that contains the data to extract from. You must select a column with a supported type, such as STRING or VARIANT, before you can continue. If the table has no supported columns, select a different table.

  3. Click Create Agent. This button is enabled only after you select a valid data source, and, for a table, a supported column.

Step 2. Configure and refine your extraction schema​

After Information Extraction processes your data, configure and refine what data you want to extract from your documents.

The AI Extract build view with the extraction schema configuration panel, the parsed document, and the extracted JSON output.

  1. Under Configuration, define your extraction schema. There are several ways to do this:

    • Enter natural language that describes the information you want to extract and click Generate Schema. Information Extraction automatically generates a JSON schema with field names and definitions for you. Edit these descriptions as needed.
    • Alternatively, click Or, Define manually to manually define your schema:
      1. Click Add field.
      2. Enter your field name, type, and description.
      3. Click Confirm.
      4. Repeat for each field you want to extract.
      5. Click Save and run extraction.
    • You can also click JSON to edit the JSON schema directly. Click Apply Changes when complete.

    Each time you update your schema and click Save and run extraction, Information Extraction updates the extraction agent, runs the extraction, and shows the results for each input.

    To configure additional options, click the options toggle Sliders icon. next to Save and run extraction, then enable one or more of the following options:

    • Precision mode: Improves accuracy for nuanced fields and long documents.
    • Citations: Shows source references for extracted values.
    • Confidence scores: Shows confidence levels for extracted values.
  2. On the left, review the parsed document and the agent's extraction. Iterate the extraction results in two ways. First, provide natural language feedback on one or more inputs, which auto-tunes your descriptions when you press Save and run extraction. Second, manually revise the schema descriptions, which take effect when you press Save and run extraction.

  3. Use versions to compare or revert to a previous configuration. Click Versions, then click Compare to compare the schema definition of a previous version with the current version. Click Restore to restore a previous version.

Step 3. Evaluate and improve extraction quality​

To measure how well your agent performs and improve it systematically, evaluate the agent against a labeled dataset. An evaluation scores each extraction against a known correct answer (ground truth), so you can track field-level accuracy across versions and target the fields that need work.

Add an evaluation dataset​

Before running an evaluation, you need to have an evaluation dataset in Unity Catalog. The dataset must be a Unity Catalog table with two columns:

  • An input column with the text or document to extract from. This can be STRING text or the VARIANT output of ai_parse_document.
  • A ground truth column with the expected extraction result as a JSON string. Each value must conform to your agent's extraction schema, matching the ai_extract advanced schema.

Run an evaluation​

To run an evaluation for your extraction agent:

  1. In the workbench, open the evaluation dropdown and click Run evaluation. This opens the Create evaluation run dialog.
  2. In Evaluation dataset table, select the Unity Catalog table that holds your labeled examples.
  3. Under Column mapping, select the Input column that holds the agent input and the Ground truth column that holds the expected response.
  4. Click Run evaluation. Information Extraction saves the current configuration as a new version and scores each row against its ground truth.

Review the evaluation results​

As the run progresses, documents stream into the Documents tab and any mismatched fields are shown per document. When the run finishes, Information Extraction shows the Overview tab, which includes:

  • A scorecard with Overall field accuracy and Documents fully correct. If a previous evaluation run exists, the scorecard shows the change from that run.
  • A Per-field scores table. Click any field to open its drilldown, which shows accuracy, precision, recall, and F1. Click Show failed documents to see each document that failed to extract that field and their extracted output.

Evaluation Overview tab showing overall field accuracy, documents fully correct, Genie Code recommendations, and the per-field scores table.

Switch to the Documents tab to inspect individual documents. Each row shows the ratio of matched fields and the fields that were mismatched. Use the Fields dropdown to filter on documents that had a specific mismatched field. Click a document to compare the agent response with the ground truth side by side.

Apply Genie Code fixes​

After a run finishes, Information Extraction analyzes the results and surfaces issues in Genie Code recommendations, ranked by number of affected documents. Each finding describes a potential quality issue. Click Fix with Genie to open an interactive Genie Code chat that proposes schema and instruction changes, validates them against the affected documents, and offers to rerun a full evaluation.

Genie Code panel showing proposed schema and instruction changes being validated against affected documents.

Step 4. Use your extraction agent​

After you're happy with the agent's performance, use the agent to extract information.

Click Use Agent in the upper-right. You can select either:

  • Run in SQL to use the agent to extract information from all your data. This opens a SQL query that uses ai_extract to extract information from your volume or table using the schema defined. For more information on using ai_extract in SQL queries, see ai_extract function.
  • Create a Lakeflow pipeline that runs on scheduled intervals to invoke your agent on new data. This creates a Lakeflow pipeline that updates a streaming table with your extracted data. You can configure the pipeline's schedule to run when new data arrives. For more information on Lakeflow pipelines, see Spark Declarative Pipelines on Lakeflow.

Extract information in SQL​

Call ai_extract in SQL to extract fields at scale:

SQL
SELECT ai_extract(
'Invoice #12345 from Acme Corp for $1,250.00 dated 2024-01-15',
'{
"invoice_id": {"type": "string", "description": "Unique invoice identifier"},
"vendor_name": {"type": "string", "description": "Legal business name"},
"total_amount": {"type": "number", "description": "Total invoice amount"},
"invoice_date": {"type": "string", "description": "Date in YYYY-MM-DD format"}
}',
options => map('version', '2.1')
);

See the ai_extract SQL reference.

Extract information with the REST API​

Use the REST API to extract structured fields from applications, services, and automation workflows. See the REST API docs for reference.

The default rate limit for ai_extract REST API requests is 120 requests per minute per workspace. Contact your Databricks account team to request a higher limit.

The following example extracts structured fields from invoice text:

Bash
curl -X POST "$DATABRICKS_HOST/api/2.0/ai-functions/ai-extract" \
-H "Authorization: Bearer $DATABRICKS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"content": "Invoice #12345 from Acme Corp for $1,250.00 dated 2024-01-15",
"schema": {
"vendor_name": {
"type": "string"
},
"total_amount": {
"type": "number"
}
}
}'

Extract information with the Asynchronous API (Beta)​

For batch processing, Databricks recommends AI Function tasks in Lakeflow Jobs. To extract fields in a job, use an ai_extract task, which does the following:

  • Runs the processing in the background and writes the results to a Unity Catalog table, so you don't submit batches, check their status, or download output files.
  • Processes only the rows that are new or changed since the previous run.
  • Shows the progress of each run, and how many inputs succeeded and failed.

Use the AI Functions Asynchronous API (Beta) instead when you want to integrate the processing into an existing application or an external orchestrator, and manage the submissions and results yourself.

Beta

This feature is in Beta. To use it, a workspace admin must turn on AI Functions Asynchronous API from the Previews page. See Manage Databricks previews.

For the request and response fields, and for the requests to cancel and list batches, see the Asynchronous API reference.

Input file​

The input file of an ai_extract batch is a JSONL file in a Unity Catalog volume, with one request on each line. Each line is a JSON object with exactly the following fields:

Field

Description

custom_id

A string that identifies the request. Each line of the output file has the custom_id of its request, so use a value that you can match the results to, such as the key of the source row.

body

The text to extract fields from, in the form {"text": "..."}.

Field

Description

custom_id

A string that identifies the request. Each line of the output file has the custom_id of its request, so use a value that you can match the results to, such as the key of the source row.

body

The text to extract fields from, in the form {"text": "..."}.

The following input file has two requests:

JSON
{"custom_id": "invoice-001", "body": {"text": "Invoice #12345 from Acme Corp for $1,250.00 dated 2024-01-15"}}
{"custom_id": "invoice-002", "body": {"text": "Invoice #12346 from Globex for $980.00 dated 2024-02-03"}}

The input file has only the text and ID of each record. You set the other parameters of ai_extract in params when you create the batch, and they apply to every request:

  • schema (required): The fields to extract, as a list of field names or as an object that maps each field to its type and description.
  • options: The function options, such as instructions and mode.

For what each parameter does, see the AI Extract REST API reference. In params.options, write each option name in camelCase, for example enableCitations rather than enable_citations.

When you create a batch, the API checks the whole input file and params first. If the input file or params breaks one of the following rules, the request fails with HTTP 400 and no batch is created:

  • The file is UTF-8 encoded, and each line is one JSON object with exactly the fields custom_id and body. The file has no blank lines.
  • Each custom_id is a non-empty string, and no two lines have the same custom_id.
  • Each body has exactly one key, text.
    • If a text value isn't a string, the batch is still created, and that request fails with the error type invalid_request.
  • The file has from 1 to 50,000 lines and is at most 200 MiB.
    • To process more records, split them into several input files and create a batch for each file.
  • params has a schema and no parameters that ai_extract doesn't take.

For a problem in a line of the input file, the error message gives the line number and the rule, for example Batch input line 3 custom_id must be unique. If you can't read the input file or write to output_ref, the request fails with HTTP 403.

The API copies the input file when it creates the batch, so changes that you make to the file afterward don't affect the batch.

Run a batch​

To extract fields with the Asynchronous API:

  1. Create the input file, and upload it to a Unity Catalog volume, for example with the Files API.

  2. Create a batch that runs ai_extract on the input file. Set input_ref to the path of the input file, params to the parameters for every request, and output_ref to a volume directory or a new file path for the output file:

    Bash
    curl -X POST "$DATABRICKS_HOST/api/2.0/ai-functions/batches" \
    -H "Authorization: Bearer $DATABRICKS_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
    "endpoint": "ai_extract",
    "input_ref": "/Volumes/my_catalog/my_schema/my_volume/requests.jsonl",
    "params": {
    "schema": {
    "vendor_name": {
    "type": "string"
    },
    "total_amount": {
    "type": "number"
    }
    }
    },
    "output_ref": "/Volumes/my_catalog/my_schema/my_volume/output/"
    }'

    The response contains the batch name. Use it in place of <batch-name> in the next steps. By default, a workspace can have up to 20 unfinished batches at a time, across all users. If it's at the limit, the request fails with HTTP 409. Create the batch again after another batch finishes or is cancelled.

  3. Check the status of the batch. The batch is finished when status is COMPLETED, CANCELLED, or FAILED:

    Bash
    curl -X GET "$DATABRICKS_HOST/api/2.0/ai-functions/batches/<batch-name>" \
    -H "Authorization: Bearer $DATABRICKS_TOKEN"

    When the output file is ready, output_ref in the response contains its path.

  4. Download the output file with the Files API:

    Bash
    curl -X GET "$DATABRICKS_HOST/api/2.0/fs/files/Volumes/my_catalog/my_schema/my_volume/output/<batch-name>_output.jsonl" \
    -H "Authorization: Bearer $DATABRICKS_TOKEN" \
    -o output.jsonl

Output file​

The Asynchronous API writes the output file of an ai_extract batch to the location in output_ref:

  • If output_ref ends with / or names an existing directory, the API writes the output file to <batch-name>_output.jsonl in that directory. Several batches can share one output directory.
  • Otherwise, output_ref is the path of the output file. The file must not exist when you create the batch.

The output file has one line for each line of the input file. The lines are in the same order as the input file. Each line has the custom_id of its request and one of the following fields:

  • response: The result of the request. It has the same fields as the AI Extract REST API response.
  • error: The type and message of the error, if the request failed. If the batch was cancelled before the request ran, the type is cancelled.

The following example shows a line for a request that succeeded and a line for a request that failed:

Text
{"custom_id": "invoice-001", "response": {...}}
{"custom_id": "invoice-002", "error": {"type": "request_failed", "message": "..."}}

Limitations​

  • Information Extraction agents have a 128k token max context length.
  • Union schema types are not supported.