Skip to main content
Last updated on

ai_parse_document task for jobs

Beta

This feature is in Beta. To use it, a workspace admin must turn on Lakeflow AI Function Task from the Previews page. See Manage Databricks previews.

To parse documents in a job, use an ai_parse_document task. The task runs ai_parse_document on each document in a Unity Catalog volume or table. The task parses text, tables, figure descriptions, and layout from documents such as PDFs and images, and writes the result for each document to a Unity Catalog table. For supported file formats and limits, see ai_parse_document.

An ai_parse_document task is a type of AI Function task. In the Jobs API and Declarative Automation Bundles, it's an ai_function_task with the ai_parse_document function set. For requirements and how AI Function tasks process data, see AI Function tasks for jobs.

Add an ai_parse_document task​

To add an ai_parse_document task:

  1. Navigate to the Tasks tab in the Jobs UI.
  2. Click Plus icon. Add task. If the job has no tasks yet, click Plus icon. Add another task type.
  3. Under Document intelligence, select Document Parsing.
  4. Enter a name into the Task name field. It defaults to parsing.
  5. Under Input, select the documents to parse. See Input.
    • Volume: Select a volume or a folder in a volume.
    • Table: Select a table and a FILE or BINARY Column.
  6. Under Output, select the Location, and optionally enter a Table name. See Output.
  7. (Optional) For a table input, add an ID column. See ID column.
  8. In Describe element types, select which parsed elements get an AI-generated description: All supported types, Figures only, or None. None generates no descriptions, which reduces compute and cost for documents with many figures. See Understanding elements.
  9. (Optional) To configure retries, run-duration or streaming-backlog thresholds, or notifications, see Advanced task settings.
  10. Click Save task.

The task runs the next time the job runs. To run the job on a schedule, see Run jobs on a schedule. To edit, clone, disable, or delete this task, see Configure and edit tasks in Lakeflow Jobs.

Define the task in Declarative Automation Bundles​

Declarative Automation Bundles let you define the job in source files, so you can version, review, and deploy changes to it like code. The following example defines a job with an ai_parse_document task that parses each file in a volume and generates descriptions for figures only. To generate no descriptions, leave out description_element_types. To save a rendered image of each page, set image_output_path to a volume path. To create the job with the Jobs API instead, send the same job settings to jobs/create.

YAML
# resources/parse_invoices.job.yml
resources:
  jobs:
    parse_invoices:
      name: parse_invoices
      tasks:
        - task_key: parsing
          ai_function_task:
            volume:
              volume_path: /Volumes/main/finance/invoices
            output:
              catalog: main
              schema: finance
            ai_parse_document:
              version: '2.0'
              description_element_types: [DESCRIPTION_ELEMENT_TYPE_FIGURE]

Output​

The response column of the output table holds the full ai_parse_document result for each document as a VARIANT. The task doesn't add a column for each value. To get a value as a column, select it from response in a query. For the format, see ai_parse_document returns. For the other columns, see Output table.

Use with other AI functions​

To extract fields from or classify the parsed documents, add an ai_extract or ai_classify task that runs after the ai_parse_document task. In the Jobs UI, the new task's input defaults to the ai_parse_document task's output table and response column. See Use the output in a later task.

The following Declarative Automation Bundles example defines a job that parses the files in a volume and then extracts an invoice ID and a total from each parsed document. The extraction task reads the parsing task's output table with {{tasks.parsing.output.table_name}}, reads its response column, and uses its id column as the ID column.

YAML
# resources/parse_and_extract_invoices.job.yml
resources:
  jobs:
    parse_and_extract_invoices:
      name: parse_and_extract_invoices
      tasks:
        - task_key: parsing
          ai_function_task:
            volume:
              volume_path: /Volumes/main/finance/invoices
            output:
              catalog: main
              schema: finance
            ai_parse_document:
              version: '2.0'
        - task_key: extraction
          depends_on:
            - task_key: parsing
          ai_function_task:
            delta_table:
              table_name: '{{tasks.parsing.output.table_name}}'
              input_column: response
              merge_key_columns: [id]
            output:
              catalog: main
              schema: finance
              table: invoice_fields
            ai_extract:
              version: '2.1'
              schema: |
                {
                  "invoice_id": {"type": "string", "description": "Unique invoice identifier"},
                  "total_amount": {"type": "number", "description": "Total invoice amount"}
                }

Additional resources​

On this page