Skip to main content
Last updated on

ai_classify task for jobs

Beta

This feature is in Beta. To use it, a workspace admin must turn on Lakeflow AI Function Task from the Previews page. See Manage Databricks previews.

To classify text in a job, use an ai_classify task. The task runs ai_classify on each row of a Unity Catalog table. The task classifies text or parsed documents into labels that you define and writes the result for each row to a Unity Catalog table.

An ai_classify task is a type of AI Function task. In the Jobs API and Declarative Automation Bundles, it's an ai_function_task with the ai_classify function set. For requirements and how AI Function tasks process data, see AI Function tasks for jobs.

Add an ai_classify task​

To add an ai_classify task:

  1. Navigate to the Tasks tab in the Jobs UI.
  2. Click Plus icon. Add task. If the job has no tasks yet, click Plus icon. Add another task type.
  3. Under Document intelligence, select Classification.
  4. Enter a name into the Task name field. It defaults to classification.
  5. Under Input, select a table and the Column to classify. To classify documents, parse them first. See Use with other AI functions.
  6. Under Output, select the Location, and optionally enter a Table name. See Output.
  7. (Optional) Add an ID column. See ID column.
  8. Define the labels to classify into. See Define the labels.
  9. (Optional) Configure the settings under Optional settings. See Optional settings.
  10. (Optional) To configure retries, run-duration or streaming-backlog thresholds, or notifications, see Advanced task settings.
  11. Click Save task.

The task runs the next time the job runs. The first run classifies every row in the table. Later runs classify only the rows added or changed since the previous run, and remove the results of deleted rows. To run the job on a schedule, see Run jobs on a schedule. To edit, clone, disable, or delete this task, see Configure and edit tasks in Lakeflow Jobs.

Define the labels​

Under Labels, add each label that an input can be classified as, with a name and an optional description. Label names must be unique. A description helps the function tell similar labels apart.

The function assigns each input the best-matching label, even when no label fits well. To catch inputs that don't fit, add a catch-all label, such as other, and describe what belongs in it. To review uncertain results, turn on Confidence scores.

An ai_classify task takes 2 to 10 labels. The ai_classify function itself takes up to 500 labels, so to use more than 10, call ai_classify in a SQL, notebook, or pipeline task.

Optional settings​

Under Optional settings, you can configure the following:

  • Confidence scores: Include a confidence score for each label in the response.
  • Rationales: Include a rationale that explains each classification in the response.
  • Multiple labels: Assign more than one label to each input.
  • Instructions: Guidance applied to every row, such as which label to choose when two labels fit.
  • Version: The ai_classify version that the task runs, shown for reference. The version is pinned, so scheduled runs don't change when the function is updated.

Define the task in Declarative Automation Bundles​

Declarative Automation Bundles let you define the job in source files, so you can version, review, and deploy changes to it like code. The following example defines a job with an ai_classify task that labels each support ticket as urgent or normal and includes a rationale for each label. To classify documents in a volume, see Use with other AI functions. To create the job with the Jobs API instead, send the same job settings to jobs/create.

YAML
# resources/classify_support_tickets.job.yml
resources:
  jobs:
    classify_support_tickets:
      name: classify_support_tickets
      tasks:
        - task_key: classification
          ai_function_task:
            delta_table:
              table_name: main.support.tickets
              input_column: body
              merge_key_columns: [ticket_id]
            output:
              catalog: main
              schema: support
            ai_classify:
              version: '2.1'
              inline_labels:
                labels:
                  - name: urgent
                    description: The customer is blocked or reports an outage.
                  - name: normal
                    description: A question or a non-blocking issue.
              instructions: Prefer urgent when the ticket mentions production.
              enable_rationales: true

Output​

The response column of the output table holds the full ai_classify result for each row as a VARIANT, including confidence scores and rationales when you turn them on. The task doesn't add a column for each value. To get a value as a column, select it from response in a query. For the format, see ai_classify returns. For the other columns, see Output table.

Additional resources​

On this page