Skip to main content

Use AI Builder: Model specialization to create a gen AI agent for text

Beta

This feature is in Beta.

This article describes how to create a generative AI agent for custom text-based tasks using AI Builder: Model specialization.

AI Builder provides a simple, no-code approach to build and optimize domain-specific, high-quality AI agent systems for common AI use cases.

What can you do with Model specialization?​

Use AI Builder: Model specialization to generate high-quality results for any domain-specific task, such as summarization, classification, text transformation, and content generation.

Examples of model specialization include:

  • Summarizing the issue and resolution of customer calls.
  • Analyzing the sentiment of customer reviews.
  • Classifying research papers by topic.
  • Generating press releases for new features.

Given high-level instruction and examples, AI Builder: Model specialization optimizes prompts on behalf of users, automatically infers evaluation criteria, evaluates the system from provided data, and deploys the model as a productionizable endpoint.

AI Builder: Model specialization leverages automated evaluation capabilities, including MLflow and Agent Evaluation, to enable rapid assessment of the cost-quality tradeoff for your specific extraction task. This assessment allows you to make informed decisions about the balance between accuracy and resource investment.

Requirements​

  • Serverless-supported workspace that includes the following:
  • Ability to use the ai_query SQL function.
  • You must have input data ready to use. You can choose to provide either:
    • A Unity Catalog table. The table name cannot contain any special characters (such as -).
    • At least 3 example inputs and outputs. If you choose this option, you'll need to specify a Unity Catalog schema destination path for the agent, and you must have CREATE REGISTERED MODEL and CREATE TABLE permissions to this schema.
  • If you want to optimize your agent, you need at least 100 inputs (either 100 rows in a Unity Catalog table or 100 manually-provided examples).

Create a model specialization agent​​

Go to AI Builder in the left navigation pane of your workspace and click Model Specialization.

AI Builder: model specialization task.

Step 1: Configure your agent​

On the Configure tab, click Show an example > to expand an example input and model response for a model specialization agent.

In the pane below, configure your agent:

  1. Under Describe your task, enter a clear and detailed description of your specialization task, including its purpose and desired outcome.

  2. Provide a labeled dataset, an unlabeled dataset, or a few examples to use to create your agent.

    The following data types are supported: string, int, and double.

    If you select Labeled dataset:

    Model Specialization: select labeled dataset.

    1. Under Select dataset as UC table, click Browse to select the table in Unity Catalog you want to use. The table name cannot contain any special characters (such as -).

      The following is an example:

      main.model_specialization.customer_call_transcripts

    2. In the Input column field, select the column you want to use as your input text. The dropdown menu is automatically populated with columns from your selected table.

    3. In the Output column (optional), select the column you want to provide as an example output for the expected transformation. Providing this data helps configure your agent to more accurately adapt to your domain-specific needs.

  3. Name your agent.

  4. Click Create agent.

Step 2: Build and improve your agent​

In the Build tab, you can review recommendations to improve your agent, review sample model outputs, and adjust your task instructions and evaluation criteria.

In the Recommendation pane, Databricks provides recommendations to help you define evaluation metrics for your agent and evaluate sample responses as good or bad.

  1. Review the Databricks recommendations for optimizing agent performance.

  2. Review suggested evaluation criteria. These recommended evaluation criteria are automatically inferred to help you optimize your agent.

    Review recommended evaluation metrics.

    For each recommendation:

    • To accept the recommendation, select Yes. This adds the evaluation criteria in the Agent configuration pane.
    • To reject the criteria, select No.
    • You can also choose to Dismiss the recommendation.

  3. Review sample model inputs and outputs and provide optional human feedback. This evaluation helps improve the model's reponses.

    Evaluate sample model inputs and outputs to improve responses.

    For each sample, select whether or not it was a good response. If No, provide optional feedback on the response and click Save to move onto the next one.

  4. After you've finished reviewing recommendations, review the Agent configuration pane.

    1. You can adjust the task instructions to be more specific to improve the model's performance.
    2. Review the evaluation criteria you added from the recommendations. You can remove criteria by clicking X.
    3. If you'd like to add more evaluation criteria, click + Add to add your own.

  5. Click Update agent to save those changes to your agent. The examples under Review model output update to show new example model outputs.

Step 3: Try out and optimize your agent​

Try out your agent in workflows across Databricks.

On the Use tab,

  1. Click Try in SQL to open the SQL editor and use ai_query to send requests to your new model specialization agent.

  2. (Optional) Click Optimize if you want to optimize your agent for cost.

    • Optimization requires at least 100 inputs. If you provided a Unity Catalog dataset, the table must contain at least 100 rows. If you did not provide a dataset, you need to provide at least 100 examples.
    • Optimization can take about an hour.
    • Making changes to your currently active agent is blocked when optimization is in progress.

When optimization completes, you are directed to the Review tab to view a comparison of your currently active agent and an agent optimized for cost. See (Optional) Step 4: Review and deploy an optimized agent.

Use or optimize the model specialization agent.

(Optional) Step 4: Review and deploy an optimized agent​

Databricks recommends at least 100 inputs (either 100 rows in your Unity Catalog table or 100 manually-provided examples) to optimize your agent. When you add more inputs, the knowledge base that the agent can learn from increases, which improves agent quality and its response accuracy.

When you select Optimize on the Use tab, Databricks compares multiple different optimization strategies to build and deploy an optimized agent. These strategies include Foundation Model Fine-tuning which uses Databricks Geos.

On the Review tab,

  1. In Evaluation results, you can review the evaluation metrics for the optimized agent. To perform evaluation, Databricks uses metrics based on the evaluation criteria you defined in the Build tab
  2. Click on a request to open more details. Here, you can see a detailed assessment of each evaluation metric, including the rationale for why it passed or failed. This uses Databricks built-in AI judges. You can also inspect the input and response.
  3. After you review these results, you can use your optimized agent.

Limitations​

  • Databricks recommends at least 100 inputs (either 100 rows in your Unity Catalog table or 100 manually-provided samples) to optimize your agent. When you add more inputs, the knowledge base that the agent can learn from increases, which improves agent quality and its response accuracy.
  • If you provide a Unity Catalog table, the table name cannot contain any special characters (such as -).
  • Only the following data types are supported as inputs: string, int, and double.
  • Usage capacity is currently limited to 100k input and output tokens per minute.
  • Workspaces that use PrivateLink, including storage behind PrivateLink, are not supported.
Last updated on