Apply AI on data using SAP Databricks AI Functions

note

This feature is in Preview.

This article describes SAP Databricks AI Functions and the supported functions.

What are AI Functions?

AI Functions are built-in functions that you can use to apply AI, like text translation or sentiment analysis, on your data that is stored on Databricks. They can be run from anywhere on Databricks, including Databricks SQL and notebooks.

AI Functions are simple to use, fast, and scalable. Analysts, data scientists, and machine learning engineers can use them to apply data intelligence to their proprietary data.

AI Functions provide general purpose and task-specific functions.

• ai_query is a general-purpose function that allows you to apply any type of AI model on your data. See General purpose function: ai_query.
• Task-specific functions provide high-level AI capabilities for tasks like summarizing text and translation. These task-specific functions are powered by state of the art generative AI models that are hosted and managed by Databricks. See Task-specific AI functions for supported functions and models.

General purpose function: ai_query

The ai_query() function allows you to apply any AI model to data for both generative AI and classical ML tasks, including extracting information, summarizing content, identifying fraud, and forecasting revenue.

Use ai_query with foundation models

The following example demonstrates how to use ai_query using a foundation model hosted by Databricks.

SQL

SELECT text, ai_query(
    "databricks-meta-llama-3-3-70b-instruct",
    "Summarize the given text comprehensively, covering key points and main ideas concisely while retaining relevant details and examples. Ensure clarity and accuracy without unnecessary repetition or omissions: " || text
) AS summary
FROM uc_catalog.schema.table;

Use ai_query with traditional ML models

ai_query supports traditional ML models, including fully custom ones. These models must be deployed on Model Serving endpoints.

SQL

SELECT text, ai_query(
  endpoint => "spam-classification",
  request => named_struct(
    "timestamp", timestamp,
    "sender", from_number,
    "text", text),
  returnType => "BOOLEAN") AS is_spam
FROM catalog.schema.inbox_messages
LIMIT 10

Task-specific AI functions

Task-specific functions are scoped for a certain task so you can automate routine tasks, like simple summaries and quick translations. These functions invoke a state-of-the-art generative AI model maintained by Databricks and they do not require any customization.

The following table lists the supported functions and what task they each perform.

Function	Description
ai_analyze_sentiment	Perform sentiment analysis on input text using a state-of-the-art generative AI model.
ai_classify	Classify input text according to labels you provide using a state-of-the-art generative AI model.
ai_extract	Extract entities specified by labels from text using a state-of-the-art generative AI model.
ai_fix_grammar	Correct grammatical errors in text using a state-of-the-art generative AI model.
ai_gen	Answer the user-provided prompt using a state-of-the-art generative AI model.
ai_mask	Mask specified entities in text using a state-of-the-art generative AI model.
ai_similarity	Compare two strings and compute the semantic similarity score using a state-of-the-art generative AI model.
ai_summarize	Generate a summary of text using SQL and state-of-the-art generative AI model.
ai_translate	Translate text to a specified target language using a state-of-the-art generative AI model.
ai_forecast	Forecast data up to a specified horizon. This table-valued function is designed to extrapolate time series data into the future.
vector_search	Search and query a Mosaic AI Vector Search index using a state-of-the-art generative AI model.