Skip to main content
Last updated on

ai_transcribe function

Applies to: check marked yes Databricks SQL check marked yes Databricks Runtime

The ai_transcribe() function transcribes an audio file into text. It returns the transcript as a set of time-stamped segments, and, when speaker diarization is enabled for your workspace, labels each segment with a speaker. The output is a VARIANT value, so you can chain it into other AI Functions.

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

Data security​

Your document data is processed within the Databricks security perimeter. Databricks does not store the parameters that are passed into the AI function calls, but does retain metadata run details, such as the Databricks Runtime version used.

Requirements​

  • This function is only available in some regions, see AI function availability.
  • If you are using serverless compute, the following is also required:
    • The serverless environment version must be set to 3 or above, as this enables features like VARIANT.
    • Must use either Python or SQL. For additional serverless features and limitations, see Serverless compute limitations.
  • The ai_transcribe function is available using Databricks notebooks, SQL editor, Databricks workflows, jobs, or Lakeflow pipelines.
  • ai_transcribe costs are recorded as part of the AI_FUNCTIONS product.

Supported input file formats​

Your input data must be the audio file's bytes, stored as a binary type column in a DataFrame or Delta table. If the source files are stored in a Unity Catalog volume, generate the binary column using the Spark binaryFile format reader, as shown in the Examples.

ai_transcribe transcribes audio only. Video files are not supported. During the Beta, the following audio formats are supported:

  • MP3
  • WAV
  • FLAC
  • Opus
  • M4A

Supported languages​

During the Beta, ai_transcribe supports English and Spanish audio. Audio in other languages might return inaccurate transcripts.

Syntax​

ai_transcribe(content)

Arguments​

content is the only required argument.

  • content: A BINARY expression that holds the audio file's bytes, such as the content column produced by read_files(..., format => 'binaryFile').

Returns​

ai_transcribe returns a VARIANT value with the transcript broken into time-stamped segments. When speaker diarization is enabled for your workspace, each segment also carries a speaker label. Otherwise, the speaker_id field is null.

The output has the following schema:

{
  "error_message": STRING,          // null on success; an error message string on failure
  "response": {
    "duration_seconds": DOUBLE,     // total audio duration, in seconds
    "segments": [
      {
        "start": DOUBLE,            // segment start time, in seconds
        "end": DOUBLE,              // segment end time, in seconds
        "speaker_id": STRING,       // speaker label ("1", "2", ...), or null when diarization is off
        "text": STRING              // transcript text for the segment
      }
    ]
  }
}

The response fields are as follows:

  • error_message: null when the call succeeds, or a string describing the failure.
  • response.duration_seconds: The total duration of the audio, in seconds.
  • response.segments: An array of transcript segments, in time order. To get the full transcript, concatenate the segments' text values in order.
  • segments[].start and segments[].end: The segment's start and end time, in seconds.
  • segments[].speaker_id: The speaker label for the segment. Speakers are labeled "1", "2", "3", and so on, in the order they first speak in the audio, and the labels are consistent across the whole recording. This field is null for every segment when speaker diarization is not enabled for your workspace.
  • segments[].text: The transcript text for the segment.

Limitations​

The following limitations apply during the Beta:

  • Each call accepts at most 1 hour of audio. Longer audio returns an error.
  • Each call accepts an input file of at most 512 MB. A larger file returns an error.
  • ai_transcribe transcribes audio only; video files are not supported.
  • Speaker labels (speaker_id) are only populated when speaker diarization is enabled for your workspace. When it is not enabled, speaker_id is null for every segment.

Examples​

The following example transcribes every audio file in a Unity Catalog volume. It reads the files as binary data with read_files(..., format => 'binaryFile') and passes the content column to ai_transcribe.

SQL
SELECT
  path,
  ai_transcribe(content) AS transcription
FROM READ_FILES('/Volumes/catalog/schema/volume/audio/', format => 'binaryFile');

The next example expands the transcript into one row per segment, with the speaker label, timestamps, and text as separate columns. It uses variant_explode table-valued function to un-nest the segments array.

SQL
WITH transcribed AS (
  SELECT
    path,
    ai_transcribe(content) AS result
  FROM READ_FILES('/Volumes/catalog/schema/volume/audio/', format => 'binaryFile')
)
SELECT
  transcribed.path,
  segment.value:start::DOUBLE AS start_seconds,
  segment.value:end::DOUBLE AS end_seconds,
  segment.value:speaker_id::STRING AS speaker_id,
  segment.value:text::STRING AS text
FROM
  transcribed,
  LATERAL variant_explode(transcribed.result:response.segments) AS segment;
On this page