Batch speech-to-text with Whisper

Open in Databricks

Use OpenAI Whisper large-v3-turbo to transcribe a batch of English speech recordings on an attached A10 AI Runtime. This notebook shows how to:

Load the Whisper large-v3-turbo model with the Transformers pipeline.
Build a batch of audio samples from the LibriSpeech test set.
Visualize the waveform and spectrogram of each sample.
Run batched transcription and compare its throughput with sequential inference.

note

This example requires the Databricks AI environment version 6 or above.

Connect to serverless GPU compute

From the notebook compute selector, select Serverless GPU.
In the Environment panel, select the A10 accelerator and the AI v6 environment.
Click Apply, then confirm the environment.

The Whisper model and the LibriSpeech sample dataset are public and do not require Hugging Face authentication.

Import libraries

The AI environment includes the PyTorch, Transformers, and Hugging Face Datasets packages used in this notebook, so no package installation is required. This cell imports them and confirms that a GPU is attached.

Python
import torch
import transformers
import datasets

print(f"Environment")
print(f"   PyTorch:      {torch.__version__}")
print(f"   Transformers: {transformers.__version__}")
print(f"   Datasets:     {datasets.__version__}")
print(f"\nGPU")
print(f"   Available:    {torch.cuda.is_available()}")
if torch.cuda.is_available():
    print(f"   Device:       {torch.cuda.get_device_name(0)}")
    mem_gb = torch.cuda.get_device_properties(0).total_memory / 1e9
    print(f"   Memory:       {mem_gb:.1f} GB")

Output
Environment
   PyTorch:      2.11.0+cu130
   Transformers: 5.8.1
   Datasets:     4.8.5

GPU
   Available:    True
   Device:       NVIDIA A10G
   Memory:       23.7 GB

Load the Whisper model

Load openai/whisper-large-v3-turbo, a distilled 809M-parameter model that delivers near-state-of-the-art accuracy at faster inference speed. The Transformers pipeline handles feature extraction, tokenization, and decoding in a single call.

Python
from transformers import pipeline
import torch

whisper_pipe = pipeline(
    "automatic-speech-recognition",
    model="openai/whisper-large-v3-turbo",
    torch_dtype=torch.float16,
    device="cuda",
)
print(f"Model loaded on {whisper_pipe.device}")

Output
Model loaded on cuda

Load and explore audio samples

Load the LibriSpeech ASR test set, a collection of clean English speech recordings with reference transcriptions, and play the first sample.

Python
from datasets import load_dataset, Audio as AudioFeature
from IPython.display import display, Audio
import numpy as np
import soundfile as sf
import io

# Load the LibriSpeech test samples (decode=False to avoid torchcodec/FFmpeg dependency)
ds = load_dataset(
    "hf-internal-testing/librispeech_asr_dummy", "clean", split="validation"
)
ds = ds.cast_column("audio", AudioFeature(decode=False))
print(f"Loaded {len(ds)} audio samples\n")

def decode_audio(raw):
    """Decode raw audio bytes with soundfile."""
    arr, sr = sf.read(io.BytesIO(raw["bytes"]))
    return {"array": arr, "sampling_rate": sr}

# Show metadata for first few samples
for i in range(5):
    audio = decode_audio(ds[i]["audio"])
    duration = len(audio["array"]) / audio["sampling_rate"]
    text_preview = ds[i]["text"][:80]
    print(f"  Sample {i+1}:  {duration:.2f}s  |  {audio['sampling_rate']} Hz  |  \"{text_preview}...\"")

# Play the first sample inline
print("\n>> Playing Sample 1:")
audio_0 = decode_audio(ds[0]["audio"])
display(Audio(audio_0["array"], rate=audio_0["sampling_rate"]))

Output
Loaded 73 audio samples

  Sample 1:  5.86s  |  16000 Hz  |  "MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME H..."
  Sample 2:  4.82s  |  16000 Hz  |  "NOR IS MISTER QUILTER'S MANNER LESS INTERESTING THAN HIS MATTER..."
  Sample 3:  12.48s  |  16000 Hz  |  "HE TELLS US THAT AT THIS FESTIVE SEASON OF THE YEAR WITH CHRISTMAS AND ROAST BEE..."
  Sample 4:  9.90s  |  16000 Hz  |  "HE HAS GRAVE DOUBTS WHETHER SIR FREDERICK LEIGHTON'S WORK IS REALLY GREEK AFTER ..."
  Sample 5:  29.40s  |  16000 Hz  |  "LINNELL'S PICTURES ARE A SORT OF UP GUARDS AND AT EM PAINTINGS AND MASON'S EXQUI..."

>> Playing Sample 1:

Visualize the audio waveforms

Plot the waveform and spectrogram of each sample side by side. The waveform shows amplitude over time, and the spectrogram shows the frequency content, where voice energy concentrates in the 100 to 4,000 Hz speech band.

Python
import matplotlib.pyplot as plt
import numpy as np

NUM_SAMPLES = 4
fig, axes = plt.subplots(NUM_SAMPLES, 2, figsize=(16, 3 * NUM_SAMPLES))
fig.suptitle(
    "Waveform & Spectrogram Profiles", fontsize=16, fontweight="bold", y=1.01
)

for i in range(NUM_SAMPLES):
    audio = decode_audio(ds[i]["audio"])
    samples = audio["array"]
    sr = audio["sampling_rate"]
    t = np.arange(len(samples)) / sr

    # --- Waveform ---
    ax_wave = axes[i, 0]
    ax_wave.plot(t, samples, linewidth=0.4, color="#1f77b4", alpha=0.8)
    ax_wave.fill_between(t, samples, alpha=0.15, color="#1f77b4")
    ax_wave.set_ylabel("Amplitude", fontsize=9)
    ax_wave.set_title(f"Sample {i+1} — Waveform ({len(samples)/sr:.1f}s)", fontsize=10)
    ax_wave.set_xlim(0, t[-1])
    ax_wave.grid(True, alpha=0.3)
    if i == NUM_SAMPLES - 1:
        ax_wave.set_xlabel("Time (seconds)", fontsize=9)

    # --- Spectrogram ---
    ax_spec = axes[i, 1]
    ax_spec.specgram(samples, Fs=sr, NFFT=1024, noverlap=512, cmap="magma")
    ax_spec.set_ylabel("Frequency (Hz)", fontsize=9)
    ax_spec.set_title(f"Sample {i+1} — Spectrogram", fontsize=10)
    ax_spec.set_ylim(0, 8000)  # Focus on speech frequencies
    if i == NUM_SAMPLES - 1:
        ax_spec.set_xlabel("Time (seconds)", fontsize=9)

plt.tight_layout()
plt.show()

Transcribe a single sample

Transcribe one sample to verify the pipeline, then compare the prediction with the reference text.

Python
import time

audio_input = decode_audio(ds[0]["audio"])

start = time.perf_counter()
result = whisper_pipe(
    audio_input["array"],
    generate_kwargs={"language": "en"},
)
elapsed = time.perf_counter() - start

duration = len(audio_input["array"]) / audio_input["sampling_rate"]

print(f"Inference time:   {elapsed:.2f}s for {duration:.1f}s audio ({duration/elapsed:.1f}x realtime)")
print(f"\nPredicted:  {result['text'].strip()}")
print(f"Reference:  {ds[0]['text']}")

Output
Inference time:   12.23s for 5.9s audio (0.5x realtime)

Predicted:  Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.
Reference:  MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL

Run batched inference

Transcribe all samples with a configurable batch_size. Batching lets the GPU process multiple audio clips in parallel, which improves throughput over sequential inference. This cell also times a sequential baseline for comparison.

Python
import time
import pandas as pd

_decoded = [decode_audio(ds[i]["audio"]) for i in range(len(ds))]
audio_inputs = [d["array"] for d in _decoded]
_sr = _decoded[0]["sampling_rate"]

# --- Sequential baseline ---
start = time.perf_counter()
seq_results = [
    whisper_pipe(a, generate_kwargs={"language": "en"}) for a in audio_inputs
]
seq_time = time.perf_counter() - start

# --- Batched inference ---
start = time.perf_counter()
batch_results = whisper_pipe(
    audio_inputs, batch_size=8, generate_kwargs={"language": "en"}
)
batch_time = time.perf_counter() - start

total_audio_sec = sum(
    len(a) / _sr for a in audio_inputs
)

print(f"Performance Comparison ({len(audio_inputs)} samples, {total_audio_sec:.1f}s total audio)")
print(f"   Sequential:  {seq_time:.2f}s  ({total_audio_sec/seq_time:.1f}x realtime)")
print(f"   Batched (8): {batch_time:.2f}s  ({total_audio_sec/batch_time:.1f}x realtime)")
print(f"   Speedup:     {seq_time/batch_time:.2f}x\n")

# --- Results table ---
rows = []
for i, res in enumerate(batch_results):
    duration = len(audio_inputs[i]) / _sr
    rows.append({
        "Sample": i + 1,
        "Duration (s)": round(duration, 1),
        "Transcription": res["text"].strip(),
        "Reference": ds[i]["text"],
    })

df = pd.DataFrame(rows)
display(df)

Output
Performance Comparison (73 samples, 481.0s total audio)
   Sequential:  16.02s  (30.0x realtime)
   Batched (8): 9.56s  (50.3x realtime)
   Speedup:     1.68x

Summary

This notebook demonstrated:

Zero-setup GPU inference: the AI Runtime environment ships with torch, transformers, and datasets pre-installed; no %pip install needed
Inline audio playback: listen to samples directly in the notebook with IPython.display.Audio
Waveform & spectrogram visualization: rendered with matplotlib, also pre-installed
Efficient batched inference: using the batch_size parameter for GPU-parallel transcription
Whisper large-v3-turbo: a fast, accurate model for production-grade speech-to-text

To adapt this for your own data, replace the HuggingFace dataset with audio files from a Unity Catalog Volume or cloud storage path.

Example notebook

Batch speech-to-text with Whisper

Open notebook in new tab Open in Databricks

Connect to serverless GPU compute​

Import libraries​

Load the Whisper model​

Load and explore audio samples​

Visualize the audio waveforms​

Transcribe a single sample​

Run batched inference​

Summary​

Example notebook​

Batch speech-to-text with Whisper

Connect to serverless GPU compute

Import libraries

Load the Whisper model

Load and explore audio samples

Visualize the audio waveforms

Transcribe a single sample

Run batched inference

Summary

Example notebook