How iOS 16 Live Captions Generate Transcriptions

Apple’s Live Captions feature in iOS 16 provides real-time, automatic text transcriptions for spoken dialogue across apps, phone calls, FaceTime, and in-person conversations. Designed primarily for deaf and hard-of-hearing users, the tool uses on-device machine learning to convert speech to text instantly without needing an active internet connection. This article explains the underlying technology, processing pipeline, and system mechanics that enable Live Captions to transcribe audio in real time.

On-Device Machine Learning and the Neural Engine

Live Captions relies on the Apple Neural Engine (ANE), the dedicated hardware accelerator integrated into Apple’s A-series chips. Because transcription demands significant computational power, iOS 16 processes speech using localized neural network models rather than sending audio data to cloud servers.

When audio is detected, the Neural Engine executes speech recognition algorithms directly on the device. This local execution accomplishes two things: it reduces latency to sub-second levels, which is essential for real-time conversation, and it maintains user privacy by ensuring that private phone calls and environmental audio never leave the iPhone.

Audio Stream Ingestion

The transcription pipeline captures audio from two distinct sources:

  1. System Audio: The operating system intercepts direct digital audio output from media apps, podcasts, streaming video, FaceTime, and standard telephone calls.
  2. Microphone Input: For in-person conversations, the system uses the iPhone's multi-microphone array. The system applies noise reduction to isolate nearby voices from ambient room noise before feeding the signal to the transcription engine.

Regardless of the source, iOS routes the raw audio through the Core Audio framework, standardizing the format into a stream suitable for the recognition model.

Real-Time Speech-to-Text Pipeline

Once the audio stream is captured, the transcription engine processes it in continuous micro-buffers through two main model layers:

  • Acoustic Modeling: The raw audio wave is broken down into spectrographic representations, which the model converts into phonemes (the distinct units of sound in a language).
  • Language Modeling: The phonemes are passed through an on-device natural language model that predicts words and phrases. Because speech patterns vary, the model continuously calculates probabilities based on sentence context to correct homophones and refine word choices dynamically as a speaker finishes a sentence.

As words are recognized, they are rendered incrementally in a floating, system-level overlay on the screen, allowing users to read along while navigating different applications.

Hardware and Language Requirements

Because of the high processing demands of continuous, on-device neural processing, Live Captions is limited to devices equipped with the A12 Bionic chip or newer. Additionally, the complex language models required for offline processing mean that initial versions were deployed for select languages, such as English (U.S. and Canada), with models pre-installed or downloaded directly to system storage to ensure full offline capability.