How iOS 15 Live Text Works in Camera Preview

Live Text in iOS 15 allows users to select, copy, and interact with text directly inside the Camera app without ever pressing the shutter button. This capability relies on real-time video buffer sampling, the Vision framework, and the computational power of the Apple Neural Engine to process live video frames locally on the device with zero shutter lag.

Real-Time Video Frame Streaming

Rather than saving a full-resolution JPEG or HEIC file to local storage, the Camera app uses Apple's AVFoundation framework to access a continuous stream of raw video frames. As the camera points at an object, the sensor feeds uncompressed pixel buffers directly into device memory. Live Text samples these frames on the fly, bypassing the standard photo-capture pipeline entirely.

Hardware Acceleration via the Apple Neural Engine

Processing high-resolution video frames at 30 or 60 frames per second requires substantial computational power. Live Text delegates this workload to the Apple Neural Engine (ANE), available on devices equipped with the A12 Bionic chip or newer. The ANE executes deep learning inference tasks concurrently with the camera's auto-exposure and autofocus algorithms without overheating the device or causing battery drain.

Two-Stage Text Extraction Pipeline

Once a frame is loaded into memory, Apple's Vision framework performs text extraction through two primary stages:

  1. Text Detection: A lightweight convolutional neural network scans the frame to identify regions of interest. It looks for visual patterns that resemble typography and maps out geometric bounding boxes around words and lines of text.
  2. Text Recognition: The system crops the identified regions and passes them through an Optical Character Recognition (OCR) model. This model converts the pixel patterns into machine-readable Unicode strings. In-line natural language processing models then check the results against language dictionaries to correct ambiguities, such as distinguishing the number "0" from the capital letter "O."

Dynamic Spatial Tracking and UI Overlays

After the text is converted into strings, the system maps the detected bounding boxes back to the coordinates of the camera preview layer. Using Metal and Core Animation, iOS renders the interactive Live Text badge and selection highlights directly over the viewfinder.

Computer vision tracking algorithms maintain the position of these bounding boxes frame-by-frame. If the user moves the phone slightly, the system tracks the motion vector and adjusts the overlay in real time, giving the illusion that the physical text itself has become an interactive software element.