How Live Text Video Scrubbing Works in iOS 16

iOS 16 expanded Apple's optical character recognition (OCR) capabilities by bringing Live Text to video, allowing users to pause, scrub through frames, and interact directly with text embedded inside moving pictures. This article explains the underlying technology behind this feature, covering how the operating system coordinates video playback frameworks, the Apple Neural Engine, and real-time OCR models to locate, extract, and search text across individual video frames.

Video Framework Integration and Frame Capture

The process begins at the media playback level, primarily handled by Apple's AVFoundation framework. When a video plays in native apps like Safari, Photos, or QuickTime, the system continuously decodes video streams into individual visual buffers.

To enable text search and scrubbing:

  1. Playback Interruption: When playback is paused or the user interacts with the progress bar, the player isolates the exact presentation timestamp (PTS).
  2. Buffer Extraction: The AVFoundation pipeline extracts the uncompressed video frame corresponding to that exact timestamp as a high-resolution pixel buffer (CVPixelBuffer).
  3. Scrubber Tracking: As the user drags the timeline scrubber, the player decodes intermediate keyframes (I-frames) and predicted frames (P-frames/B-frames) sequentially, continuously generating target frames for analysis.

Real-Time OCR via the Vision Framework

Once a frame is isolated, it is handed off to Apple's Vision framework, specifically utilizing the VNRecognizeTextRequest architecture.

  • Text Region Detection: The system first runs a lightweight detection model to locate bounding boxes where candidate text appears within the frame.
  • Character Recognition: The pixels within these bounding boxes are evaluated using machine learning models trained on character shapes, orientations, and multiple languages.
  • Text Normalization: Extracted characters are assembled into coherent words, phone numbers, addresses, and URLs, mapped directly to their coordinate spaces on the display.

Hardware Acceleration via the Apple Neural Engine

Running machine learning models on high-resolution video frames requires significant compute power. Apple offloads this entire pipeline to the Apple Neural Engine (ANE) present in the A-series and M-series chips (A12 Bionic and later).

Because the ANE performs matrix multiplications independently from the CPU and GPU, the device can run optical character recognition within milliseconds. This rapid inference speed is what allows the scrubbing experience to feel responsive. As the user scrubs to a new frame, the text detection pass completes almost instantly, avoiding noticeable latency between stopping the scrubber and revealing interactive text layers.

The Interactive Overlay and Text Search

After the text is recognized, the system projects an interactive interface directly over the video player:

  • Hit-Testing Coordinates: The software creates an invisible vector overlay mapped directly over the visual text coordinates.
  • Text Selection: Users can tap, drag, copy, share, translate, or look up text just as they would in a standard text document or webpage.
  • Search Integration: When a user initiates a text search within supported apps, the system scans cached text metadata or rapidly runs text detection against indexed video timestamps, allowing users to jump directly to the frame where that specific word or number appears.

Privacy and Performance Optimization

The entire pipeline operates locally on the device. Video frames are analyzed in temporary memory buffers without sending visual data to external cloud servers. To preserve battery life and prevent thermal throttling, the system applies adaptive sampling: full recognition is only triggered when scrubbing velocity slows down or stops, ensuring resources are not wasted on frames that move past too quickly for human inspection.