iOS 16 Live Text Frame Extraction in AVFoundation

In iOS 16, Apple expanded Live Text to video, allowing users to select, copy, and interact with text directly within paused video frames. This capability is powered by a seamless integration between AVFoundation and VisionKit. AVFoundation handles the precise, low-latency extraction of uncompressed video frames via dedicated output pipelines, while VisionKit's analysis engines process the visual data. This article explores the internal mechanisms AVFoundation uses to extract, buffer, and deliver video frames for Live Text analysis during playback.

The Trigger Mechanism: Paused State Activation

Live Text processing does not run continuously during active video playback due to thermal, battery, and processing constraints. Instead, the frame extraction pipeline is activated when the player state shifts to paused.

When an AVPlayer pauses, the system identifies the exact presentation timestamp (CMTime) of the currently displayed frame. This timestamp serves as the temporal reference point for extracting the high-resolution source image directly from the decoded video stream rather than capturing a low-quality screen render.

Frame Retrieval via AVPlayerItemVideoOutput

At the architectural core of this process is AVPlayerItemVideoOutput. While higher-level UI components like AVPlayerViewController handle the interaction automatically, custom implementations rely on this class to pull video frames into memory:

  1. Output Attachment: An AVPlayerItemVideoOutput object is configured with specific pixel buffer attributes (typically requesting native hardware-accelerated formats such as 32-bit BGRA or bi-planar YUV kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange) and added to the current AVPlayerItem.
  2. Availability Check: When playback stops, the system calls hasNewPixelBuffer(forItemTime:) to verify that a valid frame corresponds to the target presentation time.
  3. Buffer Copy: The method copyPixelBuffer(forItemTime:itemTimeForDisplay:) fetches the raw CVPixelBuffer corresponding to the exact frame visible to the user.
let outputSettings: [String: Any] = [
    kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA
]
let videoOutput = AVPlayerItemVideoOutput(pixelBufferAttributes: outputSettings)
playerItem.add(videoOutput)

// Extraction at pause
let currentTime = playerItem.currentTime()
if videoOutput.hasNewPixelBuffer(forItemTime: currentTime) {
    if let pixelBuffer = videoOutput.copyPixelBuffer(forItemTime: currentTime, itemTimeForDisplay: nil) {
        // Pixel buffer ready for VisionKit analysis
    }
}

Zero-Copy Transfer to VisionKit

To maintain responsiveness and low memory overhead, AVFoundation leverages unified memory architecture on Apple silicon. The extracted CVPixelBuffer references the decoded frame in memory managed by the Core Video framework.

Rather than converting this raw data into intermediate image representations like UIImage or CGImage, AVFoundation passes the CVPixelBuffer directly to VisionKit's ImageAnalyzer. The ImageAnalyzer.analyze(_:configuration:) API natively accepts a CVPixelBuffer, creating a zero-copy or minimal-overhead handoff directly to the Neural Engine and GPU for optical character recognition (OCR).

AVPlayerViewController Built-In Integration

For applications utilizing standard system controls via AVPlayerViewController, iOS 16 abstracts manual buffer management entirely.

Under the hood, AVPlayerViewController observes player rate changes. When the playback rate reaches zero, it attaches an internal video output consumer, acquires the current presentation frame, feeds it to an embedded ImageAnalysisInteraction, and overlays the interactive Live Text interface directly over the video layer. When the user resumes playback, the interaction is dismissed or hidden, and the frame buffers are immediately released back to the Core Video pool to ensure uninterrupted decoding performance.