Video Frameworks with Live Text Support in iOS 16

Apple introduced Live Text for video in iOS 16, allowing users to interact with, copy, look up, and translate text directly from paused video frames. This guide covers which video frameworks support Live Text interaction natively in iOS 16, how custom players can implement the feature, and the hardware requirements necessary for it to work.

AVKit (AVPlayerViewController)

The primary and easiest way to support Live Text in iOS 16 is through Apple's native AVKit framework using AVPlayerViewController.

In iOS 16, AVPlayerViewController provides out-of-the-box support for Live Text without requiring manual frame extraction or machine learning integration.

  • Configuration: Support is governed by the property allowsVideoFrameAnalysis.
  • Default Behavior: This property is set to true by default. When a user pauses a video, the system automatically analyzes the frame and displays the Live Text interaction button in the lower-right corner.
  • Opting Out: Developers can disable this feature by setting playerViewController.allowsVideoFrameAnalysis = false.

VisionKit (ImageAnalysisInteraction) for Custom Players

If an app relies on custom playback engines—such as AVPlayerLayer, Metal renderers, or third-party engines like VLCKit or IJKPlayer—AVKit cannot provide automated Live Text functionality. Instead, developers must use VisionKit.

VisionKit enables Live Text integration in custom players through two primary classes:

  1. ImageAnalyzer: Processes the visual content of a paused video frame.
  2. ImageAnalysisInteraction: An interaction object added to the custom player’s view hierarchy to render the selectable text overlays and handle user input.

To support Live Text in a custom player:

  • Observe when the video pauses.
  • Extract the current frame as a CGImage or CVPixelBuffer (e.g., using AVAssetImageGenerator or custom render pipeline outputs).
  • Pass the frame to ImageAnalyzer with the .text analysis type.
  • Feed the resulting ImageAnalysis object into the view's ImageAnalysisInteraction.

Third-Party Frameworks

Standard third-party video frameworks do not have native Live Text extraction built into their proprietary view layers. However:

  • Wrappers using AVPlayerViewController: Any third-party library acting as a wrapper around the system's standard AVPlayerViewController inherits Live Text support automatically.
  • Custom Rendering Frameworks: Any player rendering video directly to a custom UIView or CALayer requires the manual VisionKit implementation outlined above to support Live Text.

System and Hardware Requirements

Live Text analysis during video playback is resource-intensive and is not supported on all devices capable of running iOS 16. It requires:

  • Operating System: iOS 16.0 or later.
  • Hardware: Devices equipped with the Apple A12 Bionic chip or newer (iPhone XS, iPhone XR, and later models). Devices with older processors will ignore the feature, even when running iOS 16.