ARKit 6 4K Video Background Capture in iOS 16

Apple’s ARKit 6, released alongside iOS 16, introduces the ability to capture 4K video background streams during active augmented reality sessions. This update bridges the gap between high-performance spatial tracking and professional-grade video recording, allowing developers to create applications that combine realistic computer-generated assets with ultra-high-definition live video feeds. This article explores the technical mechanisms ARKit 6 uses to achieve 4K video capture, the implementation process for developers, and the performance trade-offs involved.

The Evolution of Camera Feeds in ARKit

Prior to ARKit 6, camera resolution during AR sessions was limited to lower resolutions, typically 1080p or 720p at 60 frames per second. These constraints were intentional: real-time computer vision algorithms, such as Visual Inertial Odometry (VIO), plane detection, and scene reconstruction, require significant compute bandwidth. Capping the resolution ensured that the Neural Engine and GPU could process frames fast enough to maintain accurate tracking without overheating the device or causing frame drops.

In ARKit 6, Apple decoupled the internal tracking pipeline from the final render pipeline. This architecture enables the camera sensor to stream 4K resolution (3840 × 2160) for rendering and recording while downsampling the input frames internally for tracking algorithms. Consequently, device tracking remains smooth and responsive even though the background video renders at 4K.

Selecting 4K Video Formats

To enable 4K background video, developers interact with ARWorldTrackingConfiguration or ARFaceTrackingConfiguration. ARKit exposes hardware-supported camera configurations via the supportedVideoFormats property.

In iOS 16, this list includes formats targeting 4K resolutions. Implementing 4K involves the following steps:

  1. Query Supported Formats: Inspect ARWorldTrackingConfiguration.supportedVideoFormats to locate a format matching 3840 × 2160 resolution.
  2. Assign the Format: Set the chosen format to the configuration’s videoFormat property before running the session.
  3. Run the Session: Pass the updated configuration to ARSession.run(_:options:).

Because 4K video requires higher data throughput, ARKit typically limits these 4K formats to 30 frames per second (fps) rather than 60 fps to manage thermal output and power consumption.

Advanced Sensor and Exposure Control

Along with 4K capture, ARKit 6 introduced enhanced camera control settings to assist professional video recording:

  • Manual Camera Adjustments: Developers gain explicit control over camera properties such as exposure duration, ISO settings, and white balance. This prevents unexpected shifts in brightness and color temperature during recording.
  • High-Resolution Depth and EXIF Data: The 4K frames retain rich metadata, allowing recorded video to be integrated into post-production workflows alongside depth maps.
  • High-Resolution Stills: Alongside 4K video streaming, ARKit 6 provides an API to capture high-resolution still images (up to 4K or higher, depending on the sensor) without interrupting the active AR session.

Hardware Requirements and Performance Considerations

Capturing 4K AR video requires substantial processing power and is supported on devices equipped with an A12 Bionic chip or later, though optimal performance is achieved on devices with A14 Bionic and newer hardware (such as iPhone 12 Pro and later models).

When utilizing 4K video capture, developers must account for:

  • Thermal Throttling: Sustained 4K video capture paired with real-time 3D rendering increases device temperature, which can trigger thermal throttling and reduce frame rates.
  • Storage and Memory Bandwidth: 4K video buffers consume significantly more memory than standard 1080p buffers. Applications that record to disk must use hardware-accelerated video encoding (such as HEVC via AVAssetWriter) to prevent frame dropping.
  • Tracking Fidelity: While the tracking system efficiently handles downsampled data, rapid device movements at 30 fps can result in slightly more motion blur than 60 fps configurations, potentially affecting tracking stability in fast-motion scenarios.