iOS 12 ARKit 2 Graphics Memory Allocation

This article explores how iOS 12 manages graphics memory for ARKit 2 experiences through Apple's Unified Memory Architecture, the Metal rendering pipeline, and dynamic texture streaming. It covers how camera feeds, 3D assets, environment reflection maps, and transient render targets are efficiently allocated, shared, and purged across hardware to maintain a sustained 60 frames per second without exceeding system memory limits.

Unified Memory Architecture and Zero-Copy Textures

iOS devices run on Apple Silicon systems-on-chip (SoCs), which feature a Unified Memory Architecture (UMA). In this architecture, the CPU and GPU share the same physical memory bus rather than using discrete pools of VRAM. iOS 12 takes advantage of this design within ARKit 2 to eliminate redundant data duplication.

When the camera captures raw video frames, they are written to memory as a CVPixelBuffer formatted in bi-planar YCbCr. ARKit 2 uses CVMetalTextureCache to bind these buffers directly to Metal textures without copying data across the bus. By leveraging MTLStorageModeShared, both the CPU (handling Computer Vision tracking and feature point extraction) and the GPU (rendering the camera background pass) access the exact same memory addresses, reducing memory consumption and memory bandwidth saturation.

Environment Texturing Memory Allocation

ARKit 2 introduced automatic environment texturing via AREnvironmentProbeAnchor, which dynamically creates real-time, high-dynamic-range (HDR) cube maps from the physical surroundings. To allocate graphics memory for these probes without degrading performance, iOS 12 employs structured texture allocation:

  1. Cubemap Allocation: ARKit generates standard cubemap textures using Metal, storing them in compressed 16-bit floating-point formats (typically RGBA16Float) to balance visual fidelity and memory footprint.
  2. Mipmap Generation: iOS 12 handles mipmap generation for environment maps directly on the GPU using hardware-accelerated blit commands, keeping the memory allocation compact while providing realistic roughness reflections for physically based rendering (PBR) materials.
  3. Dynamic Purging: As the user moves through space, older or low-priority environment probes are deallocated or cached down to smaller sizes based on camera distance, ensuring that cubemap allocations do not steadily accumulate over long AR sessions.

Transient Render Targets and Memoryless Storage

ARKit 2 experiences involve multi-pass rendering pipelines that require depth, stencil, and multisampling (MSAA) buffers to properly occlude virtual 3D models with real-world objects. In iOS 12, developers using Metal can allocate these transient textures using MTLStorageModeMemoryless.

Memoryless textures exist exclusively within the high-speed, on-chip Tile Memory of the Apple GPU during a render pass. Because tile memory does not use system RAM, intermediate depth and anti-aliasing calculations consume zero bytes of graphics memory in main system storage. The data is created, used, and discarded during the hardware tiling stage, which preserves bandwidth and significantly reduces system thermal throttling.

Handling ARWorldMap and Feature Tracking Overhead

ARKit 2 expanded world tracking with persistent and multi-user experiences via ARWorldMap. Unlike pure graphics textures, an ARWorldMap consists of point clouds, camera tracking history, and anchor definitions.

iOS 12 isolates visual-inertial odometry (VIO) data from the graphics rendering pipeline. While the raw spatial data sits in system RAM allocated to ARKit's background daemon (arkitd), only the actively rendered visual elements—such as visual debug point clouds or reconstructed meshes—are mirrored into vertex buffers within graphics memory. This strict separation prevents complex physical maps from overwhelming the graphics pipeline during multi-user synchronization.

System Memory Pressure and the Jetsam Policy

Because ARKit 2 relies heavily on background computer vision processes alongside 3D rendering, graphics memory allocation is tightly monitored by the iOS 12 kernel watchdog, known as Jetsam.

If an ARKit 2 application allocates high-resolution textures, high-poly geometries, and uncompressed scene assets, it risks crossing the device’s per-process memory footprint limit (which varied between roughly 1.4 GB to 2 GB on iOS 12-supported devices like the iPhone X and iPad Pro). iOS 12 mitigates this risk by internally enforcing texture compression (such as ASTC) and using dynamic frame skipping on camera texture uploads if memory pressure reaches high thresholds, ensuring the app is throttled rather than immediately terminated by the operating system.