How iOS 18 Photos Removes Subjects and Shadows

The Clean Up tool in iOS 18 Photos uses on-device Apple Intelligence to detect unwanted subjects, erase them, and realistically reconstruct the occluded scene. By combining semantic segmentation, ambient light analysis, and generative inpainting models, the system seamlessly replaces both the object and its associated shadows or reflections with context-aware background textures.

Object Detection and Semantic Segmentation

When a user taps, circles, or brushes over an object, the system activates machine learning models running locally on the device's Neural Engine. Rather than relying solely on the user's manual selection, the tool performs semantic segmentation to identify the exact boundaries of the target subject.

This step distinguishes between the primary object, incidental foreground elements, and the underlying background. The system creates a precise alpha mask around the subject, ensuring that the surrounding environment remains untouched during the removal process.

Identifying and Mapping Cast Shadows

A common issue with traditional object removal is leaving behind "floating" shadows or harsh contact patches. The iOS 18 pipeline addresses this by performing intrinsic image analysis, which separates an image into two core layers:

  1. Reflectance (Albedo): The true color and texture of the surface (e.g., grass, wood, asphalt).
  2. Illumination: The light and shadow gradients falling across those surfaces.

By calculating the trajectory of light sources and identifying ground planes, the model detects cast shadows, contact shadows (ambient occlusion), and reflections linked to the segmented object. Once identified, these darkened zones are added to the modification mask so they are not mistaken for permanent surface textures.

Generative Background Fill

Once the subject and its shadows are masked out, an on-device generative diffusion model synthesizes the missing background information. Unlike traditional content-aware fill tools that merely copy and paste adjacent pixels, the generative model predicts what should exist behind the subject.

The model evaluates contextual clues from the rest of the image, including:

  • Geometric Continuity: Extending structural lines, such as horizon lines, architectural edges, floorboards, or patterns.
  • Texture Synthesis: Generating complex, non-repeating surfaces like foliage, ocean waves, or fabric weave.
  • Depth Estimation: Ensuring that the newly generated background matches the focal plane and depth of field of the surrounding area.

Relighting and Shadow Compensation

To eliminate the shadow realistically, the system normalizes the luminance and chrominance of the affected area. Instead of erasing the surface under the shadow entirely, the algorithm boosts the exposure and adjusts the white balance of the shadowed pixels to match the surrounding, unshadowed surface.

If the shadow is too dense to recover underlying detail, the generative fill reconstructs the surface texture under uniform ambient lighting, matching the light temperature and intensity of the scene.

Edge Blending and Grain Matching

In the final step, the reconstructed area undergoes edge refinement and noise matching. The boundaries between the generated fill and the original photo are smoothed using soft-edge alpha blending to prevent visible seams.

Finally, the Neural Engine analyzes the camera sensor noise and grain of the original photograph and applies a matching noise profile to the generated patch. This prevents the edited area from looking unnaturally smooth or artificial, resulting in a cohesive final image.