Exclude Overlapping Shapes in Adobe Photoshop

In Adobe Photoshop, the "Exclude Overlapping Shapes" path operation performs a Boolean symmetric difference on overlapping vector paths, generating a single compound shape where intersecting areas are removed. This operation retains the unique, non-intersecting regions of the shapes while converting the overlapping segments into transparent voids or cutouts. The resulting geometry is a non-destructive compound path governed by the even-odd fill rule, allowing designers to create complex silhouettes and hollowed-out vector forms without rasterizing the artwork.

The Geometric Output: Symmetric Difference (XOR)

Geometrically, Exclude Overlapping Shapes executes an exclusive OR (XOR) operation, mathematically defined as \((A \cup B) \setminus (A \cap B)\). When two vector shapes, Shape A and Shape B, are merged using this method:

  • Independent areas remain filled: Any coordinate covered exclusively by Shape A or exclusively by Shape B maintains its vector fill.
  • Shared intersections become transparent: The shared region (\(A \cap B\)) is subtracted from the final vector mask, rendering that area completely empty.

The resulting vector object is a single compound path rather than two distinct layers. Photoshop treats this new path as one unified boundary with internal negative space.

The Even-Odd Winding Rule

When three or more shapes overlap, Photoshop determines fill visibility using the Even-Odd Rule (also known as the parity rule).

Under this geometric algorithm, Photoshop casts a ray from any point within the composition outward to infinity and counts how many path boundaries it crosses:

  • If the ray crosses an odd number of path boundaries, the region is filled.
  • If the ray crosses an even number of path boundaries, the region is excluded (transparent).

Consequently, stacking three overlapping shapes creates an alternating pattern: the area where all three shapes overlap will become filled again because the overlap count switches from even (subtracted) back to odd (filled).

Structural Characteristics in Photoshop

The geometric output created by this operation maintains specific vector properties:

  • Non-Destructive Vector Masking: The underlying component paths are not permanently dissolved. You can still select, move, scale, or transform the individual component paths using the Direct Selection Tool (A) without re-rasterizing the layer.
  • Unified Layer Styling: Because the output is a single compound vector shape, any applied layer styles—such as strokes, drop shadows, or bevels—treat the inner excluded edges as vector borders. Strokes will wrap around both the exterior boundaries and the interior cutouts.