Create Vector Masks with Pen Tool in After Effects

Vector masks in Adobe After Effects allow you to define visible areas of a layer, cut out unwanted elements, and create custom animations using precise paths. This guide provides a straightforward, step-by-step walkthrough on how to use the Pen Tool to draw, edit, and manipulate vector masks directly on your composition layers.

Step 1: Select the Target Layer

Before selecting the Pen Tool, click on the specific layer (video footage, solid, image, or text) in the Timeline panel that you want to mask. If no layer is selected, the Pen Tool will create a Shape Layer instead of a mask.

Step 2: Choose the Pen Tool

Activate the Pen Tool from the top toolbar or press the keyboard shortcut G. Ensure the "Tool Creates Mask" mode is active (indicated by a box icon next to the star shape icon in the toolbar, enabled automatically when a layer is selected).

Step 3: Draw the Path

  • Straight Segments: Click once in the Composition viewer to create a vertex (point), then click elsewhere to create straight lines connecting the points.
  • Curved Segments: Click and drag to create a vertex with Bezier handles. Adjust the length and angle of the handles to control the curve.
  • Close the Mask: Return to the first point you created. A small circle will appear next to the Pen cursor, indicating you can click to close the loop and complete the vector mask.

Step 4: Refine the Vector Path

Hold down the Pen Tool icon in the toolbar to access specialized path editing tools:

  • Add Vertex Tool: Click along an existing mask path to add more control points.
  • Delete Vertex Tool: Click an existing point to remove it without breaking the path.
  • Convert Vertex Tool: Click a point to toggle between sharp corners and smooth Bezier curves, or drag individual handles to break symmetry.

Step 5: Adjust Mask Properties

In the Timeline panel, expand the layer and reveal the Masks dropdown (or press M for Mask Path, or F for Mask Feather):

  • Mask Mode: Change the blend mode from Add to Subtract, Intersect, or other options depending on whether you want to keep or hide the masked region.
  • Mask Feather: Soften the edges of the vector mask for smoother blending.
  • Mask Opacity: Adjust the transparency of the masked content.
  • Mask Expansion: Expand or contract the boundary of the path without moving individual points.
  • Mask Path: Click the stopwatch icon to set keyframes and animate the vector mask over time.