How iOS 18 Math Notes Handles Matrix Multiplication

Apple’s Math Notes in iOS 18 leverages machine learning, on-device spatial recognition, and symbolic math engines to interpret and solve handwritten linear algebra expressions. When users write matrices and multiplication operators using an Apple Pencil or finger in the Notes and Calculator apps, the system parses the visual structure of rows and columns, converts them into internal mathematical representations, and computes the result automatically upon writing an equals sign.

Spatial Recognition and Bracket Detection

The interpretation pipeline begins with character and structural recognition. Unlike standard horizontal equations, matrices require two-dimensional spatial analysis.

  1. Delimiter Identification: Math Notes first detects enclosing delimiters, typically square brackets [ ] or large parentheses ( ). It evaluates the height, vertical alignment, and curvature of these marks to establish the bounding box of a matrix.
  2. Grid Segmentation: Inside the bounding box, the recognition engine segments individual numbers, variables, or sub-expressions. It analyzes horizontal and vertical whitespace to determine row and column positions, mapping entries into a structured grid (e.g., \(2 \times 2\), \(3 \times 3\), or \(m \times n\)).
  3. Smart Script Integration: Through machine learning models trained on millions of handwriting variations, the system standardizes handwriting input while preserving the user's natural style, maintaining alignment even if rows are drawn slightly unevenly.

Semantic Parsing of Linear Algebra Notation

Once the structural grid is identified, the parser assigns algebraic meaning to the visual elements:

  • Matrix Entities: The grid is transformed into a multi-dimensional array object. Each cell is parsed as a discrete scalar value, constant, or variable.
  • Operators: Math Notes recognizes standard linear algebra operators between matrices, including standard multiplication signs (\(\times\), \(\cdot\)), implicit multiplication (two matrices written directly adjacent to one another), and scalar coefficients placed immediately before a matrix.
  • Dimensional Validation: Before evaluating the expression, the parser verifies dimension compatibility. For standard matrix multiplication (\(A \times B\)), the engine checks whether the number of columns in the first matrix equals the number of rows in the second matrix.

Computational Engine and Evaluation

After parsing, the expression is sent to the underlying computational algebra engine running on the device's Neural Engine:

  • Dot Product Execution: The engine computes the dot product of each row vector from the first matrix and each column vector from the second matrix to populate the resulting array.
  • Scalar and Vector Operations: Math Notes easily handles operations involving scalars (multiplying every element by a single value), vector addition, and column/row vector interactions.
  • Complex Inner Elements: If elements within a matrix contain sub-expressions (such as \(2^3\), fractions, or trigonometric functions like \(\sin(\theta)\)), the engine evaluates those components prior to performing the global matrix multiplication.

Result Rendering

When a user writes an equals sign (=) following the expression, Math Notes triggers the solver:

  • The engine computes the product and generates the resulting matrix.
  • The output is rendered in a dynamically generated bracket format that matches the user's handwriting appearance and scale via Apple's Smart Script technology.
  • If the dimensions are incompatible (e.g., attempting to multiply a \(2 \times 3\) matrix by a \(2 \times 2\) matrix), the engine fails silently or leaves the equation uncalculated, indicating an invalid mathematical operation.