How Live Text Parses Scanned Documents in iOS 15 Files

Apple's iOS 15 introduced Live Text, a native feature that uses on-device intelligence to recognize and extract text directly from images and scanned documents. When integrated with the Files application, Live Text allows users to scan physical paper documents into digital formats and immediately interact with the printed words. This article explains the end-to-end technical process of how the Files app captures paper documents and how Live Text parses that visual data into selectable, actionable text.

Document Capture and Pre-Processing in Files

The parsing process begins with document digitization directly inside the Files app. Using the native document scanner, the device's camera captures multiple frames of the paper document. The operating system immediately applies pre-processing algorithms:

  • Edge Detection: The system identifies the boundaries of the physical paper against the background.
  • Perspective Correction: It eliminates keystone distortion caused by angled shooting, flattening the image into a standard rectangular plane.
  • Filter Application: The software adjusts contrast, eliminates shadows, and binarizes or optimizes color profiles to maximize text legibility.

The resulting image is then embedded into a standard PDF wrapper within the local file system.

On-Device Machine Learning and Neural Engine Execution

Once the document is rendered, the Vision framework and Apple's Neural Engine (ANE) handle text analysis entirely on the device. Because processing is local, no image data is transmitted to external servers.

The system relies on deep convolutional neural networks (CNNs) trained to distinguish between arbitrary background artifacts and human-readable text. The Neural Engine accelerates this pipeline, processing high-resolution visual data in milliseconds while minimizing battery consumption.

Text Detection and Optical Character Recognition (OCR)

Live Text breaks document parsing into two core algorithmic stages:

  1. Text Detection (Bounding Box Generation): The model analyzes the flattened image to detect text baselines. It groups contiguous pixels into characters, words, lines, and paragraphs, drawing invisible coordinate bounding boxes around each detected segment.
  2. Character Recognition: Within these bounding boxes, the OCR engine analyzes individual glyphs. It compares character geometry against pre-trained scripts and fonts. Machine learning models evaluate context to differentiate visually ambiguous characters (such as the number "0" and the letter "O").

Semantic Parsing via Data Detectors

After converting visual glyphs into digital Unicode text, the system applies semantic parsing using Apple's Data Detectors engine. This step gives context to the recognized strings:

  • Pattern Recognition: Regular expressions and contextual algorithms identify structured information such as phone numbers, email addresses, physical street addresses, dates, and URLs.
  • Action Assignment: Recognized patterns are flagged as interactive elements. Instead of treating a phone number as static text, the system pairs it with an actionable system trigger, such as initiating a phone call or adding a contact.

Interactive UI Overlay

The final phase maps the parsed data back to the visual document inside the Files PDF viewer. The Files app overlays an invisible text interaction layer precisely aligned with the bounding coordinates of the visual document.

When a user taps or drags a finger across the scanned document, iOS 15 registers the interaction on this invisible vector layer. This allows the user to select, copy, look up definitions, translate foreign languages, or tap contextual shortcuts just as they would with a native digital document.