How Live Text Translation Works on iOS 15

Live Text in iOS 15 allows users to instantly detect, select, and translate printed foreign text directly within the Camera app, Photos, and Safari. This feature relies entirely on on-device intelligence, combining Apple's Vision framework, the Neural Engine, and integrated translation models. By chaining together real-time optical character recognition (OCR), natural language processing, and augmented reality visual rendering, iOS 15 reads foreign scripts and overlays translated replacements directly over the original image.

Optical Character Recognition (OCR) and Text Detection

The process begins with visual analysis powered by Apple's Vision framework. When a camera frame or photo is analyzed, deep convolutional neural networks run locally via the Apple Neural Engine to identify regions containing text.

The system isolates text baselines, bounding boxes, and individual character glyphs from complex backgrounds. Unlike basic OCR, Live Text is optimized to handle varying lighting conditions, angled surfaces, curved paper, and diverse typography in real time without causing significant battery drain or latency.

Script Recognition and Language Identification

Once text regions are detected, iOS 15 passes the extracted character data to the Natural Language framework to determine the script and language:

  1. Script Parsing: The system categorizes the script type (such as Latin, Cyrillic, or Hanzi).
  2. Language Scoring: Statistical language models evaluate word combinations and character frequencies to accurately distinguish between languages sharing the same alphabet (such as French, Spanish, and Italian).
  3. Contextual Tokenization: Text is segmented into sentences or logical phrase tokens rather than isolated words to prepare for accurate grammatical translation.

On-Device Neural Machine Translation

Once the source language is identified, iOS activates its native translation engine. Unless the user specifies otherwise or downloads are missing, this translation occurs entirely on-device using pre-installed offline language packs.

The neural machine translation (NMT) architecture maps the source sentence into an abstract contextual vector and decodes it into the user’s target language. Because the model translates full phrases rather than word-for-word, it preserves grammatical context, idiomatic phrasing, and punctuation.

Visual Replacement and Dynamic In-Place Rendering

To display the translated text seamlessly over the original image, iOS uses an overlay rendering pipeline:

  • Background Inpainting: The system samples the colors and texture immediately surrounding the original text to generate a subtle background patch, masking the original foreign words.
  • Perspective Matching: Using metadata from the initial text detection phase, the system calculates the exact angle, scale, and skew of the target surface.
  • Typography and Layout Adaptation: The translated text is dynamically scaled and rendered using matching text orientation and perspective transforms. Because translated sentences often differ in length from the source text, the layout engine adjusts font size and line wrapping so the replacement naturally fills the bounding area of the original text.