How Windows 3.11 Enforced DOS Memory Limits

Windows 3.11 controlled and restricted memory allocation for individual DOS application windows by pairing the hardware capabilities of the Intel 80386 processor with internal software management systems. Operating in 386 Enhanced Mode, the operating system utilized Virtual 8086 mode, hardware-level paging, specialized Virtual Device Drivers (VxDs), and Program Information Files (.PIF) to isolate DOS applications into distinct virtual machines and strictly regulate their access to conventional, expanded, and extended memory.

Program Information Files (PIF)

The baseline configuration for any DOS application's memory footprint began with its Program Information File (.PIF), configured via the Windows PIF Editor. Inside the PIF file, users and administrators defined specific memory constraints:

  • Conventional Memory: Set via "KB Required" (the minimum memory required to launch) and "KB Desired" or "KB Limit" (the maximum conventional memory mapped to the application, capped at 640 KB).
  • Expanded Memory (EMS): Set via minimum and maximum limits for applications relying on the Lotus-Intel-Microsoft (LIM) EMS standard.
  • Extended Memory (XMS): Set via limits that dictated how much memory complying with the Extended Memory Specification the application could allocate.
  • Display and High Memory: Options to reserve or share video memory and High Memory Area (HMA) access.

If an application lacked an individual PIF file, Windows applied default constraints defined in _DEFAULT.PIF.

Virtual 8086 Mode and VM Isolation

When launched in 386 Enhanced Mode, Windows 3.11 did not run DOS applications in real mode. Instead, the 32-bit Virtual Machine Manager (VMM), contained within WIN386.EXE, spawned a dedicated Virtual Machine (VM) for each DOS session using the CPU's hardware-level Virtual 8086 (V86) mode.

In V86 mode, the processor emulated an 8086 environment with a 1 MB address space, but enforced protected-mode privilege levels (Ring 3). Any attempt by the DOS application to directly access hardware or memory outside its assigned VM triggered an exception handled by the 32-bit Windows kernel running at Ring 0.

Hardware Paging and Memory Translation

Windows 3.11 enforced conventional memory limits through the 80386 Memory Management Unit (MMU) and 4 KB page tables:

  1. Linear Address Mapping: The VMM maintained separate page directory entries for each DOS VM. The apparent 640 KB conventional space seen by the DOS application was mapped to scattered, non-contiguous physical memory pages managed by Windows.
  2. Page Fault Enforcement: If a DOS application attempted to read or write beyond the "KB Desired" threshold or into address ranges not committed to that specific VM, the CPU generated a page fault (Interrupt 14). The Windows VMM caught the fault and prevented the application from accessing unauthorized physical memory, typically terminating the application or returning a standard out-of-memory error.

Virtual Device Drivers and Memory Emulation (VxDs)

DOS applications frequently bypassed standard DOS memory limits by querying EMS or XMS drivers directly. Windows 3.11 intercepted these requests using 32-bit Virtual Device Drivers, primarily *V86MMGR (Virtual 8086 Mode Memory Manager) and *DOSMGR:

  • EMS Emulation: When a DOS app issued an INT 67h software interrupt to request expanded memory, *V86MMGR trapped the interrupt. It checked the requested allocation against the EMS limits declared in the application's PIF file. If the application requested more memory than permitted, the VxD returned an EMS error code indicating no remaining pages were available.
  • XMS Emulation: When an application called the XMS driver entry point (discovered via INT 2Fh, AX=4310h), the call was intercepted by Windows. The kernel tracked the cumulative allocations made by that specific VM handle. Once allocations reached the PIF limit, subsequent XMS allocation calls failed with error code A0h (all extended memory allocated).
  • DPMI Services: For 16-bit or 32-bit protected-mode DOS extenders, Windows provided integrated DOS Protected Mode Interface (DPMI 0.9) services via INT 31h. The DPMI host enforced limits on descriptor creation and physical page allocation based on global system resources and the application's XMS configuration.