How Windows 3.11 Isolated MS-DOS Virtual Machines

Microsoft Windows 3.11 isolated MS-DOS sessions by running exclusively in 386 Enhanced Mode, using the hardware capabilities of the Intel 80386 processor alongside its own 32-bit core components. By utilizing Virtual 8086 mode, hardware paging, the Virtual Machine Manager (VMM), and Virtual Device Drivers (VxDs), Windows 3.11 created sandboxed virtual machines for each running DOS application. This architecture prevented individual MS-DOS programs from corrupting one another's memory or directly interfering with the underlying graphical Windows operating environment.

The Virtual 8086 (V86) Mode

The foundation of DOS isolation in Windows 3.11 was the Intel 80386 processor's Virtual 8086 mode. While Windows 3.11 executed its 32-bit core services in protected mode at Ring 0 (the highest privilege level), each MS-DOS prompt operated inside a distinct V86 task at Ring 3 (the least privileged level). In this mode, the CPU simulated a dedicated Intel 8086 real-mode computer with its own 1 MB address space, conventional memory layout, and BIOS interrupts, allowing legacy real-mode software to run unchanged without realizing it was operating inside a multitasking environment.

Memory Isolation via Paging

Memory protection was enforced through the x86 processor’s memory management unit (MMU) and 4 KB paging tables. While each MS-DOS session believed it was executing within the standard real-mode address range of 0000:0000 to FFFF:FFFF, the VMM mapped those virtual memory pages to completely different physical memory locations. Because the memory spaces were physically separated, an MS-DOS program that crashed, overflowed a buffer, or attempted an illegal memory write could only corrupt its own assigned address space, leaving the System Virtual Machine (which hosted Windows and Win16 apps) and other DOS sessions untouched.

The Virtual Machine Manager and System Hierarchy

Windows 3.11 divided the operating system into multiple virtual machines managed by the 32-bit Virtual Machine Manager:

• The System VM: A single virtual machine where the Windows graphical interface, user interface components (USER, GDI, KERNEL), and all standard 16-bit Windows applications executed cooperatively.
• DOS VMs: Dedicated, independent virtual machines spawned specifically for each MS-DOS executable or command prompt.

The VMM handled scheduling, context switching, and resource allocation across all VMs, preemptively interrupting and switching between active DOS sessions and the System VM via timer interrupts.

I/O Port Protection and Virtual Device Drivers (VxDs)

MS-DOS software frequently bypassed standard system calls to communicate directly with hardware through I/O ports. To isolate these actions, Windows 3.11 used the x86 I/O Permission Bitmap (IOPB) located within the Task State Segment (TSS).

When a DOS program attempted to execute sensitive instructions—such as IN, OUT, CLI (clear interrupts), or STI (set interrupts)—the CPU triggered a hardware exception (General Protection Fault), which redirected control back to the VMM at Ring 0. Virtual Device Drivers (such as VPICD for the interrupt controller, VDD for the display, and VKD for the keyboard) intercepted these instructions, virtualized the hardware state for that specific session, and simulated the expected hardware response without allowing direct, unmediated physical access.