Windows 98 Shell Execution Paths Explained

This article explores how Microsoft Windows 98 handled file opening requests through its shell execution subsystem, detailing the pipeline from user interactions in Windows Explorer to process creation. It covers the core APIs within SHELL32.DLL, registry-based file association lookups, Dynamic Data Exchange (DDE) communication, and the final translation into low-level execution calls.

The Entry Point: ShellExecute and ShellExecuteEx

When a user double-clicked an item in Windows Explorer, entered a path into the Run dialog, or an application initiated a file launch, Windows 98 routed the request through the Windows Shell API. The primary functions responsible for this were ShellExecuteA and the more extensible ShellExecuteExA, exported by SHELL32.DLL.

These APIs accepted parameters such as a file path, execution verbs (such as open, edit, or print), working directory paths, and runtime flags. Unlike low-level process creation APIs that strictly required an executable binary, ShellExecute was designed to handle arbitrary documents, Uniform Resource Locators (URLs), and virtual shell folders.

Item ID Lists (PIDLs) and Path Parsing

Because Windows 98 featured deep integration with Internet Explorer via the Active Desktop update, the shell treated standard filesystem paths and virtual objects under a unified namespace.

  1. Namespace Resolution: Paths were converted into pointer-to-an-item-identifier-list structures (PIDLs).
  2. Object Identification: The shell inspected the target to determine if it was a physical file on a FAT16/FAT32 drive, a URL (e.g., http:// or file://), or a virtual namespace object like the Control Panel or Network Neighborhood.
  3. Executable vs. Document: If the target was a binary file (.exe, .com, .bat, .pif), the execution path prepared to launch it directly. If it was a non-executable file, the shell routed the request to the file association engine.

Registry Lookups in HKEY_CLASSES_ROOT

To process documents, Windows 98 queried the system registry under HKEY_CLASSES_ROOT (HKCR) to resolve the file extension into an actionable command.

  1. Extension Resolution: The shell extracted the file extension (such as .doc or .txt) and read the default value of HKCR\.ext to obtain the Programmatic Identifier (ProgID), such as wordpad.document.1.
  2. Verb Resolution: The shell navigated to HKCR\<ProgID>\shell. If no specific verb was provided in the API call, the shell read the (Default) value of this key to determine the default action (usually open).
  3. Command Retrieval: The shell read the command string located at HKCR\<ProgID>\shell\<verb>\command. This string contained the path to the executable and placeholders, typically %1, which represented the target file path.

Handling Dynamic Data Exchange (DDE)

Windows 98 frequently used Dynamic Data Exchange (DDE) to prevent multiple instances of large applications from launching when opening multiple documents.

If the registry key HKCR\<ProgID>\shell\<verb>\ddeexec was present:

  • The shell checked if the application registered as the DDE server was already running.
  • If the application was active, the shell initiated a DDE conversation and sent the command string formatted according to the ddeexec value (for instance, [open("%1")]), passing the document directly to the running process without creating a new one.
  • If the application was not active, the shell first launched the executable defined in the command key, waited for the application to initialize its DDE server window, and then dispatched the DDE transaction.

Process Creation and Fallback Mechanisms

Once the command line string was assembled—with the %1 argument safely replaced by the enclosed path of the target file—the shell passed the payload to the operating system kernel:

  1. Win32 Execution: For modern 32-bit Windows applications, SHELL32.DLL invoked CreateProcessA in KERNEL32.DLL, which set up the virtual memory space, loaded the Portable Executable (PE) image, and scheduled the primary thread.
  2. Win16 and DOS Compatibility: If the resolved command pointed to a 16-bit New Executable (NE) or MS-DOS binary, Windows 98 routed the request through the Win16 subsystem or initialized an MS-DOS virtual machine (using _default.pif settings unless a specific .pif was found).
  3. The "Open With" Fallback: If an extension had no entry in HKEY_CLASSES_ROOT, the shell halted execution and invoked the native "Open With" dialog, prompting the user to manually select an associated executable and optionally persist that choice back to the registry.