IPsec Authentication in Windows 2000

In Microsoft Windows 2000, Internet Protocol Security (IPsec) provides secure, encrypted network tunnels primarily by authenticating computer hosts rather than human users at the IPsec layer itself. To achieve end-to-end user authentication over an IPsec tunnel, Windows 2000 pairs IPsec with the Layer 2 Tunneling Protocol (L2TP). This article explains the mechanisms Windows 2000 uses to negotiate security associations, the machine-level authentication protocols supported by the native IPsec stack, and how user credentials are validated within L2TP/IPsec tunnels.

Machine Authentication vs. User Authentication

Native IPsec operates at the Network Layer (Layer 3) of the OSI model. Because it is bound to network interfaces and IP addresses rather than interactive user sessions, standard IPsec in Windows 2000 authenticates the computer accounts at both endpoints of the tunnel.

To authorize individual human users, Windows 2000 utilizes L2TP over IPsec. In this architecture, IPsec provides a secure, encrypted wrapper (the tunnel) based on machine identity, while L2TP encapsulates Point-to-Point Protocol (PPP) frames that handle the interactive user authentication.

IPsec Machine Authentication Methods

When establishing the IPsec tunnel, the Internet Key Exchange (IKE) protocol negotiates the connection and validates the identity of both endpoints. Windows 2000 supports three methods for this initial authentication phase:

  1. Kerberos v5 (Default): If both endpoints belong to the same Active Directory forest or trusted domains, Windows 2000 defaults to Kerberos v5. The Key Distribution Center (KDC) validates the computer accounts automatically without requiring manual certificate installation.
  2. Public Key Certificates (X.509): For cross-domain connections, remote access clients, or non-domain member systems, Windows 2000 supports digital certificates. Both endpoints must trust a common root Certificate Authority (CA) to validate the identity of the certificates presented during the IKE exchange.
  3. Preshared Keys: Administrators can configure a shared plaintext secret string on both endpoints. While simple to deploy, this method is considered less secure because the key is stored in plaintext within the local IPsec policy.

User Authentication Through L2TP over IPsec

To authenticate the actual user across the network, Windows 2000 runs L2TP inside the established IPsec security association. Once the two computers verify each other's identity via IPsec:

  1. IPsec Tunnel Establishment: IKE Phase 1 (Main Mode) establishes a secure channel using Kerberos, certificates, or a preshared key. IKE Phase 2 (Quick Mode) negotiates the Security Associations (SAs) used to encrypt network traffic via Encapsulating Security Payload (ESP).
  2. L2TP Negotiation: The client and server initiate an L2TP control connection and establish a call session inside the encrypted IPsec tunnel.
  3. PPP User Authentication: The PPP layer prompts the user for credentials. Windows 2000 supports several user-level authentication protocols across this link:
    • MS-CHAP v2: Provides mutual authentication and generates unique session keys for the client and server.
    • EAP-TLS: Enables smart card or user-certificate authentication directly integrated into an enterprise Public Key Infrastructure.
    • MS-CHAP v1 / CHAP / PAP: Supported for legacy compatibility, though generally discouraged due to weaker security profiles.

Once the PPP credentials are authenticated against Active Directory or the local Security Accounts Manager (SAM), the user is granted access to the internal network resources while IPsec maintains continuous encryption of the underlying tunnel.