IKEv1 Exchanges

In IKEv1, two modes exist for Phase 1 negotiation: Aggressive Mode and Main Mode. While Aggressive Mode is faster, Main Mode is more secure and comprehensive, making it a popular choice in enterprise environments for establishing IPSec tunnels. Main Mode consists of six distinct messages exchanged between peers to negotiate cryptographic parameters, exchange keys, and authenticate identities.

Main Mode Overview: The Six-Step Handshake of IKEv1

Main Mode in IKEv1 is a methodical and secure way for two routers or endpoints to negotiate and establish a secure communication channel over a public network. The process is structured into six discrete messages, often referred to as MMx, where x denotes the sequence number. The initiator sends odd-numbered messages (MM1, MM3, MM5), and even-numbered messages (MM2, MM4, MM6) come from the responder. This sequencing is important to understand when troubleshooting using tools like:

debug crypto isakmp

The purpose of Main Mode is to establish an authenticated and encrypted tunnel before any actual traffic is transmitted. The exchange unfolds in three primary phases:

  1. MM1 and MM2 are dedicated to negotiating cryptographic parameters, such as encryption and hash algorithms, authentication methods, and Diffie-Hellman groups. This defines how the session will be protected.
  2. MM3 and MM4 handle the key exchange, specifically using the Diffie-Hellman protocol. This ensures both sides derive a matching shared secret without ever transmitting it directly.
  3. MM5 and MM6 are used to prove the identity of each peer. The messages contain encrypted payloads that validate the sender using either pre-shared keys or certificates.

While Main Mode offers strong protection and keeps identities hidden within encrypted payloads, one notable drawback is its limited resistance to Denial of Service (DoS) attacks. Unlike IKEv2, which supports stateless cookie mechanisms to fend off resource exhaustion, Main Mode processes each request fully before authentication, making it vulnerable to spoofed initiation attempts.

Main Mode remains a foundational element in traditional IKEv1 deployments and is essential knowledge for any network engineer configuring or debugging VPN tunnels.

Breaking Down MM1 in Main Mode: Proposal and Policy Negotiation

The first message in IKEv1 Main Mode, referred to as MM1 in this context, initiates the process of establishing a secure tunnel between two peers. MM1 includes a critical piece called the CKY values (CKY_I and CKY_R), which are unique cookies generated by both the initiator and the responder. These values help identify and correlate messages between the peers. CKY_I is generated by the initiator using an MD5 hash over the IP source, IP destination, ports, time, and date. CKY_R is generated similarly by the responder upon receiving MM1. Since the timestamp will always be different, the values will also differ.

Along with the CKY values, the initiator sends an SA Payload (Security Association Payload) that contains one or more proposals. Each proposal outlines the cryptographic algorithms and methods the initiator is willing to use for securing the connection. A typical proposal might include:

  • Encryption Algorithm: e.g., DES, 3DES
  • Authentication Method: e.g., Pre-shared key (PSK)
  • Diffie-Hellman Group: e.g., Group 2
  • Hashing Algorithm: e.g., MD5 or SHA-1

Upon receiving this proposal, the responder evaluates the list and replies with their selected option, commonly called the SA Chosen Proposal, in MM2. It’s important to note that the responder can only choose from the options presented by the initiator.

If a policy match is found, negotiation continues. The SA lifetime is also included in the proposal and defaults to 24 hours if not explicitly configured. Suppose there are multiple matching policies on the device. In that case, the Internet Security Association and Key Management Protocol (ISAKMP) will select the first match based on policy priority (lowest-numbered policy first). For this reason, it’s a best practice to place the most secure policy at the top and progressively list less secure ones below.

Here’s an example of what a live ISAKMP configuration might look like:

crypto isakmp policy 10
encryption des
authentication pre
group 2
hash md5

crypto isakmp policy 20
encryption 3des
authentication pre
group 2
hash sha

These policies specify combinations of encryption algorithms, authentication methods, hashing algorithms, and Diffie-Hellman groups. The configuration is flexible; network admins can safely update or add new policies without disrupting existing sessions. Changes will take effect the next time the session renegotiates, typically when the SA expires.

MM2 in Main Mode: Responding with the Chosen Proposal

In the second message of IKEv1 Main Mode (MM#2) the responder replies to the initiator with the selected Security Association (SA) proposal. This response finalizes the cryptographic parameters that will be used for the remainder of the session. MM#2 includes the same CKY values (CKY_I and CKY_R) as a continuation of the exchange, maintaining session correlation.

The SA Chosen Proposal includes the exact combination of cryptographic elements that the responder has agreed to from the initiator’s list in MM#1. For example, the responder might choose:

  • 3DES for encryption
  • Pre-Shared Key (PSK) as the authentication method
  • Diffie-Hellman Group 2 for key exchange
  • SHA-1 for hashing

This selected combination is sent back to the initiator to confirm mutual agreement on the cryptographic policies. MM2 effectively locks in the foundational security parameters that both parties will use, paving the way for the key exchange and identity authentication that follows in MM3 through MM6.

MM3: The Key Exchange in Main Mode via Diffie-Hellman

In the third message of Main Mode (MM#3), the focus shifts to establishing a secure encryption key that both endpoints can use for future communications. Since the IKE negotiation occurs over a potentially untrusted network, such as the internet, it’s critical to securely exchange this key material without exposing it. This is where the Diffie-Hellman key exchange comes into play.

The Diffie-Hellman process allows two parties to generate a shared secret without actually transmitting the secret itself. On the initiator’s side, the device begins with a private number (let’s say a = 2) and uses a commonly agreed upon generator value (e.g., g = 3) and a large prime number (p = 5). Using the formula X = g^a mod p, the initiator generates its public value. This number is then sent to the responder.

On the responder’s side, the same g and p values are used, but it uses its own private number (b) to compute X = g^b mod p. After receiving the initiator’s public value, the responder and initiator each perform a final step to calculate the shared secret:

  • Initiator: K = (X of responder)^a mod p
  • Responder: K = (X of initiator)^b mod p

This shared secret (K) is never directly transmitted, making it highly secure against interception. What’s powerful here is that both sides independently arrive at the same encryption key, thanks to the mathematical magic of modular exponentiation.

Because Diffie-Hellman operations are computationally heavy, especially with large numbers, many enterprise environments leverage VPN hardware accelerators that handle the Diffie-Hellman, RSA, and encryption processes to reduce the load on general-purpose CPUs.

The MM#3 message includes the CKYs (the cookies from both peers), the KE payload (which contains the Diffie-Hellman public values), and Nonce I, a randomly generated number used once to add entropy and ensure uniqueness in the exchange.

This stage of the exchange ensures both parties have securely negotiated and computed a shared encryption key without ever transmitting it openly, laying the cryptographic foundation for the remainder of the session

MM4: Responder Sends Its Key Exchange and Nonce

In the fourth message of IKEv1 Main Mode (MM#4), the responder replies to the initiator with its half of the key exchange material and an additional random value known as the Nonce R. This message includes the responder’s Key Exchange (KE) payload, which contains the responder’s Diffie-Hellman public value. Combined with the initiator’s public value sent in MM#3, this allows both parties to compute the same shared secret key without ever transmitting the private parts of their Diffie-Hellman computations.

The Nonce R is similar to the initiator’s Nonce I—it’s a large, randomly generated number that ensures session uniqueness and adds entropy to the final keying material. Together, Nonce I and Nonce R are used in deriving secure session keys that will be used in later stages of the IKE and IPSec negotiation. This exchange completes the mutual cryptographic setup needed for the final identity verification steps in MM#5 and MM#6.

MM#5: Encryption Key Derivation and Identity Authentication

At this stage of the IKEv1 Main Mode exchange (MM#5) the encryption process must be in place. This means both parties must have derived an encryption key before sending any further packets. This process is grounded in the calculation of SKEYID, a foundational key from which three distinct keys will be derived. The method of calculating SKEYID depends on the type of authentication used—either Pre-Shared Key (PSK) or Public Key Infrastructure (PKI).

For PSK-based authentication, SKEYID is calculated as:
SKEYID = H(PSK | NI| NR)

This formula uses a hash of the pre-shared key and both the initiator’s and responder’s nonces. Importantly, the actual PSK is never transmitted; it’s only used for this internal key derivation. The pre-shared key is typically configured locally on the router using the following command:

crypto isakmp key <key> address <peer-ip-address>

If using PKI, the SKEYID is derived from the Diffie-Hellman shared secret instead:
SKEYID = H(K | NI| NR)
where K is the shared secret resulting from the Diffie-Hellman exchange.

Once SKEYID is computed, it leads to the creation of three derivative keys:

  • SKEYID_d (Derived) – Used to generate new keys.
    Formula: H(SKEYID | K | CKY_I | CKY_R | 0)
  • SKEYID_a (Authentication) – Used specifically for authenticating identity.
    Formula: H(SKEYID | SKEYID_d | K | CKY_I | CKY_R | 1)
  • SKEYID_e (Encryption) – The key responsible for encryption itself.
    Formula: H(SKEYID | SKEYID_a | K | CKY_I | CKY_R | 2)

These keys are independently calculated by both the initiator and responder, but assuming all values match, should yield identical results on both ends.

With the encryption keys in place, MM#5 is now compiled. This message is primarily focused on authenticating the initiator to the responder. It includes the ID Payload, which, by default, is the IP address of the initiator; however, it can also be a hostname or a Distinguished Name (DN). The configuration command to define the identity in Cisco IOS is:

crypto isakmp identity address

Within MM#5, the HASH_I is also transmitted. This value proves the initiator’s identity using previously calculated information. The hash is generated using the formula:
HASH_I = H(SKEYID | CKY_I | CKY_R | PSK_I | SA_Payload | Proposals | ID_I)

This hash and identity payload gives the responder everything it needs to verify the authenticity of the initiator and confirm that the encrypted key material aligns with the agreed-upon parameters.

MM#6: Final Authentication and Completion of Phase 1 in IKEv1 Main Mode

In the final step of IKEv1 Main Mode’s six-message exchange, MM#6 completes the mutual authentication process between the initiator and the responder. At this stage, the initiator has already sent its ID payload and HASH_I (in MM#5), and now it awaits a response from the responder.

In MM#6, the responder replies with its identity payload and a hash value known as HASH_R. This hash is calculated using the formula:
HASH_R = H(SKEYID | CKY_I | CKY_R | PSK_R | SA | Proposal | ID_R)

This securely validates the responder’s identity using all relevant session parameters, including the Security Association (SA), pre-shared key (or certificate), and the responder’s identity itself. The key element here is that both MM#5 and MM#6 are encrypted using the SKEYID_e key that was derived earlier in the exchange. This ensures that the identities and authentication hashes cannot be intercepted or tampered with in transit.

Once the exchange of MM#6 is complete, the two endpoints now have a successfully authenticated, encrypted communication channel referred to as an ISAKMP Security Association (SA). This ISAKMP SA is:

  • Secured using the encryption key SKEYID_e, and
  • Authenticated using either a Pre-Shared Key (PSK) or Public Key Infrastructure (PKI), depending on the configuration.

With this secure tunnel established by Main Mode Phase 1, the groundwork is laid for creating one or more IPSec Security Associations (SAs). These are formed during Phase 2 using a process called Quick Mode, which is the mechanism that negotiates the parameters used to protect real data traffic between the VPN peers.

IKEv1 Aggressive Mode: Fast, but at a Cost

Aggressive Mode in IKEv1 offers a quicker alternative to Main Mode by reducing the negotiation process to just three messages. Each message is denoted as AMx, where “x” represents the sequence number; odd numbers come from the initiator, and even numbers come from the responder.

While Aggressive Mode is faster and more efficient in bandwidth-constrained or latency-sensitive environments, it achieves this speed by sacrificing key security features. Notably, Aggressive Mode does not protect identities, meaning that both peers’ identification information is exposed during the exchange. Additionally, the responder is not required to authenticate first, opening a small window of vulnerability during negotiation.

Perhaps the most serious concern is that if pre-shared keys (PSK) are used, an attacker who captures the exchange can potentially perform an offline brute-force attack. Since the PSK is not protected in the same way as it is in Main Mode, this makes Aggressive Mode a risky choice for security-sensitive environments.

Due to these limitations, Aggressive Mode is generally avoided in modern secure VPN deployments in favor of Main Mode or, better yet, IKEv2.