When it comes to securing communications over public or untrusted networks, understanding the types of encryption and their application within IPSec is critical. Encryption can be symmetric or asymmetric, each with distinct use cases in network security.

Symmetric vs Asymmetric Encryption

Symmetric encryption uses the same key for both encryption and decryption. While it’s fast and efficient, it requires secure key exchange, which can be a challenge. Algorithms include:

  • DES: An older 56-bit algorithm, now considered broken (crackable in under 24 hours).
  • 3DES:  An enhancement using three 56-bit keys for encrypt-decrypt-encrypt chaining.
  • AES:  The modern standard, available in 128-bit and 256-bit variants.

Asymmetric encryption, by contrast, uses a key pair: one public, one private. This is the basis for secure certificate-based communication. Examples include:

  • Digital Certificates
  • RSA Signatures

Public Key Infrastructure (PKI)

PKI provides the trust model for secure identity exchange on insecure networks. Here’s a simplified flow:

  1. A host generates a certificate request and signs it with its RSA private key.
  2. It sends this to a certificate authority (CA).
  3. The CA issues a signed certificate and returns it.
  4. The host stores and uses this certificate to authenticate itself to peers.

This enables mutual authentication without the need for manual key exchange, a significant improvement over pre-IPSec models.

IPSec: AH and ESP

IPSec includes two main protocols for securing packets: AH (Authentication Header) and ESP (Encapsulating Security Payload). Each offers different benefits.

AH (Authentication Header)

AH provides:

  • Integrity
  • Data Origin Authentication
  • Anti-Replay Protection

It uses IP Protocol 51 and includes the Integrity Check Value (ICV) to detect tampering. However, AH authenticates the entire IP packet, including the outer IP header. This makes it incompatible with NAT, as any change (e.g., in the source IP or TTL) will cause ICV validation to fail. Due to this limitation, AH is rarely used in modern deployments where NAT is common.

ESP (Encapsulating Security Payload)

ESP offers:

  • Confidentiality (encryption)
  • Integrity
  • Authentication
  • Anti-Replay

Using IP Protocol 50, ESP can encrypt the payload and authenticate the data without validating the outer IP header, making it compatible with NAT and PAT. It also supports NAT Traversal (NAT-T) and can operate with null encryption (ESP-NULL) when only integrity and authentication are needed.

The difference is visual in packet handling:

  • In Transport Mode, the original IP header is retained; only the payload is encrypted and/or authenticated.
  • In Tunnel Mode, a new IP header encapsulates the entire packet, hiding source/destination IPs – useful for VPN scenarios.

ESPv3 and Extended Sequence Numbers

ESPv3, as defined in RFC4303, enhances the Encapsulating Security Payload (ESP) protocol used within IPSec. Traditionally, IPSec uses 32-bit sequence numbers to track packets and ensure protection against replay attacks. However, ESPv3 introduces 64-bit sequence numbers, also known as Extended Sequence Numbers (ESN). This dramatically increases the number of packets that can be sent before requiring a rekey operation, enhancing performance and scalability. While ESN is a valuable feature in high-throughput environments, it’s worth noting that Cisco IOS and IOS-XE previously did not support ESN, making it a consideration during device selection or IOS version checking.

Traffic Flow Confidentiality and Dummy Packets

To protect against traffic analysis, even when the contents are encrypted, IPSec offers Traffic Flow Confidentiality (TFC). This is achieved by adding padding to the ESP header, which can be up to 255 bytes in length. While this padding doesn’t encrypt more data, it helps mask the actual size and structure of network traffic, making it more difficult for attackers to infer communication patterns.

In addition to padding, administrators can configure dummy packets, non-functional packets sent at random intervals. These are used to further disguise the real flow of encrypted traffic or shape it to meet a predictable pattern. This helps prevent attackers from identifying when meaningful communication occurs, even if they can’t read the encrypted content.

NAT-Traversal (NAT-T)

A common challenge in VPN deployments is maintaining secure tunnels when Network Address Translation (NAT) is present. NAT-T resolves this issue by adding a UDP header (typically using UDP port 4500) before the ESP header, enabling NAT devices to recognize and process the packets properly.

NAT-T works as follows:

  • NAT-T Support: During IKE Phase 1, VPN peers exchange vendor and version information to determine if NAT-T is supported on both sides.
  • NAT-T Detection: During Phase 1, peers create a payload containing their external IPs. These are hashed and exchanged. If the hashes do not match, it means NAT is modifying the IPs in transit, indicating the presence of NAT in the path.
  • NAT-T Decision: During IKE Phase 2, if NAT is detected, a UDP header with port 4500 is prepended to ESP packets to enable proper NAT traversal.

This approach allows encrypted VPN traffic to pass through NAT devices without breaking authentication or encryption mechanisms, which would otherwise fail due to changes in packet headers.

Finally, it’s important to remember that both AH and ESP support Transport and Tunnel modes. In Transport mode, only the payload is protected, and the original IP header remains visible. In Tunnel mode, the entire original packet is encapsulated and protected. Regardless of the mode used, when NAT-T is active, ESP is the preferred protocol due to its compatibility with NAT, while AH typically cannot be used in NAT scenarios.

Crypto Maps, Tunnel Interfaces, and Crypto Sockets in IPSec

When configuring IPSec on Cisco IOS devices, two major approaches exist for applying encryption: crypto maps and tunnel interfaces with tunnel protection. Understanding how each works is key to designing and troubleshooting secure IP networks.

Crypto Maps

A crypto map is a traditional IOS mechanism for defining IPSec security associations (SAs). It acts as a configuration attribute that binds encryption policies, access control lists (ACLs), and peer configurations to a specific interface. Once applied, it enables the Security Policy Database (SPD) on that interface, identifying traffic that IPSec should protect.

Crypto maps are applied directly to interfaces, either physical or, with some limitations, tunnel interfaces, and perform IPSec protection before any interface-level encapsulation. This setup is effective for static or policy-based VPNs, where specific traffic flows are protected based on matching Access Control Lists (ACLs).

Tunnel Interfaces and Tunnel Protection

Cisco IOS also supports tunnel interfaces with tunnel protection, which represent logical interfaces used to interconnect a transport network with an overlay network. These tunnels encapsulate traffic using methods like GRE or IPSec and are typically configured as Virtual Tunnel Interfaces (VTIs).

Tunnel protection is only applicable to tunnel interfaces and applies IPSec policies after interface-level encapsulation has occurred. This is particularly useful when the entire interface’s traffic must be encrypted, regardless of source/destination pairs. Once the traffic enters the tunnel, it is encapsulated (often with GRE), and then encrypted using IPSec. This approach contrasts with crypto maps, which encrypt before encapsulation.

Tunnel Protection and Crypto Sockets

Tunnel protection is implemented by using the tunnel protection command, which attaches an IPSec profile to a tunnel interface. This ensures all packets passing through the tunnel are encrypted accordingly. The crypto policy is consistently applied, regardless of whether it is in a GRE or VTI setup.

A key feature supporting tunnel protection is crypto sockets. These are internal IOS constructs that represent IPSec sessions similarly to how network sockets represent TCP sessions. Each crypto socket is dynamically created for every unique IPSec VPN session (comprising IKE and IPSec SAs) and exists independently of IKE negotiation state.

To inspect them, use the command:

show crypto sockets

This provides visibility into sockets associated with tunnel protection mechanisms. Note that if a crypto map is in use, show crypto sockets will yield no output, as crypto maps don’t rely on sockets.

For a broader view across both crypto map and tunnel protection deployments, use:

show crypto session

This command displays active IKEv1/IKEv2 security associations, IPSec SAs, and session summaries for all peers, regardless of whether the SAs were created via crypto maps or tunnel protection.