Understanding IPSec Modes: Transport vs. Tunnel
When it comes to securing IP traffic, IPSec offers two main modes of encapsulation: Transport mode and Tunnel mode. Both modes serve the purpose of protecting data through encryption and integrity checks, but they differ significantly in how they encapsulate and route packets.
Transport Mode: Focused Encapsulation for Next-Layer Protocols
Transport mode is commonly used when security is applied directly between two endpoints, especially in overlay scenarios like tunnel interfaces. However, it’s important to note that in Transport mode, ESP (Encapsulating Security Payload) doesn’t completely encapsulate the original packet. Instead, it only encrypts the payload and adds a new ESP header between the original IP header and the transport layer payload (e.g., UDP or TCP).
This mode is handy when you want to avoid the overhead of an additional IP header. One advantage of Transport mode is its efficiency; there is no duplication of the IP header, unless it’s combined with a tunnel interface, in which case duplication may be necessary. It’s ideal for protecting next-layer protocols, such as UDP or TCP, without altering the original IP routing.
This transformation protects the payload while retaining the original IP header. Transport mode is typically deployed using crypto maps on routers and firewalls where traffic is protected as it passes through physical interfaces, not tunnel interfaces.
Tunnel Mode: Full Encapsulation for Secure Routing
Tunnel mode is the default IPSec mode on Cisco routers, and it takes a more comprehensive approach. It encapsulates the entire original IP packet (including the original IP header) within a new IP packet. A fresh IP header is added outside the ESP packet, allowing the traffic to be routed securely across untrusted networks.
Here’s what a Tunnel mode packet might look like:
Because the original IP header is entirely hidden inside the new packet, Tunnel mode is ideal for site-to-site VPNs, where private IP ranges are used internally and need to be securely transported across a public network.
However, this additional encapsulation comes at the cost of increased overhead:
- For IPv4: ~20 bytes for the new outer IP header (possibly more with options)
- For IPv6: ~40 bytes if extension headers are present
- Plus additional overhead from the ESP header and trailer
Despite the increased overhead, Tunnel mode provides robust security and is widely used due to its simplicity and compatibility with various topologies.
Deep Dive into IPSec: Configuration, Modes, and Cryptographic Foundations
Before diving into configurations, it’s important to consider MTU (Maximum Transmission Unit) sizing. Since IPSec, particularly ESP (Encapsulating Security Payload), adds overhead bytes to each IP packet, it’s possible that packets could exceed the interface’s MTU, leading to fragmentation or drops. Most modern routers automatically account for this when crypto is configured, especially when using tunnel interfaces. Still, network engineers should always verify and, if needed, manually adjust MTU settings or enable Path MTU Discovery.
IPSec Tunnel Configuration: Step-by-Step Example
Let’s walk through an example IPSec configuration.
1. Before configuring, it’s best practice to check for existing ISAKMP policies using:
show crypto isakmp policy
2. If none exist, you can define one as follows:
crypto isakmp policy seq-num encryption group hash authentication
Note: rsa-sig uses a certificate and rsa-encr is not very popular and depreciated IKEV2 . It was using public and private rsa keys without certificates to authenticate.
3. Define your shared key:
crypto isakmp key <key> address <ip-address-of-remote-peer-tunnel-endpoint>
You may verify the above ISAKMP configuration with:
show run | sec crypto isak
4. The configure the IPSec configurations:
crypto ipsec transform-set
mode
Note: The default is tunnel. No need to change it if that is what you want.
5. Set up your proxy identity using an ACL:
access-list 120 permit ip 192.168.2.0 0.0.0.255 1192.168.1.0 0.0.0.255
6. Bind it all together using a crypto map:
crypto map <name> <seq-number> ipsec-isakmp
set peer <peer-ip>
set transport-set <name-previously-used>
match address <ACL-number>
7. Apply the map to the interfaces:
interface g1 crypto map <name>
This configuration is applied to UDP port 500, the default for ISAKMP. If NAT is involved, UDP port 4500 will also be used. You can verify this with:
show udp
At this point, the tunnel won’t be fully established until interesting traffic matches the ACL, triggering the Security Association (SA) negotiation.
To verify your setup, use:
show crypto map
IPSec SA Lifetime and Additional Insights
The Security Association (SA) lifetime is either 4GB of data or 1 hour, whichever comes first. Using show crypto map, you can also confirm whether Perfect Forward Secrecy (PFS) is enabled, what interface the map is on, and the active transform-set.
IPSec Databases and Cryptographic Structure
There are three essential databases used in IPSec:
- ISAKMP SA Database: Manages phase 1 negotiations
- IPSec SA Database: Stores established phase 2 security associations
- Security Policy Database (SPD): Dictates what traffic gets encrypted or bypassed
The cryptographic suite used in IPSec VPNs consists of four algorithm types:
- Encryption algorithm (e.g., AES, 3DES)
- Integrity algorithm (e.g., SHA1, SHA256)
- Diffie-Hellman group for key exchange
- PRF (Pseudorandom Function) for key derivation
In IPSec, asymmetric cryptography is used during key exchange, but symmetric cryptography is used for bulk data encryption due to its efficiency. The key used for encryption during an IPSec session is known as SK_e. This same key is used by the peer for decryption, reflecting symmetric behavior.
Breaking Down IPSec SA Components: SAD, SPD, PAD, and Beyond
Understanding how IPSec works under the hood requires examining its core components. These aren’t just abstract technicalities; they’re essential to how encryption, policy enforcement, and secure communications function between peers. At the heart of this framework are three key elements: the Security Association Database (SAD), the Security Policy Database (SPD), and the Peer Authorization Database (PAD).
Security Association Database (SAD)
The SAD defines how traffic is protected across an IPSec connection. It holds parameters associated with each unidirectional IPSec Security Association (SA). Each SA is built on negotiated parameters that include the security protocol (ESP or AH), whether transport or tunnel mode is used, the cryptographic algorithms applied, and the local/remote peer addresses. Every IPSec peer maintains a separate SAD for each instance of the SPD.
Each SAD entry typically includes:
- Peer ID: the remote identity of the peer
- SPI (Security Parameter Index): a unique 32-bit identifier used to look up the correct SA for incoming packets
- Protocol info: such as ESP or AH, indicating what protection mechanism is in use
The SPI is derived from a hash of the SPD, ensuring a tight binding between the traffic selectors and the security parameters.
Security Policy Database (SPD)
While the SAD instructs the router on how to protect the traffic, the SPD determines which traffic requires protection. It defines the IPSec boundary between secure and unsecured interfaces, specifying which flows are subject to encryption, authentication, or both.
Key parameters in the SPD include:
- Encryption algorithms such as DES, 3DES, or AES
- Hashing algorithms like MD5 or SHA-1 for data integrity
- IPSec mode – tunnel or transport
- SA lifetime – defined in seconds or kilobytes, specifying how long a session can remain valid
Together, SPD entries ensure that specific traffic flows are matched against the right protection profiles.
Peer Authorization Database (PAD)
The PAD enforces identity-based policy rules and is the bridge between the SPD and the IKE negotiation process. It defines which remote identities (usually based on IKE identifiers like FQDNs or IP addresses) are authorized to negotiate SAs and with which policies. It essentially links authenticated peers with acceptable security policies.
Each PAD entry includes:
- The remote identity or group of peers
- The allowed authentication method (e.g., pre-shared key, RSA signatures, certificate-based auth)
- Any additional parameters, like certificate revocation lists
This setup enables granular control over who is allowed to establish secure tunnels and under what conditions.
Other Key Elements of an SA
Every SA is uniquely identified by a Security Parameter Index (SPI), a 32-bit value embedded in the IPSec header that tells the receiving device which SA to use for processing the packet. This SPI ties back to the SAD entry.
The lifetime of an SA can be defined in two ways: time-based (e.g., 1 hour) or volume-based (e.g., 4 GB of traffic). Once the lifetime is reached, a rekey or renegotiation process begins to ensure continued protection without interruption.
