VPN Deep Dive: IPSec Overhead and Fragmentation

 

When deploying IPSec, understanding overhead and fragmentation is essential for efficient and secure network design. Several factors contribute to IPSec encapsulation overhead, especially when using tunnel mode. This includes the addition of a new IP header in tunnel mode and optional encapsulation methods like GRE over IPSec, which introduces GRE overhead. Additional headers, such as the Authentication Header (AH) or Encapsulating Security Payload (ESP) header and trailer, also contribute to increased packet size. ESP-specific overhead arises from the payload structure and the characteristics of the encryption algorithm used, particularly the plaintext length, block size, and padding requirements. The integrity protection mechanisms further affect overhead, especially when using ESP with integrity checks or AH. The size of the output from the integrity algorithm, along with padding necessary to align the payload or integrity check value (ICV), also factors into the total overhead. In addition to this, Traffic Flow Confidentiality (TFC) padding may be added, particularly in environments where packet size uniformity is crucial for security.

The introduction of IPSec version 3 (as defined by RFCs 4301 through 4303) brought several key advancements. Notably, it introduced combined-mode algorithms and the concept of Traffic Flow Confidentiality. Combined-mode algorithms, such as AES-GCM, offer both encryption and integrity in a single algorithm, eliminating the need for a separate integrity function. Some combined-mode variants, such as AES-GMAC, provide only integrity, which can be used with ESP to transmit data that is unencrypted but still authenticated. This flexibility is helpful in scenarios where authentication is sufficient but encryption is not required or is already handled at a different layer.

To help obscure the traffic pattern of encrypted flows, IPSec supports the generation of dummy packets, a feature facilitated by TFC. These dummy packets are injected into the encrypted traffic stream to mask the actual volume and frequency of honest communication. On Cisco IOS, dummy traffic can be enabled with the following commands:

• crypto ipsec security-association dummy pps 0-25 to set the number of dummy packets per second
• crypto ipsec security-association dummy seconds 1-3600 to set dummy packet transmission intervals

Though not commonly seen in production, it is technically possible to combine ESP and AH in either tunnel or transport mode, resulting in double encapsulation. This configuration is rare and generally reserved for particular use cases that require both confidentiality (via ESP) and separate authentication (via AH).

 

When analyzing IPSec deployments, understanding the overhead implications of various encapsulation methods is essential, especially when working with or without GRE and across different IP versions. IPSec encapsulation overhead can differ significantly depending on whether it’s operating in tunnel or transport mode and whether it’s combined with protocols like GRE.

 

IPSec Mode Overhead (Without GRE)

In tunnel mode, both ESP (Encapsulating Security Payload) and AH (Authentication Header) introduce additional IP headers in front of the original packet. This new header is 20 bytes for IPv4 and 40 bytes for IPv6 under standard conditions, though it can be larger if IP options (IPv4) or extension headers (IPv6) are used. In IPv4, the protocol field in the new header is set to 50 for ESP or 51 for AH, which helps define the next encapsulation layer in the processing chain. For IPv6, the ESP or AH component is not a new header but rather added as an extension header, which modifies the encapsulation behavior slightly.

In contrast, the transport mode with IPSec does not add an extra IP header; only the security headers are inserted, and the original IP header remains intact. This results in lower overhead and is commonly used in host-to-host VPN scenarios. A quick comparison in terms of IP header overhead:

 

GRE Overhead and IPSec Interactions

When GRE (Generic Routing Encapsulation) is combined with IPSec, the encapsulation becomes more complex, resulting in two layers of overhead. First, the original packet is encapsulated within a GRE header. This GRE-encapsulated packet is then encrypted by IPSec.

In transport mode, the IP header from the GRE encapsulation is retained, and only the GRE header is included in the plaintext data. The original IP header and payload remain mostly untouched apart from the security layer. In tunnel mode, however, both the IP and GRE headers become part of the plaintext, and IPSec adds yet another IP header on top, further increasing the overhead.

The number of bytes added to the header and plaintext can influence padding and encryption behavior, as IPSec requires specific block sizes for its cryptographic functions. The more overhead you introduce, the more likely you are to need additional padding.

It’s also worth noting that using GRE with tunnel mode on Cisco IOS may be unnecessary in many scenarios. It adds an extra IP header that serves no practical function unless you’re dealing with NAT traversal, particularly where many hosts are behind a PAT (Port Address Translation) device but each uses unique internal private IP addresses.

 

Visual Comparison

The diagrams make this difference clear:

• Transport Mode with GRE over IPSec: The IPSec payload includes the GRE header and the original packet (including its original IP header). The GRE packet’s IP header remains outside the encrypted payload.
• Tunnel Mode with GRE over IPSec: A new IP header is added by IPSec, and both the IP and GRE headers from the original GRE packet are inside the IPSec payload, encapsulating the entire original packet. This results in more overhead but is sometimes necessary for routing and NAT scenarios.

Ultimately, the choice between transport and tunnel mode, and whether to use GRE, depends on the specific routing, NAT, and traffic flow requirements of your network. Understanding the overhead impact of each encapsulation choice is critical for optimizing performance and ensuring compatibility with network devices and encryption policies.

 

Understanding Encapsulating Security Payload (ESP) Overhead in IPSec

Encapsulating Security Payload (ESP) is one of the core components of IPSec that ensures the confidentiality, and optionally the integrity, of transmitted data. ESP achieves this by adding both a header and a trailer around the original payload, creating an encapsulated structure that significantly impacts packet size and structure.

The ESP header includes two critical 32-bit (4-byte) fields: the Security Parameter Index (SPI) and the Sequence Number. These values are not encrypted and help the receiving system identify and validate the security association and detect replay attacks. Following the header is the payload data, which is the encrypted portion of the original packet. This may also include additional fields, such as an Initialization Vector (IV), depending on the encryption algorithm in use.

ESP also appends a trailer to the end of the encrypted payload. The trailer includes two 8-bit (1-byte) fields: the Pad Length and the Next Header. The Pad Length indicates how many padding bytes were added, while the Next Header specifies the type of data being carried in the payload (such as TCP, UDP, or another protocol). These two bytes are the last encrypted part of the ESP payload and are essential for the correct interpretation of the data upon decryption.

If an integrity algorithm is used as part of the transform set, the ESP trailer includes an Integrity Check Value (ICV). This ICV is calculated over the entire ESP packet and ensures data integrity and authenticity.

Padding in ESP serves two main purposes. First, it ensures that the plaintext (including the trailer fields) aligns with the block size required by the encryption algorithm. Second, it ensures the ICV begins on a 4-byte boundary, which may be required for some implementations. Padding can range from 0 to 255 bytes and is inserted between the payload and the trailer.

Overall, the ESP overhead includes:

• 8 bytes for the ESP header (SPI and Sequence Number),
• A variable amount of padding (0–255 bytes),
• 2 bytes for the fixed trailer (Pad Length and Next Header),
• A variable-length ICV, depending on the integrity algorithm used,
• And potentially some extra bytes within the encrypted payload, like an IV, depending on the encryption mode.

This overhead must be considered when designing IPSec implementations, as it affects both bandwidth usage and MTU (Maximum Transmission Unit) calculations. The actual size of the ESP overhead will vary depending on the encryption and integrity algorithms used, as well as the structure of the data being transmitted.

 

Understanding Authentication Header (AH) Overhead in IPSec

The Authentication Header (AH) protocol in IPSec provides data integrity, data origin authentication, and optional anti-replay protection. Unlike ESP, AH does not offer encryption. It simply adds a header in front of the encapsulated packet, making its overhead more lightweight but still significant for certain use cases.

The AH includes several critical fields. It begins with two 8-bit (1-byte) values: the Next Header and the Payload Length. The Next Header field functions similarly to the equivalent field in ESP, identifying the protocol of the packet that follows. The Payload Length field indicates the size of the AH, but not the length of the actual payload. This is followed by 2 bytes (16 bits) that are reserved and set to zero. Next are two 32-bit (4-byte) fields: the Security Parameter Index (SPI) and the Sequence Number. Together, these fields total 12 bytes, forming the fixed-length portion of the AH.

The final part of the AH is the Integrity Check Value (ICV), which is calculated using the integrity algorithm defined in the transform set. The ICV ensures that the contents of the packet have not been tampered with and are authentic. Depending on the algorithm used, the ICV length may vary.

In terms of overhead, AH contributes:

• 12 bytes for the fixed-length header,
• A variable-length ICV, depending on the chosen integrity algorithm,
• And up to 7 bytes of padding, depending on alignment requirements.

When AH is used with a combined-mode algorithm like AES-GMAC, which integrates integrity into a single cryptographic function, the ICV may also be accompanied by additional data such as an Initialization Vector (IV) used by the algorithm. This can add a few extra bytes to the total overhead, though the exact amount depends on the specific cryptographic implementation.

While AH does not encrypt the data, its ability to authenticate both the payload and portions of the IP header makes it valuable in scenarios where data integrity and origin verification are critical, but confidentiality is not a requirement.

 

Encryption Overhead in ESP: Understanding Padding, IVs, and Cipher Requirements

When using the Encapsulating Security Payload (ESP) protocol in IPSec, encryption overhead is primarily dictated by the behavior of the encryption algorithm used. This includes how the algorithm structures the encrypted payload, how it handles padding, and whether it requires an Initialization Vector (IV). In most cases, block ciphers are used, which operate on fixed-size blocks of data. This means that plaintext must be padded to align perfectly with the block size before encryption can occur.

To accommodate this, ESP packets include explicit padding. As shown in the diagram, original plaintext is padded to reach the next full block size. This padded data is then encrypted into ciphertext. During decryption, the same padding must be stripped to return to the original plaintext. The amount of padding added depends on the block size of the cipher and the length of the plaintext payload. This overhead is essential but often variable, depending on the exact size of the data being encrypted.

Encryption algorithms that operate in modes such as Cipher Block Chaining (CBC) or Counter (CTR) mode require an Initialization Vector (IV). This IV must be transmitted along with the encrypted payload, typically placed before the ciphertext in the ESP payload. The IV ensures that the same plaintext block encrypted multiple times will yield different ciphertexts, improving security against replay and pattern attacks.

The size of the IV corresponds to the block size of the encryption algorithm, not the key size. For instance, AES-256 still uses a 128-bit IV, even though the key size is 256 bits. This distinction is important: block size influences how the encryption is performed, while key size affects the algorithm’s cryptographic strength.

Here’s a quick reference for some of the most common encryption algorithms used in ESP and their associated block/IV sizes and key sizes:

Selecting the right encryption algorithm involves a tradeoff between performance, overhead, and security. AES is widely preferred today due to its efficiency and robust security, particularly in GCM and CTR modes. However, network designers must always account for encryption-related overhead, including padding and IV size, when calculating the MTU and optimizing VPN throughput.

 

IPSec Integrity and Combined-Mode Algorithm Overhead Explained

When implementing IPSec, understanding the overhead caused by integrity algorithms is essential for accurately calculating packet sizes and ensuring interoperability across devices. Integrity overhead arises from the use of hash functions or algorithms that generate an Integrity Check Value (ICV). This value is appended to packets and used to verify the authenticity and integrity of the transmitted data.

The size of the ICV depends on the specific hash algorithm in use. For instance, SHA-1 produces a 160-bit (20-byte) output, and MD5 produces a 128-bit (16-byte) output. However, in IPSec implementations, these values are commonly truncated to 96 bits (12 bytes) to minimize overhead while still providing adequate security. This truncation is standardized and ensures compatibility across vendors and platforms.

To streamline encryption and integrity functions, modern IPSec deployments often utilize combined-mode algorithms, such as AES-GCM (Galois/Counter Mode). These algorithms handle both encryption and integrity in a single pass, reducing computational complexity and configuration effort. AES-GCM returns both a ciphertext and an ICV, eliminating the need for a separate integrity algorithm.

A unique variant of combined-mode use is AES-GMAC, which only performs integrity protection. AES-GMAC is useful in scenarios where encryption isn’t needed but authentication is still required. This algorithm can be used with either ESP (as a combined-mode function) or AH (as a standalone integrity function), but in both cases, it requires the use of an Initialization Vector (IV).

Combined-mode algorithms like AES-GCM, AES-CCM, and AES-GMAC share several common features:

• They allow any input size, meaning no input padding is required.
• The ciphertext output is equal in length to the input plaintext.
• An IV must be included in the ESP packet.
• AES is the underlying encryption algorithm, and the supported key sizes are 128, 192, and 256 bits.

Here’s a comparison of the most commonly used combined-mode algorithms and their properties:

Combined-mode algorithms are highly efficient and are widely adopted in high-performance and security-sensitive environments. Their integration simplifies IPSec configuration while offering robust protection for data in transit.

 

Understanding IPSec Overhead and Plaintext MTU Impact

When deploying IPSec, managing packet size and Maximum Transmission Unit (MTU) is critical to avoiding fragmentation and ensuring performance. IPSec overhead is not solely determined by the transform set (encryption and integrity algorithms); it also depends on the length of the original plaintext packet, due to the inclusion of padding for block alignment.

In tunnel mode, the packet is encapsulated with new headers and trailers, adding significant overhead. For example, when using ESP tunnel mode, combined with AES-CBC-128 for encryption and HMAC-SHA1-96 for integrity, the overhead varies depending on how the payload aligns with the cipher’s block size. As a result, encrypted packets may be longer than expected, even for small differences in input size.

Here’s a snapshot from actual calculations:

 

Notes:

• Overhead increases when padding grows to align with the AES block size (16 bytes).
• Smaller packets may benefit from minimal padding, while packets that cross certain thresholds (like from 1438 to 1439 bytes) may suddenly incur higher overhead.

Cisco IOS calculates IPSec overhead by considering the IP MTU on the egress interface and finding the largest plaintext packet that can fit inside that MTU after IPSec encapsulation. This helps avoid fragmentation but requires precise calculation.

A reliable method to estimate the maximum plaintext packet size is to subtract the overhead from the interface MTU. For example, with an interface MTU of 1496 bytes and the largest plaintext that fits being 1438 bytes, the effective overhead is 58 bytes. The disadvantage of this method is that it must be recalculated any time the output MTU changes, making automation or dynamic adjustment mechanisms ideal for production networks.

Carefully managing MTU and IPSec overhead ensures your VPN tunnels stay efficient and fragmentation-free, especially when combined with proper Path MTU Discovery or explicit configuration.

 

Understanding Maximum Overhead in IPSec: Padding, Output, and Alignment

In IPSec deployments, accurately estimating maximum overhead is vital for determining MTU behavior, minimizing fragmentation, and ensuring interoperability. The total overhead introduced by IPSec is a combination of factors that vary based on the type of algorithm used—whether it’s encryption, integrity, or combined-mode.

Maximum input padding refers to the largest amount of padding that may be added to the plaintext to align it with the required block size for the encryption algorithm. Block ciphers like AES operate on fixed-size blocks (e.g., 16 bytes), so any plaintext that doesn’t naturally align with this size must be padded. The amount of padding added ensures that the input fits the algorithm’s block boundary exactly.

Maximum output overhead accounts for the extra data produced by the algorithm in addition to the encrypted payload. This includes fields such as the Initialization Vector (IV), Integrity Check Value (ICV), or both. In the case of combined-mode algorithms like AES-GCM, this overhead is added on top of the ciphertext, which is itself the same size as the original plaintext plus any input padding. Therefore, the total output becomes: padded plaintext + IV + ICV.

Maximum ICV padding is another important consideration. For encryption algorithms, this involves padding added before the ICV to align it to a 4-byte boundary. For integrity-only algorithms—especially when used with AH (Authentication Header)—this includes the padding required after the ICV to align the entire payload to either a 4-byte or 8-byte boundary, depending on the architecture and protocol requirements.

These maximum overhead values must be considered when calculating MTU or configuring security associations. Misjudging them can lead to packet drops, excessive fragmentation, or compatibility issues, particularly in constrained or high-performance environments.

 

IPSec Maximum Overhead: Comprehensive Breakdown of Worst-Case Encapsulation

This section takes a deeper dive into worst-case scenarios for ESP and AH, offering a full accounting of maximum padding, output, and integrity-related overhead. These insights are essential for MTU planning, tunnel optimization, and capacity design in production environments.

 

Overhead by Algorithm Type

Overhead introduced by encryption, integrity, and combined-mode algorithms varies based on input padding, output components like IVs or ICVs, and protocol alignment requirements. The table below summarizes maximum input padding, output overhead, and ICV padding for commonly used algorithms:

Note: Algorithms like AES-GCM and AES-CCM use block sizes that are naturally 4-byte aligned, removing the need for post-ICV padding in most cases.

 

Calculating Maximum ESP Overhead

To compute the maximum ESP overhead, sum the following:

• 20 bytes (tunnel mode header over IPv4), or
• 40 bytes (tunnel mode header over IPv6), or
• 24–44 bytes (GRE/IP encapsulation), if GRE tunnel key is configured
• 10 bytes (ESP header and trailer: SPI + Seq Num + Pad Length + Next Header)
• Maximum input padding (e.g., 15 bytes for AES-CBC)
• Maximum output overhead (e.g., 16 bytes IV + 12 bytes ICV for AES-CBC + HMAC-SHA1-96)
• Maximum ICV padding (if required, such as 4 bytes on IPv6)

This additive approach provides the upper boundary of overhead for any given packet and is vital for determining the maximum plaintext size that can fit into a given MTU.

 

Calculating Maximum AH Overhead

When using AH instead of ESP, the overhead is different but similarly cumulative:

• 20 or 40 bytes for tunnel IP headers (IPv4 vs. IPv6)
• 24 or 44 bytes if GRE is involved
• 12 bytes for the AH fixed header
• Max output overhead of the integrity algorithm (e.g., 12 bytes for HMAC-SHA1-96)
• Max ICV padding (4 bytes in IPv6 environments)

AH adds its header before the original packet and covers more of the IP header for integrity, making it stricter in terms of alignment and padding.

 

Example: Worst-Case Encapsulation Breakdown

The diagram provided illustrates a worst-case scenario with AES-CBC and HMAC-SHA1-96 under ESP tunnel mode. It depicts a 31-byte plaintext IPv4 packet (20-byte IP header + 11-byte payload). After adding ESP structures, IV, and padding to align to the 16-byte AES block size, the final structure looks like this:

• 20 bytes – Tunnel Mode IP Header
• 8 bytes – ESP Header (SPI + Sequence Number)
• 16 bytes – IV
• 31 bytes – Encrypted original packet (20-byte header + 11-byte payload)
• 15 bytes – Padding
• 2 bytes – Trailer fields (Pad Length + Next Header)
• 12 bytes – ICV

The padding ensures that all encrypted blocks are aligned to AES’s 16-byte block size. The total overhead for this encapsulation case approaches 73 bytes, a non-trivial addition for MTU-sensitive applications.

Understanding and planning for maximum IPSec overhead is critical to avoiding fragmentation and ensuring application performance. As tunnel configurations grow more complex, with GRE, NAT-T, and combined-mode algorithms, so too does the encapsulation math. Using these tables and formulas can help you proactively manage MTU and packet sizing in real-world VPN deployments.

 

Understanding MTU in Cisco Networks: Behavior, Configuration, and Forwarding Implications

The Maximum Transmission Unit (MTU) is a critical parameter in networking, defining the largest Protocol Data Unit (PDU) that can be transmitted over a Layer 2 link without fragmentation. In the context of Cisco routers, the MTU of a physical interface is typically determined by the interface type itself. For instance, virtual Ethernet interfaces may have their own default MTU settings. Administrators can override this value using the mtu command to accommodate additional encapsulation, such as IPSec or GRE, or to match the MTU of the underlying physical infrastructure.

By default, the interface-level MTU applies globally to all protocols and is visible via the show interfaces command. However, Cisco IOS also allows for more granular control by letting you configure IP protocol-specific MTUs. You can do this with the ip mtu and ipv6 mtu commands, which affect only IPv4 or IPv6 traffic, respectively. These values are reflected in the show ip interfaces and show ipv6 interfaces outputs, making it easier to verify IP-layer MTU settings independently from the Layer 2 MTU.

When a router forwards packets, it checks the packet length against the MTU of the outbound interface before applying any encapsulation. This MTU check occurs in the outbound direction only. If a packet exceeds the MTU (or the configured IP MTU), it cannot be forwarded in its current form. In the case of IPv4, the router may attempt to fragment the packet, assuming fragmentation is allowed. However, if fragmentation is disabled (e.g., the “Don’t Fragment” bit is set), the packet will be dropped, and an ICMP “Fragmentation Needed” message may be sent back to the source.

With IPv6, the behavior is stricter. IPv6 routers do not fragment packets; fragmentation must be handled entirely by the source host. If an IPv6 packet is too large for the outbound interface’s MTU, the router must drop the packet and send an ICMPv6 “Packet Too Big” message back to the sender. This behavior enforces Path MTU Discovery (PMTUD), which is crucial in ensuring IPv6 packets are appropriately sized for transit across diverse network paths.

Understanding and correctly configuring MTU values, especially in environments with tunneling protocols like IPSec or GRE, is essential to preventing fragmentation, dropped packets, and application performance issues.

 

IPv4 Fragmentation Explained: Fields, Behavior, and Troubleshooting

IPv4 supports packet fragmentation, which allows large datagrams to be broken into smaller fragments that can traverse networks with smaller Maximum Transmission Units (MTUs). Fragmentation can occur at any point along the path from source to destination, and fragments themselves may be fragmented further. Importantly, reassembly is always performed by the destination host, not intermediate routers. Fragments stay separated in transit, and intermediary nodes simply forward them as-is.

 

Key IPv4 Header Fields for Fragmentation

Three specific fields in the IPv4 header manage fragmentation:

• Identification (16-bit): Uniquely identifies fragments of the same original datagram.
• Flags (3-bit): Includes:
  • DF (Don’t Fragment): If set to 1, prohibits fragmentation.
  • MF (More Fragments): Set to 1 in all but the final fragment.
• Fragment Offset (13-bit): Indicates where a fragment belongs within the original packet, measured in 8-byte blocks.

RFC 791 mandates that the Identification field be unique for all packets in the same 3-tuple flow (source IP, destination IP, and protocol) during their time on the wire. On high-speed networks, the 16-bit field may wrap too quickly, resulting in fragment mix-ups.

 

Fragmentation Behavior and DF Bit Logic

When an IP datagram is created, the MF and Fragment Offset are initially set to 0. The DF bit may be set or cleared depending on local settings and whether Path MTU Discovery (PMTUD) is in use. If a packet exceeds the MTU at any point and the DF bit is set to 1, the packet is dropped, and the source is notified via an ICMP Type 3, Code 4 (Destination Unreachable – Fragmentation Needed and DF Set). This ICMP message includes the MTU value and the first 64 bytes of the offending packet, allowing the sender to adjust future packet sizes.

If the DF bit is set to 0, the packet can be fragmented:

• The original header is copied to each fragment.
• MF is set to 1 for all fragments except the last.
• Fragment Offset reflects the payload offset in 8-byte units.
• The Total Length and Header Checksum are recalculated per fragment.

For example, a 1500-byte payload passing through a 1400-byte MTU link might be split into:

• One 1400-byte fragment and one 120-byte fragment,
• Or 752 and 748 bytes, depending on hardware/software implementation.

The first fragment always contains the upper-layer headers (e.g., UDP), while subsequent fragments may only include payload. A critical rule: all fragments except the last must be multiples of 8 bytes, due to how Fragment Offset is measured.

 

Reassembly and Loss Behavior

Fragments are reassembled only at the destination, using the 4-tuple: source IP, destination IP, protocol, and Identification field. If all fragments don’t arrive in time, or memory is constrained, the system discards fragments, and no retransmission occurs. A single missing fragment results in the loss of the entire packet.

 

Troubleshooting IPv4 Fragmentation in Cisco IOS

Cisco IOS and IOS-XE provide tools for monitoring and debugging fragmentation:

• show ip traffic gives general stats.
• show platform hardware qfp active feature ipfrag global (IOS-XE) provides hardware-level counters.
• Use show ip traffic | include reassembled|fragmented to quickly identify fragmentation behavior.

Understanding fragmentation is essential when designing networks with tunnels, encryption, or multiple MTU domains. Misconfigured DF bits, excessive fragmentation, or blocked ICMP messages can all result in hard-to-diagnose application failures.

 

IPv6 Fragmentation: Design Philosophy, Header Behavior, and Troubleshooting

Unlike IPv4, where routers along the path may fragment packets, IPv6 strictly prohibits in-transit fragmentation. Only the originating host is permitted to fragment packets. This architectural change places full responsibility on the sender to ensure that packets are appropriately sized before transmission, and it relies heavily on Path MTU Discovery (PMTUD) to determine the optimal MTU along the path.

If an IPv6 packet exceeds the egress interface’s MTU, it is dropped immediately. The router that encountered the issue must generate an ICMPv6 Type 2, Code 0 (“Packet Too Big”) message and send it back to the source. This message includes the MTU of the link that caused the drop, allowing the source host to adjust its PMTU accordingly. However, the ICMPv6 message is only sent if the receiving interface has ipv6 unreachables enabled, which is the default on most Cisco platforms.

To meet minimum compatibility standards, all Layer 2 links carrying IPv6 traffic must support forwarding datagrams of at least 1280 bytes, including the IPv6 header. Anything below this threshold is considered non-compliant with the IPv6 specification.

 

IPv6 Fragmentation Header

If fragmentation is needed, it is indicated by the addition of a Fragment extension header. The original packet is split into two logical parts:

1. Unfragmentable Part: Includes the IPv6 base header and any hop-by-hop or routing extension headers. These are present in every fragment and are processed by intermediate routers.
2. Fragmentable Part: Contains any remaining extension headers and the actual payload. This is the portion that gets broken into multiple fragments.

Each fragment includes:

• The Unfragmentable Part (copied to all fragments)
• A Fragment extension header, which contains:
  • 13-bit Fragment Offset, specifying the payload offset (in 8-byte units)
  • M flag (More Fragments), set to 1 for all fragments except the last
  • 32-bit Identification field, used to group fragments for reassembly

Unlike IPv4, IPv6 does not use a DF (Don’t Fragment) bit, because fragmentation is explicitly prohibited during transit.

 

Reassembly Behavior in IPv6

Reassembly occurs only at the destination host, which uses a 4-tuple of source address, destination address, protocol, and identification to identify matching fragments. All fragments must arrive within 60 seconds of the first-arriving fragment. If reassembly is incomplete—whether due to delay, loss, or reordering beyond that window—any received fragments are discarded.

If the first fragment was received but reassembly failed (e.g., a subsequent fragment was lost or timed out), the receiver sends back an ICMPv6 Type 3, Code 1 (“Fragment Reassembly Time Exceeded”) message to the source.

 

Troubleshooting IPv6 Fragmentation on Cisco IOS

Cisco IOS provides several commands to monitor IPv6 fragmentation:

• show ipv6 traffic provides general traffic stats.
• show platform hardware qfp active feature ipfrag global (on IOS-XE) reveals hardware-level counters.

For quick checks, use:

show ipv6 traffic | include reassembled|fragmented

This modern approach to fragmentation reduces router overhead and encourages better traffic shaping at the source, but it also makes ICMPv6 filtering and MTU misconfiguration more impactful in real-world deployments.

 

Path MTU Discovery (PMTUD): How It Works and Why It Matters

Path MTU Discovery (PMTUD) is a crucial mechanism that enables IP hosts to determine the maximum transmission unit (MTU) that can be sent along a network path without requiring fragmentation. By default, a source host will attempt to send IP packets as large as the MTU configured on its outbound interface. However, this MTU may not reflect the smallest MTU along the entire path, especially when traversing different link types or encapsulated networks like VPNs.

 

IPv4 vs. IPv6 PMTUD Behavior

In IPv4, PMTUD relies on setting the Don’t Fragment (DF) bit in the IP header. When a packet exceeds the MTU of any link in the path, and the DF bit is set, the router discards the packet and returns an ICMP Type 3, Code 4 message (Fragmentation Needed and DF Set). This informs the source of the correct MTU to use.

In IPv6, PMTUD is inherently built into the protocol; there is no DF bit. If a packet is too large for any link, the router sends back an ICMPv6 Type 2 message (Packet Too Big), and the packet is dropped. In both protocols, the notification includes the MTU size that caused the failure, enabling the source to adjust.

 

PMTUD Operation and Adaptability

Once enabled, PMTUD stays active throughout the connection lifecycle. It monitors changes in the path MTU, which might occur due to route changes or dynamic reconfiguration. Since PMTUD is based on negative signaling, it only acts when a packet fails due to MTU limitations; there is no proactive or positive confirmation of a higher allowable MTU.

Typically, PMTUD is triggered at the beginning of a data session but may run multiple times if path conditions change. It’s especially well-suited for connection-oriented protocols like TCP, which can store and adjust to the optimal packet size per connection. TCP already maintains per-connection state that can include the effective MTU, so it naturally accommodates PMTUD updates.

For datagram protocols like UDP or custom transport stacks, PMTUD only works if the application can adapt packet sizes based on ICMP feedback. If this adaptation isn’t possible, packet loss or dropped connections may occur.

 

Alternative: Packetization Layer PMTUD

A more modern (but rarely implemented) variant of PMTUD is called Packetization Layer Path MTU Discovery. Instead of relying on ICMP messages, it uses probes at the transport layer (such as TCP) to actively test increasing packet sizes until it finds the maximum supported MTU. This method is more reliable in environments where ICMP messages are filtered or dropped. However, it is not widely deployed.

 

Key Takeaways for Network Engineers

• PMTUD is essential for efficient delivery of large packets, especially in tunneled or encapsulated environments.
• ICMP messages must be allowed through firewalls and routers for PMTUD to function correctly.
• PMTUD is reactive: MTU issues are detected after a packet fails.
• TCP is PMTUD-friendly; stateless protocols may need additional tuning or fallback behavior.

 

TCP MSS Clamping: A Practical Workaround for PMTUD Limitations

While Path MTU Discovery (PMTUD) is essential for avoiding fragmentation, it comes with limitations. PMTUD relies on the loss and retransmission of packets to detect path MTU restrictions and requires that all intermediate devices allow ICMP “Fragmentation Needed” messages to pass back to the source. If any filtering device blocks these ICMP messages, or if a device clears the DF bit (common in IPv4), PMTUD will break, potentially leading to silent packet loss or connection stalls.

To mitigate these issues, network engineers often implement TCP Maximum Segment Size (MSS) Clamping. This technique adjusts the MSS advertised during the TCP three-way handshake to ensure that packets stay within safe size limits, even when PMTUD fails. MSS defines the largest payload of a TCP segment (excluding headers) that a host is willing to accept. It is advertised, not negotiated, so each host independently informs its peer of its MSS during the SYN and SYN-ACK exchange.

 

How MSS Clamping Works

MSS represents only the payload portion of the TCP segment. It excludes the typical 20-byte IP header and 20-byte TCP header (though this can be more with options like timestamps). Therefore, to prevent fragmentation, MSS should be lower than the path MTU minus the combined header sizes.

For instance:

• If the path MTU is 1500 bytes (standard for Ethernet),
• Subtracting 20 bytes for the IP header and 20 bytes for the TCP header,
• The maximum safe MSS would be 1460 bytes.

However, in VPN or tunnel scenarios (e.g., GRE, IPSec), where encapsulation adds overhead, the path MTU might drop to around 1456 bytes. The adjusted MSS in this case would be 1456 – 40 = 1416 bytes.

 

Configuring MSS Clamping in Cisco IOS

MSS Clamping is applied on Cisco routers using the ip tcp adjust-mss command. This modifies the MSS value in SYN packets that pass through the router.

Example configuration:

interface g1
ip address 209.165.201.1 255.255.255.252
ip tcp adjust-mss 1416
ipv6 address 2001:DB8:FFF1::10/127

This command only affects TCP flows traversing the router, not those originating or terminating on the router. For those cases, use the ip tcp mss global configuration command.

Cisco best practices recommend applying ip tcp adjust-mss on both inbound and outbound interfaces that see SYN packets. Although setting it on one side may suffice in some scenarios, configuring it on both ends ensures consistency and avoids asymmetric behavior, especially in networks with route changes or policy-based routing.

 

Key Benefits and Considerations

• Bypasses the need for ICMP-based PMTUD, which may be blocked or filtered.
• Avoids fragmentation, improving throughput and reducing packet loss.
• Particularly useful in VPNs, tunnels, or firewalled environments with complex paths.
• Must be configured conservatively, as an MSS lower than necessary may degrade performance by reducing TCP efficiency.

TCP MSS Clamping is a reliable and proactive workaround for MTU issues that may evade detection by PMTUD alone. It’s especially critical in production networks where reliability, compatibility, and throughput are non-negotiable.

 

IPSec Fragmentation and Path MTU Discovery (PMTUD): Managing Encapsulation and Packet Size

IPSec tunnel mode adds encryption and encapsulation overhead, which can affect MTU-sensitive traffic. When a plaintext packet exceeds the Path MTU (PMTU) for a given Security Association (SA), fragmentation becomes a concern, particularly in tunnel mode over IPv4 or IPv6. IPSec handles these scenarios through a combination of configurable DF (Don’t Fragment) bit behavior, ICMP message processing, and encapsulation logic.

 

DF Bit Behavior in Tunnel Mode

In IPv4 tunnel mode, the outer IP header’s DF bit may be:

• Cleared
• Set
• Copied from the inner IPv4 header

All three options are configurable using the crypto ipsec df-bit {clear | set | copy} command. By default, the DF bit is copied from the inner packet. This behavior dictates whether a packet is eligible for fragmentation before or after IPSec encryption.

When a plaintext packet exceeds the Path MTU:

• If DF is clear, the packet can be fragmented before or after encryption (configurable).
• If DF is set or in the case of IPv6 traffic, the packet is dropped, and an ICMP (Type 3/4 for IPv4 or Type 2/0 for IPv6) message is sent to the source to enable PMTUD.

 

PMTUD and SA Path MTU Management

Each entry in the Security Association Database (SAD) includes a Path MTU value. This is initially computed as:

Output interface MTU – IPSec overhead

When the system receives an ICMP “Fragmentation Needed” or “Packet Too Big” message, it updates the SA’s Path MTU for the associated SPI (Security Parameter Index), using the SPI from the dropped packet to identify the SA entry.

Whether fragmentation happens before or after encryption is controlled via the command:

crypto ipsec fragmentation {before-encryption | after-encryption}

This applies to both ESP and AH, and by default, IPSec fragments before encryption.

 

Tunnel Interfaces vs IPSec Layer Behavior

It’s important to note that these commands apply only to the IPSec layer, not to the tunnel interface (e.g., VTI or GRE/IPSec). Tunnel interfaces are evaluated before IPSec encryption in the forwarding path, so their tunnel MTU determines whether the packet should be fragmented before encryption occurs.

For example:

• IPSec transport mode only allows fragmentation after encryption.
• IPSec tunnel mode allows for pre- or post-encryption fragmentation, if configured.

 

Monitoring and Verifying MTU Behavior

Cisco IOS provides several commands for visibility:

• show crypto ipsec sa  – Displays the current Path MTU as seen by IPSec, including encapsulation overhead.
• show interfaces – Shows the Path MTU if the IPSec SA is bound to a Virtual Tunnel Interface (VTI).
• show crypto ipsec sa | include mtu – Filters MTU-specific information from the IPSec SA output.
• show ipv6 mtu – Reveals the MTU used for IPv6 endpoints, which is critical for IPSec-over-IPv6 tunnel scenarios.

Managing fragmentation in IPSec tunnels requires awareness of both the tunnel interface’s MTU and the configured behavior of the IPSec layer. Ensuring that the DF bit is appropriately set, and that PMTUD can function properly (i.e., ICMP messages are allowed), helps avoid dropped packets and silent application failures. Tools like MSS clamping can also assist in maintaining clean end-to-end transmission when fragmentation isn’t viable.

 

Fragmentation on Tunnels: IPSec, GRE, and PMTUD in Practice

When tunneling protocols like IPSec or GRE are used, fragmentation becomes a critical factor, especially when encapsulation overhead reduces the effective path MTU. This section breaks down fragmentation behavior across different tunnel types, including VTI (Virtual Tunnel Interface), GRE, and GRE over IPSec, along with the role of Path MTU Discovery (PMTUD).

 

IPSec-Only Tunnels (VTI)

Native IPSec tunnel interfaces (sVTI, dVTI) directly encapsulate IP traffic into IPSec. By default, the IP MTU of the tunnel matches the MTU used by the IPSec SA. However, this value can be overridden using ip mtu or ipv6 mtu. Still, these commands only apply to IP traffic, not other protocols, since VTI only supports IP.

• The physical output interface ultimately determines the real MTU used, since the VTI references this interface when calculating overhead.
• You can see which physical interface the VTI maps to with show cef interface.
• Fragmentation for VTI always occurs before encryption, regardless of the crypto ipsec fragmentation setting. This is because the forwarding plane performs MTU checks prior to crypto processing.

The only scenario where a VTI might attempt post-encryption fragmentation is if the configured tunnel MTU exceeds the IPSec SA Path MTU (due to misconfiguration or incorrect PMTUD behavior).

 

GRE-Only Tunnels

GRE tunnel interfaces calculate their IP MTU as the output interface MTU minus GRE overhead, which ranges from 24 to 56 bytes depending on whether optional fields (like keys or sequence numbers) are used. The GRE tunnel is aware of the physical interface MTU, even if sourced from a loopback.

• GRE supports multiple protocols (IP and non-IP), and adjusts its MTU accordingly.
• View tunnel MTU with:
  show interface tunnel0 | include MTU
  show ip interface tunnel0 | include MTU
• You can adjust the tunnel IP MTU using ip mtu or ipv6 mtu on the interface.
• GRE over IPv4 by default clears the DF bit, which can result in unexpected fragmentation—even if the original packet had DF set.

 

GRE over IPSec

When IPSec is added on top of a GRE tunnel using the tunnel protection command, the GRE packet is created first and then encrypted. The MTU of the GRE tunnel must account for IPSec encapsulation overhead, making it essential to subtract both GRE and IPSec overhead from the physical MTU.

• This adjusted MTU can be set on the GRE interface via ip mtu or ipv6 mtu.
• TCP MSS clamping is typically used alongside this to avoid fragmentation entirely.

 

Tunnel PMTUD

Tunnel-based PMTUD allows GRE or VTI interfaces to dynamically adjust their MTU based on ICMP “Fragmentation Needed” messages. To function correctly:

• ICMP messages (type 3/4 for IPv4, type 2/0 for IPv6) must be received on the source interface of the tunnel.
• GRE-over-IPSec tunnels do not copy the DF bit by default from the inner packet to the outer header. This breaks PMTUD.
• To fix this, enable PMTUD explicitly:
  interface tunnel 1
  tunnel path-mtu-discovery

This command ensures the DF bit is copied and enables the interface to act on received ICMP CF messages.

 

You can monitor PMTUD effectiveness with:

show interface tunnel1 | include Path MTU

 

Configure PMTUD behavior using:

tunnel path-mtu-discovery age-time {10-30 | infinite}
tunnel path-mtu-discovery min-mtu 92-65535

 

PMTUD Interaction with Manual MTU Settings

When both ip mtu and tunnel path-mtu-discovery are configured:

• The lower of the two values takes precedence.
• If the MTU discovered via PMTUD is lower than the statically configured MTU, the PMTUD-derived value is used. Otherwise, the configured MTU is enforced.

For VTI, PMTUD works out-of-the-box, provided the inner traffic marks DF. If DF suppression is required, use the crypto ipsec df-bit clear command either globally or per VTI interface.

 

Tunnel interfaces introduce significant complexity to fragmentation and MTU management. Key takeaways:

• VTI always fragments before encryption, using the physical interface MTU.
• GRE requires manual adjustment of MTU, especially when layered with IPSec.
• PMTUD is powerful but fragile—ICMP must be allowed, DF must be set, and tunnel behavior must align.
• Combine with MSS clamping to minimize risk in TCP traffic.

 

The Hidden Costs of IP Fragmentation in Modern Networks

While IP fragmentation allows oversized datagrams to traverse networks with smaller MTUs, it introduces significant performance and reliability challenges—particularly in environments relying on high-speed forwarding, security filtering, and advanced routing.

When a device needs to fragment an IP datagram, it incurs additional CPU and memory overhead. Fragmentation isn’t just a computational task—it also involves generating multiple new packet headers, updating checksums, and managing fragmentation metadata. This overhead grows substantially at the receiving end, where fragments must be reassembled into a complete datagram. The reassembly process forces the receiver to allocate memory buffers for each fragment, holding them until all pieces arrive, and only then does it reconstruct the original payload. This is especially taxing when multiple fragmented packets are received simultaneously or out of order.

This inefficiency becomes particularly problematic on routers, whose primary role is to forward packets quickly. Reassembly responsibilities can severely degrade a router’s performance, and for this reason, most routers are not optimized to reassemble fragmented traffic. Instead, reassembly is offloaded to the destination host, which increases complexity and resource usage there as well.

Beyond performance, fragmentation poses serious challenges for network devices such as NAT gateways and load balancers. These devices often rely on the first packet in a flow to apply policy or session-based logic. If IP fragments arrive out of order, the device may drop non-initial fragments because they don’t contain the necessary headers to match firewall rules or NAT tables. As a result, the receiving host can’t reconstruct the datagram, leading to broken connections or corrupted payloads.

Additionally, Access Control Lists (ACLs) often do not inspect or reconstruct fragments. They may either blindly allow or block all fragmented packets, leaving critical gaps in security posture or application behavior.

In short, while fragmentation is a functional part of IP, it creates performance bottlenecks, security gaps, and application-breaking behavior in many modern network designs. For this reason, techniques like Path MTU Discovery (PMTUD) and TCP MSS Clamping are heavily favored to prevent fragmentation altogether.