Why post-quantum TLS is making “packet size” a security and availability requirement especially over IPsec tunnels and PPPoE
Deep SSL/TLS inspection is one of the most complex things a firewall can do: it has to reassemble TCP streams, parse TLS handshakes, generate substitute certificates, and re-encrypt traffic often at high scale.
In 2026, a previously “boring” network setting is becoming a top cause of SSL inspection failures:
MTU (and its TCP counterpart, MSS).
The reason is simple: TLS handshakes are getting bigger as modern browsers and major platforms roll out post-quantum (PQ) / hybrid key exchange (e.g., ML-KEM). Bigger handshakes + reduced MTU links (PPPoE, IPsec) + inline inspection = more fragmentation, higher tail risk, and browser errors that look random.
This article explains what’s changing, why it matters, and how to engineer Deep SSL inspection to survive real-world packet paths.
What changed: PQ TLS makes the handshake larger
Post-quantum hybrid key exchange adds large key shares to TLS 1.3. Meta documented a key operational reality: Kyber768 public key share is 1184 bytes, and when combined with common ClientHello extensions (PSK resumption, supported groups, etc.), the ClientHello can exceed a typical MSS and must be split across multiple TCP segments.
Google also described Chrome’s move toward ML-KEM in TLS 1.3, with new hybrid groups and behavior designed for the web.
Net effect:
More sessions now start with larger, more complex ClientHello messages, which increases stress on any path that relies on clean MTU behavior and correct reassembly through multiple network devices.
Why MTU becomes a Deep Inspection requirement (not just “network tuning”)
Deep SSL inspection is a middlebox function. Middleboxes are most likely to break when they see:
- TCP streams split across multiple segments earlier than expected
- fragmented IP packets due to MTU mismatch
- PMTUD failing because ICMP is blocked
- encapsulation overhead shrinking the effective path MTU (tunnels/PPPoE)
- “tail latency” spikes that cause timeouts during handshake parsing/reassembly
In 2026, larger TLS handshakes make these conditions far more common.
MTU vs MSS: the practical view
- MTU = maximum IP packet size on a link (Ethernet commonly 1500).
- MSS = maximum TCP payload per segment (derived from MTU minus IP/TCP headers).
When the real path MTU is smaller than assumed, the network has only two ways to cope:
- Fragment IP packets, or
- Signal the sender via PMTUD (ICMP “Fragmentation Needed” / “Packet Too Big”) so it sends smaller packets.
If PMTUD signals are blocked or inconsistent, you get the classic failure mode:
Black-hole MTU
Packets that are “too big” disappear silently. TCP retransmits. TLS handshake stalls. The browser throws errors like ERR_SSL_PROTOCOL_ERROR or “connection reset,” depending on how far it got.
Why PPPoE and IPsec tunnels are the biggest culprits
PPPoE (smaller MTU by design)
PPPoE adds overhead, so the common MTU becomes 1492 instead of 1500. That seems small, but it reduces MSS and increases the chance that a large ClientHello spans multiple segments or triggers fragmentation on misconfigured links.
IPsec tunnels (encapsulation reduces effective MTU)
IPsec adds encapsulation headers (ESP + outer IP + optional NAT-T/UDP). The effective MTU can drop significantly depending on:
- IPv4 vs IPv6 outer headers
- NAT-T enabled or not
- ciphers/auth modes
- additional overhead (VLAN tags, provider WAN, SD-WAN overlays, etc.)
In real deployments, “safe” tunnel MTUs often land somewhere in the 1350–1450 range, but the only correct number is the one you measure end-to-end.
This is where Deep SSL inspection gets sensitive:
The TLS handshake is time-critical. If the early packets are fragmented or black-holed, the connection fails before any application retry logic can help.
What it looks like when MTU breaks Deep SSL inspection
Common symptoms:
- Some websites fail, others work (because handshake sizes vary)
- Failures appear “random” or “new” after browser updates (because the ClientHello grew)
- Errors happen more often over:
- PPPoE internet circuits
- remote sites over IPsec
- SD-WAN overlays or carrier networks with inconsistent PMTUD
- Packet captures show:
- repeated retransmissions early in the TLS handshake
- TLS alerts or resets after partial handshake
- stalled sessions that never complete the handshake
Engineering guidance: how to make Deep SSL inspection resilient
1) Treat MTU as part of the security design
If you’re doing inline decryption, you’re building a real-time chain of packet handling. Availability issues caused by MTU are security issues (loss of access, broken authentication, broken updates).
2) Always validate path MTU end-to-end (not just on one interface)
For IPsec and PPPoE paths, validate the real maximum payload that survives the entire route. Document it per underlay, not just “the WAN.”
3) Use TCP MSS clamping where you can’t guarantee PMTUD
MSS clamping forces TCP endpoints to use a smaller segment size, preventing oversized segments that would rely on fragmentation or fragile PMTUD behavior.
This is often the single highest-impact mitigation when you have:
- PPPoE links
- IPsec tunnels
- mixed carriers / last-mile variability
- sites where ICMP is filtered or rate-limited
4) Make your tunnel MTU explicit (don’t let it be “accidental”)
For IPsec, set a consistent MTU/MSS strategy aligned to:
- whether NAT-T is used
- IPv4 vs IPv6 outer transport
- any SD-WAN overlays
- any intermediate provider encapsulation
5) Monitor for “tail events,” not just averages
Handshake failures often come from:
- rare spikes
- microbursts
- reassembly pressure
- queueing delays
So monitoring should include:
- retransmission rate
- TLS handshake failures by policy/path
- packet loss and PMTU/ICMP counters (where available)
A practical checklist for 2026 deployments
If you answer “yes” to 2+ of these, MTU/MSS needs active design:
- Do you decrypt SSL/TLS inline for a broad user population?
- Do you have PPPoE anywhere in the estate?
- Do you route business-critical traffic over IPsec site-to-site tunnels?
- Do you operate SD-WAN or multiple underlays with different MTUs?
- Do you block or tightly restrict ICMP (including “Packet Too Big”)?
- Have you seen new SSL errors after recent browser updates?
Final thought
Post-quantum TLS is not just a cryptography upgrade it changes traffic shape. Larger handshakes mean your Deep SSL inspection chain must be engineered for real path MTU, especially on PPPoE and IPsec where the true maximum packet size is smaller than “standard Ethernet.”
If you want, share:
- whether your remote connectivity is mostly PPPoE or DIA
- your typical IPsec topology (hub/spoke, full mesh)
- IPv4/IPv6 usage
- whether ICMP is permitted across WAN/tunnels
…and I’ll propose a clean, vendor-neutral MTU/MSS strategy and rollout plan that avoids breaking modern browsers.
Sources
Meta engineering on PQ TLS readiness and the operational impact of larger TLS handshakes.
Google Security Blog on the transition path for Kyber/ML-KEM on the web.
