IPsec

Why it exists

IP forwards packets without confidentiality or integrity. IPsec is a framework — ESP or AH for packet protection, IKE for keying, and databases that decide what must be protected — so two peers can agree policy without trusting the path between them. RFC 4301 §3.2How IPsec WorksRFC 4301 · PROPOSED STANDARD · December 2005

Most operational pain is not “encryption failed” but a mismatch among those pieces: selectors, mode, lifetimes, or proposal algorithms. RFC 4301 §4.4.1The Security Policy Database (SPD)RFC 4301 · PROPOSED STANDARD · December 2005

IKE builds the Child SA, ESP carries the traffic

IKEv2 negotiates an IKE SA then a Child SA. ESP uses the Child SA’s SPI to protect packets that match the traffic selectors.

How it works

The Security Policy Database (SPD) says what to do with a packet: discard, bypass, or protect. The Security Association Database (SAD) holds the concrete SAs — SPI, algorithms, keys, mode, sequence state. RFC 4301 §4.4.1The Security Policy Database (SPD)RFC 4301 · PROPOSED STANDARD · December 2005

ESP provides confidentiality and/or integrity. The SPI and sequence number travel in the clear; the payload and trailer are (when encryption is used) ciphertext, followed by an ICV. RFC 4303 §2Encapsulating Security Payload Packet FormatRFC 4303 · PROPOSED STANDARD · December 2005

AH authenticates without encrypting and covers immutable parts of the outer IP header — which is why NAT breaks AH while ESP with NAT traversal does not. RFC 4302 §1IntroductionRFC 4302 · PROPOSED STANDARD · December 2005

IKEv2 builds an IKE SA, then Child SAs for ESP/AH. Rekey creates a new Child SA before the old one expires when both ends agree on lifetimes and rekey behaviour. RFC 7296 §2.8RekeyingRFC 7296 · INTERNET STANDARD · October 2014

On the wire

Constructed examples, encoded from the field table below them — not captured traffic.

Configure it

When it breaks

Symptom first, because that is what you have when it happens.

1. Phase 1 completes and phase 2 never does, or the tunnel comes up and passes no traffic.

   Narrow it down

   1. Compare the traffic selectors — the interesting-traffic definition — on both ends.
   2. Check whether one side is policy-based and the other route-based.
   3. Read the phase 2 proposal on each end and find the first field that differs.

   Cause

   The selectors do not match. Policy-based IPsec negotiates a security association per selector pair and both ends must agree exactly; route-based negotiates one for any traffic and relies on routing to decide what enters. Mixing them produces a tunnel that establishes and carries nothing.

   Fix

   Make both ends the same style. If one is route-based, configure the other with a selector of any-to-any and let routing choose.

   RFC 4301 §4.4.1The Security Policy Database (SPD)RFC 4301 · PROPOSED STANDARD · December 2005

2. The tunnel works for interactive traffic and fails for file transfers or anything with large packets.

   Narrow it down

   1. Find the largest packet that crosses the tunnel with the do-not-fragment bit set.
   2. Add up the overhead: ESP header, IV, padding, trailer, and the outer IP header.
   3. Check whether ICMP is filtered between the endpoints.

   Cause

   Encapsulation overhead pushes a full-size packet over the path MTU. The tunnel endpoint should signal that, but if ICMP is filtered the sender never learns and simply retransmits.

   Fix

   Clamp the TCP MSS on the tunnel interface. It is the fix that does not depend on ICMP surviving a path you do not control.

3. A tunnel drops on a schedule — every eight hours, or every hour — and re-establishes on its own.

   Narrow it down

   1. Compare the security association lifetimes configured at each end.
   2. Check whether rekeying is completing before the old association expires.
   3. Look for one end rekeying by volume and the other by time.

   Cause

   Mismatched lifetimes. The shorter one expires first, and if rekeying does not complete cleanly the tunnel drops rather than rolling over.

   Fix

   Match the lifetimes, and ensure both ends support and use the same rekey behaviour. IKEv2 handles this considerably better than IKEv1.

   RFC 7296 §2.8RekeyingRFC 7296 · INTERNET STANDARD · October 2014

4. IKE_SA_INIT or Child SA negotiation fails with NO_PROPOSAL_CHOSEN or similar.

   Narrow it down

   1. Compare encryption, integrity, PRF, and Diffie-Hellman groups on both ends.
   2. Check whether one end still offers only IKEv1-era groups the other refuses.
   3. Confirm ESP proposals separately from IKE proposals — they are different negotiations.

   Cause

   The peers have no overlapping cryptographic proposal. Authentication never starts if IKE proposals fail; Child SA failure leaves an IKE SA with no usable data SA.

   Fix

   Publish an overlapping proposal set deliberately. Prefer modern suites both platforms support rather than copying an old checklist from one vendor’s defaults.

   RFC 7296 §2.7Cryptographic Algorithm NegotiationRFC 7296 · INTERNET STANDARD · October 2014

5. Outbound encaps counters increment; the peer shows inbound discards or no decrypt.

   Narrow it down

   1. Compare the SPI each side installed for the Child SA.
   2. Confirm the outer destination address matches the SA endpoint.
   3. Look for an old SA still selected after rekey.

   Cause

   The SPI (with destination and protocol) selects the SA. A stale SA, asymmetric install, or traffic hitting the wrong peer address produces ciphertext the receiver cannot map.

   Fix

   Clear and renegotiate Child SAs after config changes; verify both SAD entries list matching SPI pairs and endpoints.

   RFC 4303 §2.1Security Parameters Index (SPI)RFC 4303 · PROPOSED STANDARD · December 2005

Design notes

Tunnel versus transport is topology, not strength. Gateways use tunnel mode; host-to-host often uses transport. RFC 4303 §3.1ESP Header LocationRFC 4303 · PROPOSED STANDARD · December 2005

A Child SA that is “up” with idle encaps counters usually means selectors or routing disagree about interesting traffic — not that ESP is broken.

WireGuard solves a narrower problem with static keys and allowed-IPs. Mixing IPsec SPD vocabulary with WireGuard peer lists is how configs get copied wrong.

Misconceptions

“IPsec is a protocol.”

It is a framework of several: ESP or AH for protecting packets, IKE for negotiating the keys, and a policy database deciding what gets protected. Most confusion comes from a mismatch in one of those three being described as "IPsec is broken". RFC 4301 §3.2How IPsec WorksRFC 4301 · PROPOSED STANDARD · December 2005

“AH and ESP both encrypt.”

AH authenticates and does not encrypt at all. ESP can do both, and in practice ESP is used with authentication enabled — which is why AH is nearly extinct, and why it breaks through NAT while ESP with NAT traversal does not. RFC 4302 §1IntroductionRFC 4302 · PROPOSED STANDARD · December 2005

“Tunnel mode and transport mode are a security choice.”

They are a topology choice. Tunnel mode wraps the whole original packet in a new IP header, which is what lets a gateway protect traffic on behalf of hosts behind it. Transport mode protects the payload of a packet the endpoints send themselves. Both use the same ciphers.

More walkthroughs

Terms

SPI

Security Parameters Index: a 32-bit value in the ESP or AH header that, with the destination address and security protocol, selects which Security Association will process the packet.

Traffic selector

The set of addresses, protocols, and ports that a Child SA is allowed to protect. Mismatched selectors are the usual reason an IPsec tunnel is up but carries no traffic.

Tunnel mode

IPsec mode that encrypts (and/or authenticates) an entire inner IP packet and adds a new outer IP header — what gateways use to protect traffic for hosts that are not themselves IPsec peers.

Check yourself

• A tunnel establishes and carries no traffic. What is the most likely cause?

  • The pre-shared key is wrong
  • The traffic selectors do not match, often because one end is policy-based and the other route-based
  • The wrong Diffie-Hellman group
  • NAT traversal is disabled
• Why does clamping the MSS fix large-packet failures better than relying on path MTU discovery?

  • It is faster
  • It does not depend on ICMP surviving a path you do not control
  • It increases the MTU
  • It disables fragmentation
• What does AH provide that ESP does not?

  • Encryption
  • Nothing ESP cannot — AH authenticates without encrypting, which is why it is little used
  • NAT traversal
  • Perfect forward secrecy
• A tunnel drops every eight hours and comes straight back. What should you check?

  • The certificate expiry
  • The security association lifetimes at each end, and whether they match
  • The routing protocol timers
  • The MTU
• What does the Security Policy Database decide?

  • Only which Diffie-Hellman group to use
  • Whether a packet is discarded, bypassed, or protected — and with what selectors
  • The TCP window size
  • DNS resolution for the peer
• When do you need tunnel mode rather than transport mode?

  • Whenever you want stronger ciphers
  • When a gateway must protect traffic for hosts that are not themselves IPsec peers
  • Only on IPv6
  • Never — transport always wraps a new outer header
• What is the SPI for in an ESP header?

  • Encrypting the outer IP addresses
  • Selecting, with destination and protocol, which SA processes the packet
  • Carrying the Diffie-Hellman public value
  • Marking the packet as AH instead of ESP
• What is the relationship between IKEv2 and ESP?

  • ESP negotiates keys; IKE only authenticates users
  • IKEv2 negotiates SAs and keys; ESP uses those SAs to protect packets
  • They are the same packet format
  • IKE replaces ESP in tunnel mode

Related

• WireGuard

  A small cryptographic VPN with static public keys, cryptokey routing, and one UDP-carried tunnel interface per peer list.

• DMVPN

  mGRE plus NHRP plus (usually) IPsec: spokes register with a hub, and Phase 3 builds on-demand spoke-to-spoke shortcuts from NHRP redirect and shortcut switching.