Protocol Atlas

OVERLAY · overlay · draft

SD-WAN

Software-Defined WAN overlay architecture

A controller plane distributes overlay policy and keys; site routers still need underlay IP reachability — not a single IETF SD-WAN protocol.

Presenter modeEmbed this figure

Security AssociationsRFC 4301 · PROPOSED STANDARD · December 2005IP Traffic ProcessingRFC 4301 · PROPOSED STANDARD · December 2005Structure of a GRE Encapsulated PacketRFC 2784 · PROPOSED STANDARD · March 2000

Why it exists

Enterprises want centralized policy, easier multi-circuit WAN, and encrypted site mesh without hand-building every tunnel ACL on every router. Cisco SD-WAN — Design Guide / Architecture §System Overview

There is no single IETF “SD-WAN” protocol. Products compose underlay IP, tunnels (often IPsec per RFC 4301, sometimes GRE per RFC 2784), and proprietary controllers. IntroductionRFC 4301 · PROPOSED STANDARD · December 2005

Edge joins the control plane

A site router authenticates to the vendor controller fabric, receives identity and policy, and only then forms authorized overlay tunnels to other sites.

Edge · siteController fabricAuthenticate edgeRemote edge
  • Link
  • Blocking
  • Packet in flight
  • Discarded
  • Emphasis
Select a device to read its state. Arrow keys walk the topology.
Text equivalent of this diagram
Devices and links at this step
ElementKindState
Edge · siterouter—
Controller fabriccloud—
Remote edgerouter—
Edge · site — Controller fabriclinkup
Remote edge — Controller fabriclinkup
Edge · site — Remote edgelinkdown
1 / 3

The edge presents certificates or credentials to the controller fabric. Without a successful control join, overlay peers are not authorized — even if underlay IP works.

What changed

  • Control join: Edge · site → Controller fabric
  • Authenticate edge

How it works

Edges authenticate to a vendor control/management fabric, receive identity and policy, then form authorized overlay tunnels to other sites. Cisco SD-WAN — Design Guide / Architecture §System Overview

The underlay is ordinary IP reachability to controllers and remote outer addresses. Overlay traffic is encapsulated and typically encrypted. IP Traffic ProcessingRFC 4301 · PROPOSED STANDARD · December 2005

GRE (RFC 2784) appears in some designs as an encapsulation building block; IPsec provides confidentiality and integrity for overlay payloads in many fabrics. IntroductionRFC 2784 · PROPOSED STANDARD · March 2000

On the wire

Constructed examples, encoded from the field table below them — not captured traffic.

IHL 5, so no options. DF is set because the sender is doing path MTU discovery — which is why a filtered ICMP turns this into a black hole rather than a retry at a smaller size.

Link layer
Ethernet with EtherType 0x0800, or whatever the medium uses to say "IPv4 follows". RFC 791
IPv4 header
Version, length, differentiated services, fragmentation control, TTL, protocol, addresses. RFC 791
Payload
Whatever the protocol field names — 1 for ICMP, 6 for TCP, 17 for UDP, 89 for OSPF, 103 for PIM. RFC 791

Configure it

Minimal system identity toward controllers so the edge can join the control plane.

Cisco IOS-XE (Catalyst SD-WAN) 17.12 · Catalyst 8000, ISR 1000draft

  1. system
 host-name EDGE-SITE1
 system-ip 10.255.0.11
 site-id 100
 organization-name <configured out of band>
 vbond <vbond.example.invalid>

    Site identity and controller finder (vBond) begin onboarding. Real org names and validator addresses stay out of band — placeholders only.

    Common mistake: Duplicating system-ip across two edges — control join fails or flaps unpredictably.

    Cisco SD-WAN — System Overview §System Overview

  2. vpn 0
 interface GigabitEthernet0/0/0
  ip address 192.0.2.11/30
  tunnel-interface
   encapsulation ipsec
   color biz-internet
   allow-service all
  no shutdown

    Underlay WAN in VPN 0 with a tunnel-interface color. IPsec encapsulation matches the common overlay building block (RFC 4301 architecture), while control still depends on controller reachability.

    Common mistake: Debugging overlay apps before confirming underlay reachability to vBond/vManage/vSmart.

    RFC 4301 §4

Verify

show sdwan control connections
Control connections up to controller components after successful join.
show sdwan control local-properties
system-ip, site-id, and organization match intended identity.

Caveats

  • Marked draft: Catalyst SD-WAN CLI and feature names change across releases; not copy-paste production.
  • Organization names, certificates, and validator addresses are placeholders only.
  • SD-WAN is not one IETF protocol — IPsec/GRE RFCs are building blocks only.

When it breaks

Symptom first, because that is what you have when it happens.

  1. Edge never forms overlay peers despite underlay routes.

    Narrow it down

    1. Check control connection state and certificates.
    2. Verify organization/chassis identity and system IP uniqueness (vendor-specific).

    Cause

    Control join or authorization failed; data-plane peers were never permitted.

    Fix

    Repair controller reachability, identity, and time; re-onboard per vendor procedure.

    Cisco SD-WAN — Design Guide / Architecture §System Overview

  2. Operators declare SD-WAN down when only the ISP path failed.

    Narrow it down

    1. Test underlay reachability to TLOCs/controllers separately from overlay BFD/SA state.
    2. Confirm the related configuration and counters.

    Cause

    Fault-domain confusion between underlay IP and overlay fabric.

    Fix

    Use a split runbook: circuit → underlay → control → overlay.

    Structure of a GRE Encapsulated PacketRFC 2784 · PROPOSED STANDARD · March 2000

  3. Alarms flood during controller loss while sites still pass traffic.

    Narrow it down

    1. Confirm data-plane tunnel state versus control connection state.
    2. Check vendor docs for grace / last-known policy behavior.

    Cause

    Control plane unreachable; data plane still forwarding on cached policy.

    Fix

    Restore controllers; do not assume traffic loss equals control loss (or vice versa).

    Cisco SD-WAN — Design Guide / Architecture §System Overview

  4. After prolonged controller outage, tunnels drop later.

    Narrow it down

    1. Correlate IPsec/SA lifetime and rekey requirements with control availability.
    2. Confirm the related configuration and counters.

    Cause

    Security associations or policy refresh required control services that stayed down.

    Fix

    Restore control before lifetimes expire; size redundancy accordingly.

    Definition and ScopeRFC 4301 · PROPOSED STANDARD · December 2005

  5. One site can reach another only in one direction.

    Narrow it down

    1. Compare application/path policy and tunnel allow-lists on both edges.
    2. Check underlay asymmetry separately.

    Cause

    Central policy or local TLOC preference creates one-way overlay permission.

    Fix

    Align control policy and verify bidirectional tunnel health.

    Cisco SD-WAN — Design Guide / Architecture §System Overview

Design notes

Never treat “ping the remote public IP” as proof of overlay health. Split underlay, control, and data-plane checks.

Controller redundancy and certificate/time hygiene decide whether joins succeed after outages — crypto building blocks still need correct clocks and trust anchors. Security AssociationsRFC 4301 · PROPOSED STANDARD · December 2005

Misconceptions

“SD-WAN is defined by a single IETF RFC.”
SD-WAN is a product architecture. RFCs such as 4301 (IPsec) and 2784 (GRE) are common building blocks, not “the SD-WAN protocol.” IntroductionRFC 4301 · PROPOSED STANDARD · December 2005
“If the underlay pings, the overlay is up.”
Underlay IP reachability is necessary but not sufficient. Control authorization and tunnel/SA state are separate. IP Traffic ProcessingRFC 4301 · PROPOSED STANDARD · December 2005
“Controller down always means all site traffic stops immediately.”
Many fabrics keep established data tunnels for a time. Behavior is vendor-specific; verify documentation instead of assuming instant blackhole. Cisco SD-WAN — Design Guide / Architecture §System Overview

More walkthroughs

Underlay reachability versus overlay servicedesign-choice

The underlay is ordinary IP (and often provider circuits). The overlay is the encrypted, policy-shaped fabric. Confusing the two leads to “SD-WAN is down” tickets that are really ISP routing.

Site edgeUnderlay · ISP IPOrdinary routingOverlay · tunnels
  • Link
  • Blocking
  • Packet in flight
  • Discarded
  • Emphasis
Select a device to read its state. Arrow keys walk the topology.
Text equivalent of this diagram
Devices and links at this step
ElementKindState
Site edgerouter—
Underlay · ISP IPzone—
Overlay · tunnelscloud—
Site edge — Underlay · ISP IPlinkup
Underlay · ISP IP — Overlay · tunnelslinkup
1 / 3

Site WAN interfaces need IP reachability to controllers and remote TLOCs. GRE (RFC 2784) or IPsec tunnel outer headers still ride that underlay.

What changed

  • Underlay IP: Site edge → Underlay · ISP IP
  • Ordinary routing

Controller outage versus data-plane survivalfailure

Vendor fabrics usually keep existing overlay forwarding for a time when controllers are unreachable, but new joins, policy changes, and key/policy refresh stop. Know what your vendor guarantees.

Edge AEdge BControllers · downControl unreachableData may continue
  • Link
  • Blocking
  • Packet in flight
  • Discarded
  • Emphasis
Select a device to read its state. Arrow keys walk the topology.
Text equivalent of this diagram
Devices and links at this step
ElementKindState
Edge Arouter—
Edge Brouter—
Controllers · downcloud—
Edge A — Edge Blinkup
Edge A — Controllers · downlinkdown
Edge B — Controllers · downlinkdown
1 / 3

Control connections fail. Operators see alarms. Existing IPsec/overlay sessions may continue forwarding with last-known policy — vendor-specific grace behavior.

What changed

  • Control unreachable
  • Emphasis: Data may continue

Check yourself

  • Is there a single IETF RFC that defines “SD-WAN” as a protocol?
  • What is the underlay in an SD-WAN design?
  • What must typically succeed before authorized overlay tunnels form?
  • During a controller outage, what often still works for a time?
  • Which RFC is commonly cited for IPsec traffic processing used in overlays?
  • Where does site-to-site permission usually come from in SD-WAN?
  • Why can tunnels fail hours after controllers go down?
  • How does SD-WAN teaching differ from classic DMVPN teaching?