VXLAN — Virtual eXtensible Local Area Network

Why it exists

A VLAN tag has twelve bits, so a data centre runs out of segments at four thousand. Worse, extending a segment between racks means extending a broadcast domain, and with it spanning tree — a protocol whose failure mode is the whole fabric.

VXLAN puts the Ethernet frame inside a UDP datagram and gives the segment a 24-bit identifier. The network between the two switches becomes an ordinary routed IP fabric: no spanning tree, no tenant state, and ECMP across every link.

It solves a third problem quietly. A leaf switch only needs MAC entries for the hosts in the segments it actually carries, rather than for every host in the data centre — which is what RFC 7348 calls inadequate table sizes at the top of rack.

A frame in a UDP datagram

The ingress switch wraps the tenant frame and sends it as ordinary IP. Everything between the two switches forwards it without knowing what it carries.

Link
Blocking
Packet in flight
Discarded
Emphasis

Select a device to read its state. Arrow keys walk the topology.

Text equivalent of this diagram

Devices and links at this step
Element	Kind	State
Host A · VLAN 10	host	—
Leaf 1 · VTEP 10.0.0.1	switch	VLAN 10 →: VNI 10010
Spine	router	—
Leaf 2 · VTEP 10.0.0.2	switch	—
Host B · VLAN 20	host	—
Host A · VLAN 10 — Leaf 1 · VTEP 10.0.0.1	link	up
Leaf 1 · VTEP 10.0.0.1 — Spine	link	up
Spine — Leaf 2 · VTEP 10.0.0.2	link	up
Leaf 2 · VTEP 10.0.0.2 — Host B · VLAN 20	link	up

1 / 6

Host A sends an ordinary Ethernet frame. Leaf 1 classifies it into VLAN 10 on the access port and maps that VLAN to VNI 10010.

The local VLAN number is local. Leaf 2 carries the same segment as VLAN 20, and that is not a mistake — the VNI is what makes the two the same segment.

What changed

Leaf 1 · VTEP 10.0.0.1: VLAN 10 → → VNI 10010
Emphasis on Host A · VLAN 10 ↔ Leaf 1 · VTEP 10.0.0.1

How it works

The ingress switch — the VTEP — classifies a frame into a local VLAN, maps that to a VNI, and wraps it in outer Ethernet, IP, UDP to port 4789 and an eight-octet VXLAN header. Fifty octets over IPv4, seventy over IPv6.

The outer UDP source port carries no meaning: it is a hash of the inner headers, present so the fabric has entropy to spread flows across equal-cost paths. Without it every packet between a pair of leaves would hash identically and take one link.

In a flood-and-learn fabric the egress VTEP learns by association: the inner source MAC is bound to the outer source IP. That is the entire remote MAC table, and it is why a host that never transmits cannot be reached.

Broadcast has to be arranged, because a routed underlay has none. Either each VNI maps to an underlay multicast group, or the ingress leaf replicates a unicast copy per peer — and EVPN both builds that peer list and removes most of the need for it.

On the wire

Constructed examples, encoded from the field table below them — not captured traffic.

UDP: Destination port 4789. The outer IP addresses are the VTEPs, not the tenants. RFC 7348
VXLAN: VNI identifies the overlay L2 segment. RFC 7348

Configure it

```
feature nv overlay
feature vn-segment-vlan-based
feature bgp
nv overlay evpn
```
Four separate switches for one capability. Missing `nv overlay evpn` in particular leaves a fabric that encapsulates correctly and never learns anything from BGP.
```
vlan 10
 vn-segment 10010
```
The local VLAN to VNI mapping. The VLAN number is local to this leaf; the VNI is what makes the segment the same one across the fabric.
Common mistake: Assuming the VLAN number must match on every leaf. It need not — but a VNI that does not match is a different segment, and the local configuration looks fine.
RFC 8365 §5.1.2
```
interface nve1
 no shutdown
 source-interface loopback1
 host-reachability protocol bgp
 member vni 10010
  ingress-replication protocol bgp
```
The tunnel source is a loopback, and that loopback address is what the underlay has to reach. `ingress-replication protocol bgp` builds the flood list from EVPN type 3 routes instead of a hand-written list.
RFC 7348 §4.1
```
interface loopback1
 ip address 10.0.0.1/32
 ip router ospf UNDERLAY area 0
```
The line whose absence produces an overlay that cannot form while every interface is up. Monitor reachability between these loopbacks, not between the switches.
```
system jumbomtu 9216
interface Ethernet1/1
 mtu 9216
```
Fifty octets of wrapper on top of the tenant frame. A fabric left at 1500 works for everything small and stalls every large transfer.
RFC 7348 §4.3
```
ip access-list VXLAN-EDGE
 deny udp any any eq 4789
 permit ip any any
interface Ethernet1/48
 ip access-group VXLAN-EDGE in
```
VXLAN has no authentication. Anything that can reach a VTEP on 4789 can name any VNI, so the fabric boundary is the security boundary.
RFC 7348 §7

Verify

show nve peers: Which remote VTEPs this leaf has, and whether they are up.
show nve vni: VNI state and the replication mode in use.
show l2route evpn mac all: Remote MACs and the VTEP each was learned from.
ping 10.0.0.2 source 10.0.0.1: Underlay reachability between the addresses the tunnel uses.

Caveats

The tunnel is built between loopbacks. Every interface can be up while the overlay is entirely down.
Raise the MTU on spine-to-spine links too — they are routinely missed because no host is attached.
There is no authentication in VXLAN; filter UDP 4789 at every fabric boundary.

When it breaks

Symptom first, because that is what you have when it happens.

Two hosts on the same VNI on different leaves cannot reach each other, while hosts on the same leaf are fine.
Narrow it down
1. Confirm both leaves have the VNI configured and the VTEP interface is up.
2. Check underlay reachability between the VTEP source addresses, not just between the leaves.
3. Verify the MAC has been learned remotely — on the far VTEP, not locally.
Cause
The overlay depends entirely on the underlay reaching the VTEP source address. A loopback that is not advertised, or a missing underlay route, breaks the tunnel while every interface stays up.
Fix
Advertise the VTEP loopback in the underlay and verify reachability between loopbacks, which is the address the tunnel actually uses.
Traffic works up to a size and then stops, and only across the fabric.
Narrow it down
1. Find the largest packet that crosses two leaves.
2. Add up the encapsulation: 50 octets over IPv4, 70 over IPv6.
3. Compare against the underlay MTU on every link, including spine to spine.
Cause
Encapsulation adds 50 octets and the underlay was left at 1500. A tenant frame at the standard MTU no longer fits once wrapped, and the VTEP sets Don’t Fragment so it is discarded rather than split.
Fix
Raise the underlay MTU to at least 1550, or 9216 as most fabrics do. The tenant MTU is not the number that matters.
Broadcast and unknown unicast reach some leaves and not others, so ARP resolves intermittently.
Narrow it down
1. Establish whether the fabric floods with multicast or ingress replication.
2. For multicast, check the underlay group state on every VTEP.
3. For ingress replication, check the remote VTEP list on the leaf that is failing.
Cause
Flooding has to be arranged explicitly. Either the underlay multicast group is not built everywhere, or the replication list is missing a VTEP — usually one added after the fabric was built.
Fix
Let EVPN distribute the replication list rather than maintaining it by hand, which removes the class of fault.
One spine link saturates while the others are idle, and the traffic is all overlay.
Narrow it down
1. Check whether the outer UDP source port varies between packets of different inner flows.
2. Look at what the spine hashes on — some platforms hash only the outer five-tuple.
3. Compare per-link utilisation between leaf pairs rather than in aggregate.
Cause
Every packet between two leaves shares outer source, destination and destination port. If the ingress VTEP does not vary the source port, or the spine ignores it, the fabric sees one flow.
Fix
Confirm the VTEP derives the source port from the inner headers, and that the underlay hash includes the UDP source port.
A host reaches the wrong segment, or two tenants that should be separate can see each other.
Narrow it down
1. Compare the VLAN-to-VNI mapping on every leaf carrying the segment.
2. Check for a VNI reused between tenants on different leaves.
3. Look at which VNI the traffic actually arrives with, rather than what was intended.
Cause
The local VLAN number is local and the VNI is global. A leaf that maps its VLAN 20 to the wrong VNI joins the wrong segment, and nothing about the configuration looks wrong locally.
Fix
Treat VNI allocation as a global registry and audit the mapping per leaf. A local VLAN that matches everywhere is a convention, not a safeguard.

Design notes

Raise the underlay MTU before anything else. Fifty octets on a 1500-byte fabric produces a network where everything small works and every large transfer stalls, and the ICMP that would explain it goes to the VTEP rather than the host.

Advertise the VTEP loopback in the underlay and monitor reachability between loopbacks, not between leaves. The tunnel is built between those addresses, and every interface can be up while the overlay is entirely down.

Prefer ingress replication with EVPN over underlay multicast. Multicast is the more efficient design and the more expensive one to operate — the failure mode is intermittent ARP, and the diagnosis is a multicast problem in a fabric nobody thinks of as running multicast.

VXLAN has no authentication and no encryption. RFC 7348 is explicit that it assumes a trusted underlay: anything that can inject a UDP datagram to port 4789 at a VTEP can inject a frame into any segment. Filter port 4789 at every fabric boundary.

Misconceptions

“VXLAN replaces VLANs.”: It carries them. A frame is still classified into a VLAN on the access port; the VNI identifies the segment across the fabric. Leaves map local VLAN to VNI, and the two numbers are frequently different on different leaves.
“The 24-bit VNI means sixteen million usable segments.”: The identifier space is that large. What runs out first is hardware — the MAC and ARP table sizes on a leaf, and the number of multicast groups or replication entries the fabric can carry.
“VXLAN needs multicast in the underlay.”: The original flood-and-learn design used it. Ingress replication removes the requirement and EVPN removes most of the flooding, so a modern fabric usually runs neither.
“VXLAN isolates tenants, so it is a security boundary.”: It has no authentication and no encryption at all. RFC 7348 assumes a trusted underlay — anything that can send a UDP datagram to port 4789 at a VTEP can put a frame into any segment it names. The isolation is administrative, not cryptographic.

More walkthroughs

Three ways to flood across a fabric that cannot flooddesign-choice

An Ethernet segment needs broadcast. A routed underlay has no broadcast, so the fabric has to arrange it — with multicast, with replication, or by not needing it.

Link
Blocking
Packet in flight
Discarded
Emphasis

Select a device to read its state. Arrow keys walk the topology.

Text equivalent of this diagram

Devices and links at this step
Element	Kind	State
Leaf 1 · ingress	switch	Needs: flood to every VNI 10010 peer
Leaf 2	switch	—
Leaf 3	switch	—
Leaf 4 · no VNI 10010	switch	—
EVPN route reflector	router	—
Leaf 1 · ingress — Leaf 2	link	up
Leaf 1 · ingress — Leaf 3	link	up
Leaf 1 · ingress — Leaf 4 · no VNI 10010	link	standby
Leaf 1 · ingress — EVPN route reflector	link	up

1 / 5

Host A sends an ARP request. On a real Ethernet segment it reaches everyone; on a routed fabric there is nobody to send it to.

What changed

Leaf 1 · ingress: Needs → flood to every VNI 10010 peer
Emphasis on Leaf 1 · ingress

Fifty octets that nobody budgeted forfailure

The fabric works for everything small. Large transfers stall, and the tenant MTU is not the number that explains it.

Link
Blocking
Packet in flight
Discarded
Emphasis

Select a device to read its state. Arrow keys walk the topology.

Text equivalent of this diagram

Devices and links at this step
Element	Kind	State
Host A · MTU 1500	host	Fails at: first 1500-byte packet
Leaf 1	switch	—
Spine · MTU 1500	router	—
Leaf 2	switch	—
Host A · MTU 1500 — Leaf 1	link	up
Leaf 1 — Spine · MTU 1500	link	up
Spine · MTU 1500 — Leaf 2	link	up

1 / 5

Ping works. SSH works. A file copy between the same two hosts stalls at the first full-size segment and never recovers.

The signature is that the failure correlates with size rather than with destination, which rules out routing and points at something on the path adding bytes.

What changed

Emphasis: Small fine, large gone
Host A · MTU 1500: Fails at → first 1500-byte packet

Terms

VNI: 24-bit VXLAN Network Identifier in the eight-octet VXLAN header. It is the overlay segment, not a VLAN until a VTEP maps it to one.
VTEP: The switch or host that adds and removes the VXLAN wrapper. The tunnel is built between VTEP source addresses, which is why underlay reachability has to be checked between loopbacks rather than between switches.
Ingress replication: Flooding by sending one unicast copy per remote peer instead of using underlay multicast. The list is either maintained by hand or distributed by EVPN, and a peer missing from it fails only for hosts nobody has learned yet.

Check yourself

Hosts on the same VNI cannot reach each other across leaves, but every interface is up. Where do you look first?
How much does VXLAN over IPv4 add to a frame?
What does the VNI identify?
What replaced multicast for handling broadcast in modern fabrics?
What is the outer UDP source port used for?
In a flood-and-learn fabric, how does a leaf learn where a remote MAC lives?
What protects a VXLAN segment from a forged frame injected at the underlay?
Leaf 1 carries the segment as VLAN 10 and Leaf 2 as VLAN 20. Is that broken?

Why it exists

A frame in a UDP datagram

How it works

On the wire

Configure it

Verify

Caveats

When it breaks

Two hosts on the same VNI on different leaves cannot reach each other, while hosts on the same leaf are fine.

Narrow it down

Cause

Fix

Traffic works up to a size and then stops, and only across the fabric.

Narrow it down

Cause

Fix

Broadcast and unknown unicast reach some leaves and not others, so ARP resolves intermittently.

Narrow it down

Cause

Fix

One spine link saturates while the others are idle, and the traffic is all overlay.

Narrow it down

Cause

Fix

A host reaches the wrong segment, or two tenants that should be separate can see each other.

Narrow it down

Cause

Fix

Design notes

Misconceptions

More walkthroughs

Three ways to flood across a fabric that cannot flooddesign-choice

Fifty octets that nobody budgeted forfailure

Terms

Check yourself

Related

EVPN

VLANs

Ethernet

Asymmetric versus symmetric IRB