OVERLAY · overlay · reviewed

EVPN

Ethernet VPN

Advertises MAC addresses in BGP instead of discovering them by flooding, which is what makes all-active multihoming and predictable mobility possible.

Presenter modeEmbed this figure

MAC/IP Advertisement RouteRFC 7432 · PROPOSED STANDARD · February 2015Designated Forwarder ElectionRFC 7432 · PROPOSED STANDARD · February 2015MAC MobilityRFC 7432 · PROPOSED STANDARD · February 2015VXLAN/NVGRE EncapsulationRFC 8365 · PROPOSED STANDARD · March 2018Symmetric and Asymmetric IRBRFC 9135 · PROPOSED STANDARD · October 2021The BGP EVPN IP Prefix RouteRFC 9136 · PROPOSED STANDARD · October 2021

Why it exists

Every earlier way of stretching a layer-two segment learned MAC addresses by flooding. That works, and it means a host nobody has heard from is unreachable, a topology change is resolved by ageing timers, and two switches attached to one host cannot both forward.

EVPN moves that information into BGP. A leaf that sees a host advertises it; every other leaf installs it from a routing update. The data plane is unchanged — usually VXLAN, sometimes MPLS — and what changes is where knowledge comes from. MAC/IP Advertisement RouteRFC 7432 · PROPOSED STANDARD · February 2015

The consequences are larger than the mechanism suggests. Advertising a MAC makes all-active multihoming expressible, makes mobility an ordered event rather than a race, and turns the flood list into something the control plane maintains.

A MAC address as a routing update

The leaf that sees a host tells everyone. Remote leaves install the MAC from a BGP route rather than learning it from a flooded frame.

Host A sends its first frame — usually a gratuitous ARP. Leaf 1 learns the MAC on the access port exactly as any switch would. Host A. Leaf 1: Local MAC A on Eth1/1. Route reflector. Leaf 2. Host B.

Host ALeaf 1Local: MAC A on Eth1/1Route reflectorLeaf 2Host B
  • Link
  • Blocking
  • Packet in flight
  • Discarded
  • Emphasis
Select a device to read its state. Arrow keys walk the topology.
Text equivalent of this diagram
Devices and links at this step
ElementKindState
Host Ahost
Leaf 1switchLocal: MAC A on Eth1/1
Route reflectorrouter
Leaf 2switch
Host Bhost
Host ALeaf 1linkup
Leaf 1Route reflectorlinkup
Route reflectorLeaf 2linkup
Leaf 2Host Blinkup
1 / 5

Host A sends its first frame — usually a gratuitous ARP. Leaf 1 learns the MAC on the access port exactly as any switch would.

What changed

  • Leaf 1: Local → MAC A on Eth1/1
  • Emphasis on Host A ↔ Leaf 1

How it works

A type 2 route carries a MAC, optionally the IP it was seen with, and the VNI. The IP is what lets a remote leaf answer ARP locally instead of flooding it, which removes most fabric-wide broadcast. Remote LearningRFC 7432 · PROPOSED STANDARD · February 2015

A type 3 route says a leaf wants broadcast traffic for a VNI. Collected, those routes are the ingress replication list — the same list a flood-and-learn fabric maintains by hand and forgets to update. Constructing Inclusive Multicast Ethernet Tag RouteRFC 7432 · PROPOSED STANDARD · February 2015

Multihoming keys entirely on the Ethernet segment identifier. Type 4 routes discover peers on the segment and elect a designated forwarder; the ESI label lets a peer drop a frame the host originated; type 1 routes give remote leaves both paths and let a link failure be withdrawn once rather than once per MAC. Aliasing and Backup PathRFC 7432 · PROPOSED STANDARD · February 2015

Routing between segments is IRB. Asymmetric routes at ingress and bridges at egress, so every leaf needs every VNI it might route into; symmetric routes at both ends through a transit L3VNI, so a leaf needs only the VNIs it has hosts in. Symmetric and Asymmetric IRBRFC 9135 · PROPOSED STANDARD · October 2021

On the wire

Constructed examples, encoded from the field table below them — not captured traffic.

AFI IPv6, SAFI unicast, 16-octet next hop, one /32 prefix. The IPv4 NLRI field of UPDATE is empty when reachability lives here.

BGP path attribute
Optional non-transitive. Type code 14. RFC 4760

Configure it

A leaf with symmetric IRB and an all-active multihomed host.

Cisco NX-OS 10.3 · Nexus 9300, Nexus 9500draft

  1. evpn
     vni 10010 l2
      rd auto
      route-target import auto
      route-target export auto

    `auto` derives both the distinguisher and the targets from the AS and the VNI, so every leaf computes the same value. Configuring one leaf by hand and the rest automatically is the quiet EVPN failure: sessions up, routes advertised, nothing imported.

    RFC 8365 §5.1.2.1

  2. vrf context TENANT-A
     vni 50001
     rd auto
     address-family ipv4 unicast
      route-target both auto evpn
    interface nve1
     member vni 50001 associate-vrf

    The transit L3VNI symmetric IRB routes through. It has to exist on every leaf that participates — a leaf without it bridges perfectly and cannot route, which is the exact symptom of a fabric that "works within a VNI".

  3. fabric forwarding anycast-gateway-mac 0000.2222.3333
    interface Vlan10
     vrf member TENANT-A
     ip address 10.10.10.1/24
     fabric forwarding mode anycast-gateway

    The same gateway address and MAC on every leaf, so a host keeps its default gateway when it moves. Without the shared MAC a move invalidates the host’s ARP cache and traffic stops until it re-resolves.

    RFC 7432 §10.1

  4. interface port-channel10
     switchport mode trunk
     ethernet-segment 10
      esi 0300.0000.0000.0000.0001

    The ESI names the segment. This exact value must appear on the peer leaf too — byte for byte. A mismatch means no designated forwarder election and no split horizon, with nothing logged on either side.

    Common mistake: Deriving the ESI from the switch — its serial, its address, its number. It identifies the link bundle, so both leaves must carry the same one.

    RFC 7432 §5

  5. evpn
     vni 10010 l2
      mac duplication limit 5 time 180 action freeze

    The backstop for an address that keeps moving. Freezing protects the control plane and leaves the address unreachable, so alert on it — a frozen MAC is always a symptom of something else.

    RFC 7432 §15.1

  6. router bgp 65000
     neighbor 10.0.0.100
      remote-as 65000
      address-family l2vpn evpn
       send-community extended

    `send-community extended` is what carries the mobility sequence and the ESI-derived targets. Without it, mobility silently stops being ordered and every move looks like a tie.

    RFC 7432 §15

Verify

show bgp l2vpn evpn summary
That the address family is negotiated, not just the session.
show nve ethernet-segment
The ESI, the peer leaf, and which VLANs this leaf is DF for.
show l2route evpn mac all
Remote MACs and the sequence number each carries.
show bgp l2vpn evpn <mac>
The extended communities, which is where a stripped one shows up.

Caveats

  • Mixing auto-derived and hand-configured route targets produces a fabric that advertises correctly and imports nothing.
  • The ESI must be identical on both leaves of a multihomed pair; a mismatch is silent on both.
  • Without extended communities the mobility sequence is lost and moves stop converging.

When it breaks

Symptom first, because that is what you have when it happens.

  1. A host moves between leaves and traffic keeps going to the old one for several minutes.

    Narrow it down

    1. Check the MAC mobility sequence number on the type 2 route at both leaves.
    2. Confirm the new leaf advertised with a higher sequence than the old one.
    3. Look for the old leaf still advertising and not withdrawing.

    Cause

    Mobility is resolved by a sequence number carried in an extended community. If the new leaf does not increment it, or a device in the path strips the community, remote leaves have nothing to compare and keep what they had.

    Fix

    Verify the mobility extended community survives end to end. A route reflector that drops unknown extended communities produces exactly this.

    MAC MobilityRFC 7432 · PROPOSED STANDARD · February 2015
  2. Duplicate packets arrive at a multihomed host, or some flows toward it disappear entirely.

    Narrow it down

    1. Compare the Ethernet segment identifier on each leaf of the pair — they must match exactly.
    2. Check which leaf won designated forwarder election for each VLAN.
    3. Verify type 4 routes are being received by both leaves.

    Cause

    The two leaves do not agree they are attached to the same segment. With mismatched ESIs each considers itself the only attachment, both forward broadcast, and split-horizon filtering never engages.

    Fix

    Make the ESI identical on both leaves. It identifies the segment, not the leaf, and that is the mistake almost every time.

    Designated Forwarder ElectionRFC 7432 · PROPOSED STANDARD · February 2015
  3. Hosts in the same VNI reach each other but cannot reach another VNI, despite a gateway being configured.

    Narrow it down

    1. Establish whether the fabric uses symmetric or asymmetric IRB.
    2. For symmetric, check the L3VNI is configured and consistent on every leaf involved.
    3. Check whether type 5 routes are present for the remote prefix.

    Cause

    Symmetric IRB routes between VNIs through a transit L3VNI that has to exist at both ends. A leaf missing it can bridge and cannot route, which is exactly this asymmetry.

    Fix

    Configure the L3VNI and its VRF consistently everywhere, and confirm type 5 routes are advertised and imported.

    Symmetric IRB ProceduresRFC 9135 · PROPOSED STANDARD · October 2021
  4. ARP resolution is slow or intermittent across the fabric, though unicast between resolved hosts is fine.

    Narrow it down

    1. Check whether ARP suppression is enabled and whether the leaf holds the binding.
    2. Confirm type 2 routes carry the IP address as well as the MAC.
    3. Look at how broadcast is being replicated for that VNI.

    Cause

    Suppression answers from EVPN-learned bindings. If type 2 routes carry only the MAC the leaf cannot answer and falls back to flooding — and if replication is also incomplete, resolution depends on which path happens to work.

    Fix

    Ensure type 2 routes advertise the IP alongside the MAC, and verify the replication list separately.

    MAC/IP Advertisement RouteRFC 7432 · PROPOSED STANDARD · February 2015
  5. BGP sessions are established, routes are being advertised, and no leaf installs anything.

    Narrow it down

    1. Compare the route targets attached to advertised routes against those imported at the receiver.
    2. Check whether one side auto-derives the target and the other configures it by hand.
    3. Look at the received-routes count before policy versus after.

    Cause

    The import policy does not match what is being exported. Auto-derivation builds the target from the AS and the VNI, so a manual value that differs by one digit is accepted everywhere and imports nothing.

    Fix

    Auto-derive on every leaf, or configure by hand on every leaf. The failure comes from mixing the two.

    Auto-Derivation of RTRFC 8365 · PROPOSED STANDARD · March 2018

Design notes

The ESI names the segment, not the switch. It must be byte-identical on every leaf attached to the same host, and a mismatch produces duplicate delivery and no split horizon with nothing logged anywhere — each leaf’s configuration is internally complete.

Choose symmetric IRB unless there is a specific reason not to. Asymmetric is simpler to explain and needs every routed VNI configured on every leaf, which is the first thing to exhaust leaf table space in a fabric of any size. Symmetric and Asymmetric IRBRFC 9135 · PROPOSED STANDARD · October 2021

Let route targets auto-derive where the platform supports it. Hand-configured targets are the quiet failure of EVPN deployments: everything is configured, the session is established, and nothing imports. Auto-Derivation of RTRFC 8365 · PROPOSED STANDARD · March 2018

Enable duplicate MAC detection and alert on it. It exists to protect the control plane from an address that keeps moving, and a frozen MAC is always a symptom of something else that needs finding. MAC Duplication IssueRFC 7432 · PROPOSED STANDARD · February 2015

Misconceptions

EVPN is a data plane.
It is a control plane — an address family in BGP. The data plane is usually VXLAN and can be MPLS. What EVPN replaces is flood-and-learn: MAC addresses arrive in routing updates rather than being discovered by flooding. BGP MPLS-Based EVPN OverviewRFC 7432 · PROPOSED STANDARD · February 2015
Symmetric and asymmetric IRB are two names for the same thing.
Asymmetric routes at ingress and bridges at egress, so every leaf needs every VNI it might route into. Symmetric routes at both ends through a transit L3VNI, so a leaf needs only the VNIs it has hosts in. The second scales; the first is simpler and runs out of table space first. Symmetric and Asymmetric IRBRFC 9135 · PROPOSED STANDARD · October 2021
The Ethernet segment identifier identifies a leaf.
It identifies the segment — the link bundle a multihomed host is attached by — and must be identical on every leaf attached to it. Treating it as per-device is the most common EVPN multihoming misconfiguration. Ethernet SegmentRFC 7432 · PROPOSED STANDARD · February 2015
EVPN requires MPLS, as its RFC title says.
RFC 7432 defines it over MPLS because that came first. RFC 8365 defines the same control plane over VXLAN, which is what almost every data centre runs. The encapsulation is a separate choice from the control plane. VXLAN/NVGRE EncapsulationRFC 8365 · PROPOSED STANDARD · March 2018

More walkthroughs

Two leaves, one host, both forwardingdesign-choice

All-active multihoming needs three separate mechanisms to agree, and all three key on one identifier that has to be configured the same on both leaves.

Both leaves are configured with the same Ethernet segment identifier. It names the link bundle, not the switch — and that is the sentence to remember. Multihomed host. Leaf 1: ESI 00:11:...:01. Leaf 2: ESI 00:11:...:01. Leaf 3 · remote.

Multihomed hostLeaf 1ESI: 00:11:...:01Leaf 2ESI: 00:11:...:01Leaf 3 · remoteESI 00:11...:01ESI 00:11...:01
  • Link
  • Blocking
  • Packet in flight
  • Discarded
  • Emphasis
Select a device to read its state. Arrow keys walk the topology.
Text equivalent of this diagram
Devices and links at this step
ElementKindState
Multihomed hostswitch
Leaf 1switchESI: 00:11:...:01
Leaf 2switchESI: 00:11:...:01
Leaf 3 · remoteswitch
Multihomed hostLeaf 1linkup · ESI 00:11...:01
Multihomed hostLeaf 2linkup · ESI 00:11...:01
Leaf 1Leaf 3 · remotelinkup
Leaf 2Leaf 3 · remotelinkup
1 / 6

Both leaves are configured with the same Ethernet segment identifier. It names the link bundle, not the switch — and that is the sentence to remember.

What changed

  • Leaf 1: ESI → 00:11:...:01
  • Leaf 2: ESI → 00:11:...:01
  • Emphasis on Multihomed host ↔ Leaf 1
  • Emphasis on Multihomed host ↔ Leaf 2

The address that kept movingfailure

Mobility is decided by a sequence number rather than a clock. When two hosts genuinely share an address, that number climbs until the fabric stops believing it.

A virtual machine migrates from Leaf 1 to Leaf 2. Leaf 2 sees it, and advertises the same MAC with the mobility sequence incremented. Leaf 1. Leaf 2: Type 2 MAC X · seq 1. Route reflector. Leaf 3 · remote.

Leaf 1Leaf 2Type 2: MAC X · seq 1Route reflectorLeaf 3 · remote
  • Link
  • Blocking
  • Packet in flight
  • Discarded
  • Emphasis
Select a device to read its state. Arrow keys walk the topology.
Text equivalent of this diagram
Devices and links at this step
ElementKindState
Leaf 1switch
Leaf 2switchType 2: MAC X · seq 1
Route reflectorrouter
Leaf 3 · remoteswitch
Leaf 1Route reflectorlinkup
Leaf 2Route reflectorlinkup
Route reflectorLeaf 3 · remotelinkup
1 / 5

A virtual machine migrates from Leaf 1 to Leaf 2. Leaf 2 sees it, and advertises the same MAC with the mobility sequence incremented.

A sequence number rather than a timestamp, because comparing timestamps would need the fabric’s clocks to agree closely enough to order two events milliseconds apart.

What changed

  • Leaf 2: Type 2 → MAC X · seq 1
  • Emphasis on Leaf 2 ↔ Route reflector

Terms

ESI
Ethernet Segment Identifier. EVPN type 1 and type 4 use it for multihoming, aliasing, mass-withdraw, and DF election. It is not a VNI.
IMET route (type 3)
An EVPN advertisement saying this leaf wants broadcast and multicast traffic for a VNI. Collected across the fabric, the type 3 routes are the ingress replication list.
MAC mobility sequence
A counter in an extended community, incremented each time a MAC moves, that orders two advertisements without needing synchronised clocks. Strip the community anywhere in the path and every move looks like a tie.
Symmetric IRB
Routing between segments at both ends through a transit L3VNI, so a leaf needs only the VNIs it has hosts in. Asymmetric routes at ingress and bridges at egress, which needs every routed VNI on every leaf.

Check yourself

  • What does EVPN replace in a VXLAN fabric?
  • Two leaves multihome one host but both forward its broadcast traffic. What is wrong?
  • What does symmetric IRB require that asymmetric does not?
  • How does EVPN resolve a MAC address that has moved between leaves?
  • What is a type 3 route for?
  • What stops a multihomed host receiving back a broadcast it just sent?
  • RFC 7432 is titled “BGP MPLS-Based Ethernet VPN”. Does EVPN need MPLS?
  • Two hosts genuinely share one MAC address, one behind each leaf. What does the fabric do?