← All chapters

VXLAN MP-BGP EVPN Control Plane

Overlay networks are designed to provide communication channels for all the different kinds of traffic that is sent over the network. VTEP-to-VTEP communication is relatively simple to handle due to its unicast nature with known sources and destinations. Handling other types of traffic is also very important.

Non-unicast traffic and unknown unicast traffic in overlay networks that must be processed by the control plane is referred to as BUM:

Broadcast

Frames that are sent to all devices in a VLAN (for example, ARP requests)

Unknown Unicast

Frames that are sent to a destination MAC address that is not in the forwarding table.

Multicast

Frames that are sent to a group of devices that have joined a multicast group.

The scalability of an overlay network depends on its ability to handle BUM data. The control plane of overlay network technology defines how to manage BUM traffic. In VXLAN, the standard does not describe the control plane signaling protocol for determining the location of endpoints. Therefore, you can choose the control plane implementation that fits your specific network environment.

The two most widely used control planes for VXLAN technology with Cisco equipment are as follows:

Multicast control plane (flood and learn)

Relies on the underlying multicast capability of the network below the VXLAN overlay. This network distributes the VTEP MAC information through the data-plane learning, initially flooding the traffic. Once multicast communication is established, all members of a multicast group receive the traffic. This approach is prone to slow convergence, high CPU usage on switches, scalability limitations, potential MAC spoofing attacks, and worse bandwidth utilization. Worse bandwidth utilization is where too much traffic gets forwarded due to the multicast nature of communication.

MP-BGP EVPN control plane

The VTEP information is distributed through Multiprotocol Bridge Gateway Protocol (MP-BGP) updates. This is a much more controlled way of handling control plane updates, but is more complex and requires devices to support the MP-BGP control plane. This approach removes the scalability issues, initial learning broadcasts, MAC spoofing attack vulnerability, and provides a fine-tuned way to establish communication between endpoints, also enabling application-centric deployment.

In modern data center design, the MP-BGP EVPN control plane is by far the preferred approach. It is the only approach that can enable very fine-grained connectivity control not limited to multicast group membership. As shown in the image the Cisco ACI system uses MP-BGP EVPN to create an application-centric SDN infrastructure managed by the APIC controller.

MP-BGP EVPN Control Plane Overview

The control plane in any overlay must establish communication across diverse, multivendor data center environments, where device and protocol interoperability are considered throughout the entire data center lifecycle.

MP-BGP EVPN is a standard-based VXLAN control protocol. It introduces control-plane learning for end hosts behind remote VTEPs. It provides control-plane and data-plane separation and a unified control plane for both Layer 2 and Layer 3 forwarding in a VXLAN overlay network.

The MP-BGP EVPN control plane offers eight main benefits:

With MP-BGP EVPN, VXLAN control plane devices might be MP-IBGP EVPN peers or route reflectors, or MP External BGP (MP-EBGP) EVPN peers. Their operating system software needs to support MP-BGP EVPN so that it can understand the MP-BGP EVPN updates and distribute them to other MP-BGP EVPN peers using the standards-defined constructs.

For data forwarding, IP transport devices perform IP routing based only on the outer IP address of a VXLAN encapsulated packet. They do not need to support the VXLAN data encapsulation and decapsulation functions.

Similar to other network routing control protocols, MP-BGP EVPN distributes Network Layer Reachability Information (NLRI) for the network. A unique feature of EVPN NLRI is that it includes both the Layer 2 and Layer 3 reachability information for end hosts that reside in the EVPN VXLAN overlay network. In other words, it advertises both MAC and IP addresses of EVPN VXLAN end hosts.

This ability to advertise both MAC and IP addresses forms the basis for VXLAN-integrated routing and bridging support.

MP-BGP protocol distributes the VTEP information while VXLAN routing allows you to route traffic between different VXLANs.

The entire process of MP-BGP EVPN control plane operation includes the following main steps that are also shown in the image.

Step 1

A VTEP in the MP-BGP EVPN learns the MAC addresses and IP addresses of locally attached end hosts through local learning. This learning can be based on the local data plane using the standard Ethernet and IP learning procedures.

Step 2

After the VTEP learns the local host’s MAC and IP addresses, it advertises this information in the MP-BGP EVPN control plane. This allows other VTEPs to receive the host details. This approach enables EVPN VTEPs to learn the remote end hosts in the MP-BGP EVPN control plane. VTEPs advertise EVPN routes through the Layer 2 VPN EVPN address family.

The BGP Layer 2 VPN EVPN routes include the following information:

MP-BGP EVPN uses the BGP extended community attribute to transmit the exported route targets in an EVPN route. When an EVPN VTEP receives an EVPN route, it compares the route-target attributes in the received route to its locally configured route-target import policy. It does this to decide whether to import or ignore the route.

Using the BGP extended community attribute approach uses the decades-old MP-BGP VPN technology (RFC 4364) and provides scalable multitenancy. Scalable multitenancy is where a node that does not have a VRF instance locally does not import the corresponding routes.

When a VTEP switch originates MP-BGP EVPN routes for its locally learned end hosts, it uses its own VTEP address as the BGP next hop. This BGP next hop must remain unchanged through the route distribution across the network. It must remain unchanged because the remote VTEP must learn the originating VTEP address as the next hop for VXLAN encapsulation when forwarding packets for the overlay network.

Step 3

The route reflector works like a neighbor relationship aggregation point and provides BGP update exchange.

Step 4

As a result of the MP-BGP EVPN control plane operation, every VTEP knows how to reach every end system.

VTEP Peer Discovery and Authentication in MP-BGP EVPN

Before MP-BGP EVPN, VXLAN did not have a control protocol-based VTEP peer-discovery mechanism or a method for authenticating VTEP peers. These limitations present major security risks in real-world VXLAN deployments because they allow easy insertion of a rogue VTEP into a VNI segment to send or receive VXLAN traffic.

For security when connecting different endpoints, the VTEP that is sending traffic needs to be in the VTEP peer list for its traffic to be forwarded.

In the image VTEPs 1, 2, and 3 have established peer adjacency and are known to one another. VTEP-4 is trying to communicate with an IP in VTEP-1, but since the adjacency with VTEP-4 has not been established, its traffic is not allowed and is discarded.

With the MP-BGP EVPN control plane, VTEPs use the following peer discovery and authentication mechanism:

Step 1

The VTEP device first needs to establish BGP neighbor adjacency with other VTEPs or with Internal Border Gateway Protocol (IBGP) route reflectors. In addition to the BGP updates for end-host Network Layer Reachability Information (NLRI), VTEPs exchange the following information about themselves through BGP:

Step 2

When a VTEP receives BGP EVPN route updates from a remote VTEP BGP neighbor, it adds the VTEP address from that route advertisement to the VTEP peer list.

Step 3

Local VTEP uses this VTEP peer list as an allow list of valid VTEP peers.

Step 4

If your local VTEP receives a packet from another VTEP that is not on that allow list, it considers the source of the packet as invalid or unauthorized.

Step 5

Your local VTEP will discard VXLAN encapsulated traffic from an invalid VTEP.

For additional security, you can apply the existing BGP MD5 authentication to the BGP neighbor sessions. Then, the switches cannot become BGP neighbors to exchange MP-BGP EVPN routes until they successfully authenticate each other.

Distributed Anycast Gateway in MP-BGP EVPN

In a distributed overlay system, Layer 3 termination needs a consistent method to end Layer 2 domains so application communication isn’t disrupted during migration.

Applications typically use default gateways to communicate with external systems over Layer 3. Default gateways in traditional networks need to be configured in such a way that all devices have reachability to the gateway, but in VXLAN configurations, this may not be possible.

In MP-BGP EVPN, you can configure any VTEP in a VNI as the distributed anycast gateway for end hosts in its IP subnet. In this case, all VTEPs will have the same virtual gateway IP address and the virtual gateway MAC address.

With EVPN anycast gateway, hosts in a VNI use their local VTEP as the default gateway to send traffic outside their IP subnet.

The capability to use the VNI endpoint as the default gateway enables optimal forwarding for northbound traffic from end hosts in the VXLAN overlay network. A distributed anycast gateway also offers the benefit of seamless host mobility in the VXLAN overlay network.

All VTEPs in a VNI share the same gateway IP and virtual MAC address. So, when an end host moves to another VTEP, it doesn’t need to send a new ARP request to find the gateway MAC address.

ARP Suppression in MP-BGP EVPN

To reduce network flooding caused by broadcast traffic from ARP requests, the MP-BGP EVPN control plane provides an ARP suppression feature. When you enable the ARP suppression feature for a VNI, each VTEP maintains an ARP suppression cache table. This table includes known IP hosts and their associated MAC addresses in the VNI segment.

Local VTEP keeps an extended ARP cache of local and remote addresses and responds to local ARP requests from its cache reducing flooding.

When an end host in the VNI sends an ARP request, its local VTEP intercepts it. The VTEP then checks its ARP suppression cache for the requested IP address. If it finds a match, the local VTEP sends an ARP response on behalf of the remote end host.

The local host learns the MAC address of the remote host in the ARP response. If the local VTEP doesn’t have the IP address in its ARP suppression table, it floods the ARP request to other VTEPs in the VNI.

This ARP flooding can occur for the initial ARP request to a silent host in the network. VTEPs don’t see any traffic from a silent host. They only see it when another host sends an ARP request for its IP address and the silent host replies.

After the local VTEP learns about the MAC and IP address of the silent host, the information is distributed through the MP-BGP EVPN control plane to all other VTEPs. Any subsequent ARP requests do not need to be flooded.

Most end hosts send Gratuitous ARP (GARP) or Reverse ARP (RARP) requests to announce themselves to the network right after they come online. This allows the local VTEP to immediately have the opportunity to learn their MAC and IP addresses and distribute this information to other VTEPs through the MP-BGP EVPN control plane.

VTEPs should learn most active IP hosts in the VXLAN EVPN through local learning or control plane-based remote learning. As a result, ARP suppression reduces the network flooding that host ARP learning behavior causes.

The entire process of ARP suppression in MP-BGP EVPN consists of the following steps:

Step 1

Host 1 in VLAN 10 sends an ARP request for the IP address of Host 2 (IP-2).

Step 2

VTEP-1 intercepts the ARP request and checks its ARP suppression cache. It finds a match for IP-2 in VLAN 10 in its ARP suppression cache.

Step 3

VTEP-1 sends an ARP response to Host 1 with MAC-2.

Step 4

Host 1 learns the IP-2 and MAC-2 mapping.

Multicast Control Plane (Flood and Learn)

The MP-BGP EVPN control plane mode for VXLAN is the more powerful option but requires a dedicated control plane to distribute end-host reachability information. It is the most established approach in the modern data enters and it is used by SDN network solutions such as Cisco ACI.

The second type of VXLAN control plane operation is flood-and-learn mode, which uses a multicast-enabled transport IP network. In this mode, the process of traffic forwarding triggers end-host information learning and VTEP discovery. There is no control protocol to distribute end-host reachability information among VTEPs.

Instead of a dedicated control plane, the native multicast capability of the underlying network is used to register all the end-host information within a VNI to a multicast group. Shared or individual multicast groups can be used for each VNI. However, using a single multicast group will cause excessive traffic flow in most designs, but is a simple way to implement VXLAN flood-and-learn.

The end-host information that is distributed through the multicast group is stored in each VTEP's local MAC-VTEP table. BUM traffic sent through multicast initiates updates and initial learning, while the data plane itself is processed through VXLAN VTEP-to-VTEP communication and not multicast.

ARP requests are forwarded to the VNI-assigned multicast group in the underlay network. This queries VTEPs for their local host information.

For the flood-and-learn control plane, the transport IP infrastructure must support Any-Source Multicast (ASM). Every VTEP that you configure for a specific VXLAN with a certain VNI will join the same multicast group. Initially, each VTEP will only learn the MAC addresses of its local end hosts and virtual machines. To learn remote MAC addresses, your VTEP will use the conversational MAC address learning technique.

The following describes how this mechanism works. For example, End System A needs to send some data to End System B.

Step 1

End System A generates an ARP request to discover the MAC address of End System B. The ARP request is sent with the source MAC address MAC-A and the destination MAC address is broadcast (FF:FF:FF:FF:FF:FF). When the VTEP receives the Layer 2 frame, it looks in its local table for the proper destination for this frame. This destination can be a local interface or remote VTEP IP address.

Step 2

In BUM traffic, the VTEP will encapsulate the original Layer 2 frame and send it over the multicast group that you configured for the specific VNI. VTEP adds VXLAN, outer UDP, IP, and Ethernet headers to your original ARP request. In the outer IP header, you can see the multicast address (239.1.1.1) as the packet destination IP address. This allows you to deliver that packet to all VTEPs that work with the same VNI. For the outer destination MAC address, your VTEP also uses the multicast MAC address (01:00:5E:01:01:01). VTEP obtained this value from a known IP multicast address. Then the VTEP forwards the encapsulated Layer 2 frame to the multicast rendezvous point, which forwards a copy of the packet to every VTEP that has joined the multicast group.

Step 3

Each VTEP will receive and decapsulate the VXLAN packet and learn the MAC address of End System A (MAC-A) pointing to the remote VTEP address (IP-1).

Step 4

Each VTEP forwards the ARP request to its local network. At this moment, End System B receives the ARP request from End System A. The next step is to send an ARP response back to End System A.

ARP replies are forwarded to the VNI-assigned multicast group. This updates the source VTEP database with remote VTEP information.

Step 5

End System B generates a unicast ARP response with the source MAC address, MAC-B, and the destination MAC address, MAC-A.

Step 6

When VTEP-2 receives the ARP response, it will look in its local table and will find a corresponding entry. Now VTEP-2 knows that traffic going to End System A must be sent to the VTEP-1 address. VTEP-2 encapsulates the ARP response with a VXLAN header and sends the unicast packet to VTEP-1.

Step 7

VTEP-1 receives and decapsulates the original ARP response. At this moment, VTEP-1 learns the source MAC address of End System B and creates an entry in its local table with information about how to reach End System B.

Step 8

Then VTEP-1 sends an original ARP response frame to End System A.

ElementRole
VNIUniquely identifies a Layer 2 segment.
Multicast Group (G)Carries traffic for VNI across the underlay.
Ingress VTEPsSends BUM to the Multicast Group (G).
Egress VTEPJoin a Multicast Group (G) using IGMP/MLD.
Underlay Protocol-Independent MulticastBuilds multicast trees (any source multicast / single source multicast).