UniFi Gateway Consoles WiFi Switching Cameras & Security New Integrations Accessory Tech Identity UISP
Support
UniFi Help Articles UISP Help Articles Video Tutorials Community Forums UniFi Design Center UISP Design Center RMA and Warranty Contact Support
Store
Support
UniFi Help Articles UISP Help Articles Video Tutorials Community Forums UniFi Design Center UISP Design Center RMA and Warranty Contact Support
UniFi Gateway Consoles WiFi Switching Cameras & Security New Integrations Accessory Tech Identity UISP Store
  1. Ubiquiti Support and Help Center
  2. UniFi Data Center
  3. UDC Switching
Help Center
Back to Top

UDC Leaf (EA) - Building a VXLAN network

2022-11-22 15:33:48 UTC

This article will describe how to make a VXLAN tunnel between two servers using UDC Leaf switches. It shows two different configurations that allow two servers to communicate with each other across a VXLAN tunnel, like in a large data center. The principles and directions of this article scale to a much larger network.

This article assumes that you have already set up your UDC Leaf units and can access the command line using SSH. See this article for an explanation on setting up a UDC Leaf unit. 

Why VXLAN is useful

VXLAN is a technology to create virtual, fully-separated L2 networks across a data center. They are similar to L2 VLANs, but a single network can have up to ~16 million VXLANs. Devices on each VXLAN network can only communicate with other devices on the same VXLAN network, unless routing is enabled between multiple VXLAN networks, allowing for secure multitenancy.

This enables you to build large networks where devices at any location in the data center can be on the same virtual L2 network, with a very large number of virtual L2 networks supported. All of this can take place at 100 GbE performance levels using Leaf and Spine switches. This is highly useful for modern, virtual workloads such as live migration of virtual machines and software-defined storage in large, multitenant data centers.

Key terminology

  • Virtual Network Identifier (VNI): The identifier of one VXLAN virtual network
  • Virtual Tunnel Endpoint (VTEP): The role each UDC Leaf switch plays in terminating a VXLAN tunnel and providing a virtual L2 interface to the devices connected to them (e.g., Servers)
  • L2 overlay: A virtual network configured for a VNI across any two VTEPs for two devices (e.g., Servers) to communicate to each other as if on the same L2 switched network
  • L3 underlay: A routed L3 network between Leaf switches through which L2 overlays traffic is directed from one VTEP to another

Overview of example configurations

We will look at two different configurations, each utilizing two UDC Leaf switches.

Configuration 1: Two UDC Leaf switches alone

In this configuration, we will use only two UDC Leaf switches. They will directly connect to each other with no intermediary. One server will connect to each UDC Leaf switch.

In this example, we will call our two switches Leaf A and Leaf B. We will connect the UDC Leaf switches to each other on port 54 (QSFP28), and connect servers to each UDC Leaf switch on port 5 (SFP28).

Configuration 1 diagram

Leaf_Setup_UISP_Leaf_VXLAN_v01.png

Configuration 2: Two UDC Leaf switches with a third-party spine switch

In this configuration, we will use two UDC Leaf switches, with one Mellanox SN2010 switch sitting between them, as a stand-in Spine switch. This is a basic version of the networks created for much larger data centers, wherein each Leaf switch connects to each Spine switch, and there are multiple Spine switches. This topology (known as "Clos topology") enables efficient traffic routing across large data centers with multiple VXLAN networks.

Note that the Mellanox SN2010 is used as an example, but this configuration should work with other 100 GbE routers and L3 switches as well.

In this example, we will call our two UDC Leaf switches Leaf A and Leaf B, and our Mellanox SN2010 switch Spine A. We will connect Leaf A on port 54 to Spine A on port 19. We will connect Leaf B on port 54 to Spine A on port 20. We will connect servers to each UDC Leaf switch on port 5 (SFP28).

Configuration 2

Spine_Setup_UISP_Leaf_Spine_VXLAN_v01.png

Configuration 1: Two UDC Leaf switches alone

In this configuration, we will form a VXLAN connection by connecting the two UDC Leaf switches directly. See below for how the configuration is built, including specific IPs that are configured for this network.

Configuration 1 diagram with IP addresses

Leaf_Setup_UISP_Leaf_VXLAN_v02.png

Create VLANs on Leaf A and Leaf B

On each of the Leaf switches, VLAN 100 will connect to the Server, and VLAN 200 will connect to the other Leaf. In later steps, VLAN 100 will map out to the VXLAN tunnel that we are creating.

In this step, also set the MTU of each port to maximum (9216) in order to support jumbo frames for maximum performance and ensure all packets are allowed through without dropping.

Commands for both Leaf A and Leaf B:

enable

vlan database
vlan 100
vlan 200
exit

configure
interface 0/5
vlan participation exclude 1
vlan participation include 100
vlan pvid 100
mtu 9216
exit

interface 0/54
vlan participation exclude 1
vlan participation include 200
vlan pvid 200
mtu 9216
exit

Set port performance to 25G

SFP28 ports on UDC Leaf default to 10G performance. If you are using them to connect to 25 GbE NICs, you should set them manually to 25G performance. In this case:

interface 0/5
speed 25G full-duplex
exit

Configure VXLAN tunnel on Leaf A and Leaf B

VXLAN configuration consists of several steps repeated on both Leaf switches:

  • Define a new VXLAN tunnel; in this case, we will use VNI 200
  • Map the VXLAN tunnel to the inbound ports on which traffic will be received; in this case, that is interface 0/54
  • Map to the VLAN we defined earlier; in this case, VLAN 100
  • Define the local IP and MAC as sending / receiving traffic
  • Define the remote IP of the other VTEP
    • The local and remote IPs should be on the same subnet in this configuration
  • Define the local MAC as sending / receiving traffic
  • Define the MAC of the next destination of traffic; in this case, the MAC of the other Leaf
    • Note that both MAC addresses will be self-assigned. In this case, 00:00:00:00:00:01 and 00:00:00:00:00:02

Commands for Leaf A:

vxlan 200
source-interface 0/54
vlan 100
ip 100.0.10.1/24
remote-ip 100.0.10.2
mac 00:00:00:00:00:01
neighbor 00:00:00:00:00:02

Commands for Leaf B:

vxlan 200
source-interface 0/54
vlan 100
ip 100.0.10.2/24
remote-ip 100.0.10.1
mac 00:00:00:00:00:02
neighbor 00:00:00:00:00:01

Configuration 2: Two UDC Leaf switches with a third-party spine switch

In this configuration, we have two layers: An L3 underlay and an L2 overlay. See below for how the configuration is built, including specific IPs that are configured for this network.

Configuration 2 diagram with IP addresses

Spine_Setup_UISP_Leaf_Spine_VXLAN_v02__1_.png

In this case, Spine A is a Mellanox SN2010 switch running the Mellanox Onyx Network Operating System. Commands are written specifically for Onyx, using two of the SN2010's QSFP28 100 GbE ports. The same principles apply to any L3 router.

Configure the L3 underlay

An L3 underlay is just an L3 network between Leaf A and Leaf B on top of which you can build many virtual L2 overlay networks. Routing is done using the Mellanox SN2010 as Spine A. Traffic in the L2 overlay networks are not aware of the L3 underlay.

Configure the L3 Router and enable routing on interfaces 1/19 and 1/20

In this example, we will use Spine A's routing functionality to direct traffic between Leaf A and Leaf B on different subnets. The configuration is as follows:

  • Leaf A: Connected to interface 1/19 using the 100.0.10.0/24 subnet. Leaf A will operate on IP 100.0.10.1/24, while interface 1/19 will operate on IP 100.0.10.2/24.
  • Leaf B: Connected to interface 1/20 using the 200.0.10.0/24 subnet. Leaf A will operate on IP 200.0.10.1/24, while interface 1/29 will operate on IP 200.0.10.2/24.

On Spine A, configure interfaces 1/19 and 1/20:

> enable
# configure terminal
(config) # ip l3
(config) # interface ethernet 1/19
(config interface ethernet 1/19) # shutdown
(config interface ethernet 1/19) # speed 100G
(config interface ethernet 1/19) # no switchport force
(config interface ethernet 1/19) # ip address 100.0.10.2/24
(config interface ethernet 1/19) # no shutdown
(config interface ethernet 1/19) # exit
(config) # interface ethernet 1/20
(config interface ethernet 1/20) # shutdown
(config interface ethernet 1/20) # speed 100G
(config interface ethernet 1/20) # no switchport force
(config interface ethernet 1/20) # ip address 200.0.10.2/24
(config interface ethernet 1/20) # no shutdown
(config interface ethernet 1/20) # exit

NOTE: In some versions of the Mellanox Onyx network operating system, the command ip l3 has been replaced by the command ip routing

Then, use VRF to route traffic to Leaf A and Leaf B:

(config) # ip routing vrf default
(config) # show ip route
(config) # ip arp vrf default 100.0.10.1 00:00:00:00:00:01
(config) # ip arp vrf default 200.0.10.1 00:00:00:00:00:02
(config) # show ip arp

Note that we used two MAC addresses here: 00:00:00:00:00:01 for Leaf A and 00:00:00:00:00:02 for Leaf B. These correspond to MAC addresses that we will choose for Leaf A and Leaf B, respectively, in their VTEP capacities. This will be done in the next section.

Configure the L2 overlay

A L2 overlay is a virtual L2 network that will utilize the L3 underlay as a bridge across VTEPs.

Create VLANs on Leaf A and Leaf B

On each of the Leaf switches, VLAN 100 will connect to the Server, and VLAN 200 will connect to the Spine. In later steps, VLAN 100 will map out to the VXLAN tunnel that we are creating.

Commands for both Leaf A and Leaf B:

enable

vlan database
vlan 100
vlan 200
exit

configure
interface 0/5
vlan participation exclude 1
vlan participation include 100
vlan pvid 100
exit

interface 0/54
vlan participation exclude 1
vlan participation include 200
vlan pvid 200
exit

Optional: Using multiple VXLANs on the same physical port

It is possible to have multiple VXLAN networks run over the same physical SFP28 port to the same device, if that device has multiple systems that would need to be assigned to different VXLAN networks. The common use case for this would be VMs on the same server / hypervisor with different MAC addresses being put into different VXLAN networks. Using that, it would be easily possible to put the different VMs into different subnets as detailed in Connecting a DHCP Server and Internet Gateway.

This can be done without having to set any VLAN configuration on the hypervisor / virtual switch, or alerting the hypervisor / virtual switch to the structure of the network at all. To achieve this, have the physical port be included in multiple VLANs, avoid using vlan pvid, and use the vlan association mac command to tie the each VM to its VXLAN network.

For instance, consider a network where a single server is connected to physical interface 0/5 and has VM A and VM B. VM A's MAC address is AA:AA:AA:AA:AA:AA and VM B's MAC address is BB:BB:BB:BB:BB:BB. Consider that you would want to have VM A participate in VXLAN 100 tied to VLAN 100, and VM B participate in VXLAN 150 tied to VLAN 150.

Instead of the above commands for interface 0/5, you would type the following:

enable
configure
interface 0/5
vlan participation exclude 1
vlan participation include 100
vlan participation include 150
exit
exit

vlan database
vlan association mac AA:AA:AA:AA:AA:AA 100
vlan association mac BB:BB:BB:BB:BB:BB 150
exit

Set port performance to 25G

SFP28 ports on UDC Leaf default to 10G performance. If you are using them to connect to 25 GbE NICs, you should set them manually to 25G performance. In this case:

interface 0/5
speed 25G full-duplex
exit

Configure VXLAN tunnel on Leaf A and Leaf B

VXLAN configuration consists of several steps repeated on both Leaf switches:

  • Define a new VXLAN tunnel; in this case, we will use VNI 200
  • Map the VXLAN tunnel to the inbound ports on which traffic will be received; in this case, that is interface 0/54
  • Map to the VLAN we defined earlier; in this case, VLAN 100
  • Define the local IP and MAC as sending / receiving traffic
  • Define the remote IP of the other VTEP
    • The local and remote IPs should not be on the same subnet in this configuration
  • Define the local MAC as sending / receiving traffic
  • Define the MAC of the next destination of traffic; in this case, the MAC of the Spine
    • Note that this can be determined with the show interface ethernet 1/19 or show interface ethernet 1/20 command on the Mellanox SN2010.

Commands for Leaf A:

vxlan 200
source-interface 0/54
vlan 100
ip 100.0.10.1/24
remote-ip 200.0.10.1
mac 00:00:00:00:00:01
neighbor SPINE_MAC

Commands for Leaf B:

vxlan 200
source-interface 0/54
vlan 100
ip 200.0.10.1/24
remote-ip 100.0.10.1
mac 00:00:00:00:00:02
neighbor SPINE_MAC

Note: In larger applications, it is possible to connect each Leaf to multiple Spine switches without using LAG / LACP. Each VXLAN will correspond to one source-interface containing a connection to a Spine switch. Each source-interface may have multiple VXLANs connected to it. Each source-interface should also have a different VLAN assigned to it.

(E.g., in the above example, interface 0/54 is connected to a Spine and is assigned to VLAN 100; interface 0/53 could also be used as a Spine connection for different VXLANs by being assigned to another VLAN, such as VLAN 101.)

Testing the configurations

In both configurations, you should be able to connect servers to port 5 of Leaf A and B, assign the servers IP addresses manually, and proceed across the network. For instance, using Linux:

Server A (network adapter ens1f0 connected to Leaf A, port 5):

> sudo ip address add 10.10.10.10/24 broadcast 10.10.10.255 dev ens1f0

Server B (network adapter ens1f0 connected to Leaf B, port 5):

> sudo ip address add 10.10.10.20/24 broadcast 10.10.10.255 dev ens1f0
> ping -I ens1f0 10.10.10.10
Was this article helpful?
16 out of 17 found this helpful