This article explains how to configure a DHCP server and internet gateway to provide IP addresses and internet access in a VXLAN network. It will build upon the principles and network topology described in Building a VXLAN Network, so it is strongly recommended to read that article first.

• End goal
• Example configuration
• How to build this network

End goal

The goal is to be able to use a DHCP server to assign IP addresses to all devices in a VXLAN network, and provide internet service to them. In a Leaf / Spine architecture, internet access and DHCP services are not provided by the Leaf or Spine switches directly. Rather, they require a device attached to one of the Leaf devices to serve as a DHCP server and gateway. 

In this example we use an EdgeRouter Infinity as the DHCP server and gateway, but other similar devices can also be used.

DHCP and internet will be enabled for just one VXLAN network (i.e., one VNI) in a broader Leaf / Spine network. IPs will correspond to a unique subnet for that VXLAN network. This will work for all devices on this VXLAN network, both directly attached to the same Leaf as the DHCP server / gateway, and attached to other Leaf switches instead. Devices on the VXLAN will behave as if they are connected via an L2 network (e.g., a switch) to the DHCP server / gateway.

The steps in this article can be repeated to provide internet other VXLAN networks (with other VNIs) on the same broader Leaf / Spine network. In that case, addresses will be assigned out of different subnets. However, you can use the same device to provide DHCP / internet across multiple VXLAN networks, and it can be attached to the same port on the same Leaf switch.

Example configuration

In this configuration, there are two layers: an L3 underlay and an L2 overlay. See the image below to understand how the configuration is built, including specific IPs that are configured for this network.

How to build this network

Begin by building a standard VXLAN network with an L3 overlay and L2 underlay. Then layer the components that we need to enable DHCP and internet support on our VXLAN network on top of it.

Note that the VXLAN network configuration and steps are very similar to configuration 2 in the guide Building a VXLAN Network. If you have not done so, we recommend reviewing that guide before attempting this configuration.

Build a VXLAN network

Configure the L3 underlay

An L3 underlay is just an L3 network between Leaf A and Leaf B on top of which you can build many virtual L2 overlay networks. Routing is done using the Mellanox SN2010 as Spine A. Traffic in the L2 overlay networks are not aware of the L3 underlay.

Configure the L3 Router and enable routing on interfaces 1/1 and 1/2

In this example, Spine A's routing functionality is used to direct traffic between Leaf A and Leaf B on different subnets. The configuration is as follows:

• Leaf A: Connected to interface 1/19 using the 100.0.10.0/24 subnet. Leaf A will operate on IP 100.0.10.1/24, while interface 1/19 will operate on IP 100.0.10.2/24.
• Leaf B: Connected to interface 1/20 using the 200.0.10.0/24 subnet. Leaf A will operate on IP 200.0.10.1/24, while interface 1/29 will operate on IP 200.0.10.2/24.

On Spine A, configure interfaces 1/19 and 1/20:

> enable
# configure terminal
(config) # ip l3
(config) # interface ethernet 1/19
(config interface ethernet 1/19) # shutdown
(config interface ethernet 1/19) # speed 100G
(config interface ethernet 1/19) # no switchport force
(config interface ethernet 1/19) # ip address 100.0.10.2/24
(config interface ethernet 1/19) # no shutdown
(config interface ethernet 1/19) # exit
(config) # interface ethernet 1/20
(config interface ethernet 1/20) # shutdown
(config interface ethernet 1/20) # speed 100G
(config interface ethernet 1/20) # no switchport force
(config interface ethernet 1/20) # ip address 200.0.10.2/24
(config interface ethernet 1/20) # no shutdown
(config interface ethernet 1/20) # exit

Note: In some versions of the Mellanox Onyx network operating system, the command ip l3 has been replaced by the command ip routing.

Then, use VRF to route traffic to Leaf A and Leaf B:

(config) # ip routing vrf default
(config) # show ip route
(config) # ip arp vrf default 100.0.10.1 00:00:00:00:00:01
(config) # ip arp vrf default 200.0.10.1 00:00:00:00:00:02
(config) # show ip arp

Note that two MAC addresses were used here: 00:00:00:00:00:01 for Leaf A and 00:00:00:00:00:02 for Leaf B. These correspond to MAC addresses that you will choose for Leaf A and Leaf B, respectively, in their VTEP capacities. This will be done in the next section.

Configure the L2 overlay

A L2 overlay is a virtual L2 network that will utilize the L3 underlay as a bridge across VTEPs.

Create VLANs on Leaf A and Leaf B

On each of the Leaf switches, VLAN 100 will connect to the Server, and VLAN 300 will connect to the Spine. In later steps, VLAN 100 will map out to the VXLAN tunnel that we are creating.

Commands for both Leaf A and Leaf B:

enable 
vlan database
vlan 100
vlan 300
exit

configure 
interface 0/5
vlan participation exclude 1
vlan participation include 100
vlan pvid 100
exit

interface 0/54
vlan participation exclude 1
vlan participation include 300
vlan pvid 300
exit

Set port performance to 25G

SFP28 ports on UDC Leaf default to 10G performance. If you are using them to connect to 25 GbE NICs, you should set them manually to 25G performance. In this case:

interface 0/5
speed 25G full-duplex
exit

Configure VXLAN tunnel on Leaf A and Leaf B

VXLAN configuration consists of several steps repeated on both Leaf switches:

1. Define a new VXLAN tunnel; in this case, we will use VNI 200
2. Map the VXLAN tunnel to the inbound ports on which traffic will be received; in this case, that is interface 0/54
3. Map to the VLAN we defined earlier; in this case, VLAN 100
4. Define the local IP and MAC as sending / receiving traffic
5. Define the remote IP of the other VTEP
   • The local and remote IPs should not be on the same subnet in this configuration
6. Define the local MAC as sending / receiving traffic
7. Define the MAC of the next destination of traffic; in this case, the MAC of the Spine
   • Note that this can be determined with the show interface ethernet 1/19 or show interface ethernet 1/20 command on the Mellanox SN2010.

Commands for Leaf B:

vxlan 200
source-interface 0/54
vlan 100
ip 100.0.10.1/24
remote-ip 200.0.10.1
mac 00:00:00:00:00:01
neighbor SPINE_MAC
exit

Commands for Leaf B:

vxlan 200
source-interface 0/54
vlan 100
ip 200.0.10.1/24
remote-ip 100.0.10.1
mac 00:00:00:00:00:02
neighbor SPINE_MAC
exit

Connect the DHCP server / internet gateway

The next step is to allow the DHCP server / internet gateway (in this case, EdgeRouter Infinity) to be a part of the VXLAN 200 network, in order to serve IP addresses and provide internet access to all other devices on VXLAN 200, but no other devices.

In the next section, we will configure the DHCP server / internet gateway to serve IP addresses out of the 192.168.5.0/24 subnet. In this section, just set Leaf B up to allow the DHCP server / internet gateway to participate in VXLAN 200 once it is configured.

In this configuration, the EdgeRouter Infinity is connected to interface 31. It must be assigned to VLAN 100, which is tied to VXLAN 200. Additionally, VLAN 100 tagging should be enabled on this interface.

Commands for Leaf B:

interface 0/31
vlan participation exclude 1
vlan participation include 100
vlan pvid 100
vlan tagging 100
exit

Note that the DHCP server could serve IPs from different subnets to different VXLAN networks to the same physical interface. All that would be required would be repeating the vlan participation include and vlan tagging (no need for vlan pvid) commands for the VLANs tied to those VXLAN networks.

Configure the DHCP server and internet gateway

Create a new DHCP server with the subnet from which you would like to assign IPs for this VXLAN. You should specify:

• The subnet itself (in this case, 192.168.5.0/24) and the range of IPs that should be used for assignment (in this case, 192.168.5.2 through 192.168.5.255)
• A router (in this case, we can use 192.168.5.1; this will be assigned to the correct port later
• A DNS server (in this case, we can use the primary DNS for the EdgeRouter: 192.168.1.1)

Attach this DHCP server to the correct interface and VLAN

Tie traffic from the VXLAN network to the router / DHCP server defined in the previous step. You should specify:

• Interface on the router to which a Leaf switch is connected (in this case, eth7)
• VLAN carrying traffic for the VXLAN network that will be tied to this DHCP server / gateway (in this case, VLAN 100, which corresponds to our VXLAN VNI 200)
• IP address for the router (192.168.5.1/24, as defined above)

Ensure routing is in place

With EdgeRouter, this step is handled automatically given the previous step. However, with other routing devices, you may have to ensure that static routing is enabled in such a way that destinations in the configured subnet are sent to this VXLAN network.

In this example, you would ensure that destinations on the 192.168.5.0/24 subnet are forwarded to interface 7, VLAN 100 (corresponding to VXLAN VNI 200).

Enable DNS forwarding

Tie the DNS server to the interface (in this case, 7) and VLAN (in this case, 100) corresponding to our VXLAN network:

How to test this network

At this point, the server should retrieve an IP from the EdgeRouter Infinity over the VXLAN overlay. You can use standard tools like ifconfig (Linux) or ipconfig (Windows) to check that IPs have been assigned on both Server 1 and Server 2. Both servers should also be able to access the internet using standard browsers, and ping each other.