Building Linux L3 switch/router on x86 - Part4 - Openvswitch Installation and Configuration

In a last tutorial we switched our Linux layer 3 switch to wireless Master mode in order to provide wireless services to wireless LAN users.We will continue with building our Linux L3 switch and install Open vSwitch on CentOS in order to connect wired LAN users. The introduction tutorial, the list of software and hardware requirements is here.

Although they are several projects available that can help us to bridge Ethernet interfaces together and force the Linux box to act as a switch (VDE switch, LiSA or bridge utils) I would like to use Openvswitch to do this job as it offers many features you can find in hardware switches.

They are four GigabitEthernet cards presented in PCI slots. Three of them are Intel e1000 - 82540EM Gigabit controllers and one is D-Link System DGE-528T controller. There is also Broadcom BCM4318 802.11g Wireless LAN Controller presented and using for connecting wirelless LAN users as we have shown here.

We will use DGE-528T Gigabit controller to connect our Linux box to the Internet network. The IP address for this interface will be automatically assigned from a provider's DHCP server.

Remaining three Intel e1000 Gigabit controllers and a Wireless controller will be bridged together and traffic forwarded between these interfaces based on destination Ethernet address rather than IP address. Computers connected to those ports belong to the same IP subnet - 172.18.0.0/16 .

There must be also a virtual interface created and configured with the IP address - 172.18.100.150/16. The IP address is used as the default gateway IP address for all hosts on the subnet 172.18.0.0/16 . All computers will forward a traffic to this IP address if the destination IP address is out of the scope their subnet.

1. Check if 8021q and bridge module is loaded

$ sudo su
# lsmod | grep 8021q

8021q 19587 0
garp 5901 1 8021q

If 8021q module is not loaded, load a module to the kernel with modprobe 8021q command.

The Open vSwitch datapath requires bridging support (CONFIG_BRIDGE) to be built as a kernel module. The bridge module must not be loaded or in use.

# lsmod | grep bridge

bridge 61159 0
stp 1563 2 bridge,garp
llc 4392 3 bridge,garp,stp

In this case, the bridge module is running and we must remove it.

# rmmod bridge

2. Download Openvswitch and Extract it

# wget http://openvswitch.org/releases/openvswitch-1.2.0.tar.gz
# tar zxvf ./openvswitch-1.2.0.tar.gz

3. Install Necessary Dependences for Openvswitch Compilation

# yum install python PyQt4 python-zope-interface python-twisted-conch python-simplejson perl kernel-devel

4. Openvswitch Installationafter restart

# cd ./openvswitch-1.2.0
# ./configure --with-linux=/lib/modules/`uname -r`/build
# make
# make install

5. Load Openvswitch kernel module

After building, a kernel module openvswitch_mod_ko is saved in ./datapath/linux/ directory. We can try to load it to LInux kernel with insmod command.

# insmod ./datapath/linux/openvswitch_mod.ko

Check if module is loaded:

# lsmod | grep openvswitch_mod

openvswitch_mod 67954 0

6. Load openvswitch kernel module and make it loaded during boot time

Even the module can be loaded to kernel with insmod command it cannot be loaded with modprobe command. We are going to fix it.

a) Unload a module from the kernel and copy it to the standard location

# rmmod openvswitch_mod
# mkdir /lib/modules/2.6.32-71.29.1.el6.i686/kernel/net/openvswitch/
# cp ./datapath/linux/openvswitch_mod.ko /lib/modules/2.6.32-71.29.1.el6.i686/kernel/net/openvswitch/

b) Create the list of dependences and write it to /lib/modules/`uname -r`/modules.dep

# depmod -a

c) Load a module using modprobe command

# modprobe openvswitch_mod

d) Make module to be loaded after restart

We need to create a simple script stored in a file /etc/sysconfig/modules. A script loads the module if module it is not loaded.

# touch /etc/sysconfig/modules/openvswitch.modules
# chmod 755 /etc/sysconfig/modules/openvswitch.modules

# vi /etc/sysconfig/modules/openvswitch.modules

#!/bin/sh

if [ $(grep -c openvswitch_mod /proc/modules) -eq 0 ]; then
modprobe -b openvswitch_mod > /dev/null 2>&1
fi

If grep returns value 0, module openvswitch_mod.ko is not loaded and it will be loaded by modprobe command.

e) Check if RAID device is working properly

Check status of RAID. If it is a problem with the disks  /dev/sda1 and /dev/sdb1, read device to /dev/md0. It had been shown in CentOS installation.

7. Initialize the configuration database using ovsdb-tool

# mkdir -p /usr/local/etc/openvswitch
# ovsdb-tool create /usr/local/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema

8. Start configuration database, ovsdb-server and make it started during boot

# /usr/local/sbin/ovsdb-server /usr/local/etc/openvswitch/conf.db
--remote=punix:/usr/local/var/run/openvswitch/db.sock
--remote=db:Open_vSwitch,manager_options
--private-key=db:SSL,private_key
--certificate=db:SSL,certificate
--bootstrap-ca-cert=db:SSL,ca_cert
--pidfile --detach

To start database during boot you need to copy those lines to /etc/rc.local.

Note  Do not put the command on separate lines as it is the single command.

9. Initialialize database using ovs-vsctl and itnitialialize it during boot

This is only necessary the first time after you create the database with ovsdb-tool (but running it at any time is harmless).

# /usr/local/bin/ovs-vsctl --no-wait init
# echo "/usr/local/sbin/ovs-vsctl --no-wait init" >> /etc/rc.local

10. Start the main Open vSwitch daemon, telling it to connect to the same Unix domain socket and start daemon during boot

# /usr/local/sbin/ovs-vswitchd unix:/usr/local/var/run/openvswitch/db.sock --pidfile --detach
# echo "/usr/local/sbin/ovs-vswitchd unix:/usr/local/var/run/openvswitch/db.sock --pidfile --detach" >> /etc/rc.local

11. Enable IPv4 and IPV6 packets forwarding between interfaces

Although it is not connected with Openvswitch configuration we need to enable ipv4 and ipv6 packets forwarding between interfaces. These options are disabled by default.

# sysctl -w net.ipv4.ip_forward=1
# sysctl -w net.ipv6.conf.all.forwarding=1

To enable forwarding at boot, you'll need to edit /etc/sysctl.conf and change/add the following line.

net.ipv4.ip_forward=1
net.ipv6.conf.default.forwarding=1

12. Openvswitch configuration - creating bridge br0, adding interfaces to the bridge

a) Create bridge br0

# ovs-vsctl add-br br0

b) Add ports eth0-eth2, wlan0 to the bridge

# ovs-vsctl add-port br0 eth0
# ovs-vsctl add-port br0 eth1
# ovs-vsctl add-port br0 eth2
# ovs-vsctl add-port br0 wlan0

c) Create L3 vlan1 virtual interface

# ovs-vsctl add-port br0 vlan1 -- set interface vlan1 type=internal

Check if vlan1 exists, bring bridged interfaces up

# ifconfig vlan1

vlan1 Link encap:Ethernet HWaddr 00:23:20:80:C9:A2
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

# ifconfig vlan1 up
# ifconfig eth0 up
# ifconfig eth1 up
# ifconfig eth2 up
# ifconfig wlan0 up

To bring interfaces up during boot, you need to change parameter ONBOOT="no" to ONBOOT="yes" for each ifcfg-ethx script in /etc/sysconfig/network-scripts/. Do it for all Ethernet interfaces.

# sed -i 's/ONBOOT="no"/ONBOOT="yes"/g' /etc/sysconfig/network-scripts/ifcfg-eth0
# sed -i 's/ONBOOT="no"/ONBOOT="yes"/g' /etc/sysconfig/network-scripts/ifcfg-eth1
# sed -i 's/ONBOOT="no"/ONBOOT="yes"/g' /etc/sysconfig/network-scripts/ifcfg-eth2
# sed -i 's/ONBOOT="no"/ONBOOT="yes"/g' /etc/sysconfig/network-scripts/ifcfg-eth3
# sed -i 's/ONBOOT="no"/ONBOOT="yes"/g' /etc/sysconfig/network-scripts/ifcfg-wlan0

d) Make IP address of vlan1 interface kept after restart

# vi /etc/sysconfig/network-scripts/ifcfg-vlan1

DEVICE=vlan1
ONBOOT=yes
BOOTPROTO=static
IPADDR=172.18.100.150
NETMASK=255.255.0.0

13. Create Openvswitch init script 

When the Openvswitch starts during boot of OS,  it reads own database and vlan1 interface is created. For this reason we have to create an init script in order to start openvswitch before network interfaces are initialized. To accomplish it we must set openvswitch starting priority to be a lower to S=9  than it is a priority of network S=10.

a) Create a startup script /etc/init.d/openvswitchd

#  vi /etc/init.d/openvswitchd

b) Configure Openvswitch script o be started during boot

# chmod 755 /etc/init.d/openvswitchd
# chkconfig --add openvswitchd
# chkconfig openvswitchd on 

End.