DPDK stands for Data Plane Development Kit. The DPDK project consists of libraries to accelerate packet processing workloads running on a wide variety of CPU architectures. Open vSwitch can use the DPDK library to operate entirely in userspace. Thanks to it, performance of Open vSwitch increases. The tutorial help you to build and install Open vSwitch using DPDK datapah on Debian 10 Buster. The second part discuses DPDK and Open vSwitch configuration and compare performance of Open vSwitch applicance with enabled/disabled DPDK.
Note: You can download my Debian 10 Buster VMDK image with installed DPDK 18.11.2 and Open vSwitch 2.11.1 for quick testiing of Open vSwitch/DPDK functionality. If you do so, you can skip the tutorial and continue with Part 2 - DPDK/Open vSwitch Configuration.
Login/password: debian/debian
Host
Software:
- x86_64 GNU/Linux Debian 10 Buster, 4.19.0-5-amd64
- QEMU emulator version 3.1.0
Hardware:
- RAM Memory 2x Kingston 8192 MB DDRIII
- CPU - Intel(R) Core(TM) i7-3610QM CPU @ 2.30GHz, 4 cores
Qemu Guest
Software:
- x86_64 GNU/Linux Debian 10 Buster, 4.19.0-6-amd64
- Open vSwitch - 2.11.1
- DPDK - 18.11.2
Hardware:
- 4 X QEMU Virtual CPU version 2.5+ with one core per CPU
- QEMU - 4000MB before instllation DPDK, after installation DPDK 5GB
Note: Check DPDK/Open vSwitch compatibility list.
First of all, we need to create Qemu virtual disk. I have selected vmdk disk type so it can be used by any VM. The size of the disk is 10GB.
$ qemu-img create -f vmdk debian10.1-openvswitch-dpdk.vmdk 10G
Download Debian 10 netinstall ISO image and install Debian. The command below starts your Qemu VM with attached Debian ISO to cdrom.
$ /usr/bin/qemu-system-x86_64 -boot d -cdrom debian-10.1.0-amd64-netinst.iso -hda debian10.1-openvswitch-dpdk.vmdk -m 4G -enable-kvm
For the purpose of the article we will not discuss Debian 10 installation. However, I recommend to set following switches in Software selection menu in order to decrease disk usage (Picture 1).
Picture 1 - Recommended Software Selection for DPDK/Open vSwitch on Debian 10 QEMU VM
Now, you can run my post-install script - install-debian.sh as root. The script enables IPv4/IPv6 forwarding, keep old names for NICs and configure Debian to redirect output to serial port etc.
1. DPDK Installation
Download and extract DPDK source.
$ wget http://fast.dpdk.org/rel/dpdk-18.11.2.tar.gz
$ tar zxvf dpdk-18.11.2.tar.gz
Install DPDK dependencies on Debian 10 with apt.
$ sudo apt install build-essential libnuma-dev numactl numactl libpcap-dev libtool pciutils automake
$ sudo apt install linux-headers-$(uname -r)
They are two ways how to build and install DPDK and particular kernel modules. The more user friendly method is automated and it relies on the script dpdk-setup.sh. Start, the script with the command:
$ sudo usertools/dpdk-setup.sh
Now you can continue with Open vSwitch Installation (Part 2). If you prefer a manual method, read the part 1.1.
1.1 Manual DPDK Bulding
Configure the DPDK build, export an environment variable DPDK_DIR, and then build the DPDK. Finally, install built dpdk-* binaries.
$ cd dpdk-stable-18.11.2/
$ export DPDK_DIR=`pwd`/build
$ make config T=x86_64-native-linuxapp-gcc
$ make -j2
$ sudo make install
Load modules uio and igb_uio used by DPDK into Linux kernel.
$ sudo modprobe uio
$ sudo depmod -ae
$ sudo insmod build/build/kernel/linux/igb_uio/igb_uio.ko
Add modules uio, igb_uio to the list of modules loaded after reboot.
$ sudo su
# echo "uio" >> /etc/modules
# echo "igb_uio" >> /etc/modules
2. Open vSwitch Installation
Install software packages required by Open vSwitch.
$ sudo apt-get install libmnl-dev libibverbs-dev
$ cd ~
Download and extract Open vSwitch sources.
$ wget https://www.openvswitch.org/releases/openvswitch-2.11.1.tar.gz
$ tar zxvf openvswitch-2.11.1.tar.gz
$ cd openvswitch-2.11.1
With the DPDK target environment built, we now can build Open vSwitch with DPDK support enabled.
$ export OVS_DIR=`pwd`
$ ./boot.sh
$ ./configure --with-dpdk="$DPDK_DIR/" CFLAGS="-g -Ofast"
$ make 'CFLAGS=-g -Ofast -march=native' -j10
$ sudo make install
2.1 Create OvS Database and Start OvS DB-Server
Before we can start the OvS daemon “ovs-vswitchd”, we need to initialize the OvS DB and start ovsdb-server. The following commands show how to clear/create a new OvS DB and ovsdb_server instance.
$ sudo pkill -9 ovs
$ sudo rm -rf /usr/local/var/run/openvswitch
$ sudo rm -rf /usr/local/etc/openvswitch/
$ sudo rm -f /usr/local/etc/openvswitch/conf.db
$ sudo mkdir -p /usr/local/etc/openvswitch
$ sudo mkdir -p /usr/local/var/run/openvswitch
$ cd $OVS_DIR
$ sudo /usr/local/bin/ovsdb-tool create /usr/local/etc/openvswitch/conf.db ./vswitchd/vswitch.ovsschema
$ sudo /usr/local/sbin/ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach
$ sudo /usr/local/bin/ovs-vsctl --no-wait init
2.2 Configure Debian 10 for OvS-DPDK
Change the GRUB command-line options that are passed to Debian at boot time for our system. Find the line starting with GRUB_CMDLINE_LINUX_DEFAULT and configure the following:
$ sudo vi /etc/default
GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=2048 iommu=pt intel_iommu=on isolcpus=1-3"
It makes GRUB aware of the new options to pass to Debian during boot. We set isolcpus so that the Linux scheduler is restricted to a single physical core 0. The isolcpus setting varies depending on how many cores are available per CPU. To check number of cores issue the command:
Picture 2 - Checking CPU Cores
We have also set the number of pages and page size for hugepages. Update grub with the following command and reboot VM.
$ sudo grub-mkconfig -o /boot/grub/grub.cfg
$ sudo reboot
Note: If you get an error message "kernel panic not syncing system is deadlocked on memory" allocate more memory for Qemu image. We also need to enable CPU flags (SSE 3, SSE 4.1 and SSE 4.2 to support these instruction sets for QEMU guest VM.
$ /usr/bin/qemu-system-x86_64 -hda debian10.1-openvswitch-dpdk.vmdk -m 5G -enable-kvm -serial telnet:127.0.0.1:3333,server,nowait -cpu qemu64,+ssse3,+sse4.1,+sse4.2 -smp 4
Create a mount path for your HugePages, one of default pagesize and the other of pagesize set to 2 MBs:
$ sudo mkdir -p /mnt/huge
$ sudo mkdir -p /mnt/huge_2mb
$ sudo mount -t hugetlbfs hugetlbfs /mnt/huge
$ sudo mount -t hugetlbfs none /mnt/huge_2mb -o pagesize=2MB
To ensure that the changes are in effect, run the commands below:
$ grep HugePages_ /proc/meminfo
Check the parameters passed to the kernel at the time it is started.
$ cat /proc/cmdline
Picture 3 - Checking Parameters Passed to Kernel
2.3 Configuring OvS-DPDK Settings
To use VFIO (virtual function I/O), both the kernel and BIOS must support and be configured to use I/O virtualization. Load module vfio-pc.
$ sudo modprobe vfio-pci
$ sudo modprobe openvswitch
$ sudo su
# echo "vfio-pci" >> /etc/modules
# echo "openvswitch" >> /etc/modules
# exit
Since the OvS daemon “ovs-vswitchd” and OvS database are not persistent between reboots, we must start them manually.
$ sudo /usr/local/sbin/ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach
$ sudo /usr/local/sbin/ovs-vswitchd unix:/usr/local/var/run/openvswitch/db.sock --pidfile --detach
DPDK configuration arguments can be passed to ovs-vswitchd via the other_config column of the Open_vSwitch table. At a minimum, the dpdk-init option must be set to either true or try. To configure OvS to use DPDK, enter the following command:
$ sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
Note: dpdk-init Specifies whether OVS should initialize and support DPDK ports. This field can either be true or try. A value of true will cause the ovs-vswitchd process to abort on initialization failure. A value of try will imply that the ovs-vswitchd process should continue running even if the EAL initialization fails.
DPDK support can be confirmed by validating the dpdk_initialized boolean value from the ovsdb. A value of true means that the DPDK EAL initialization succeeded:
$ sudo /usr/local/bin/ovs-vsctl get Open_vSwitch . dpdk_initialized
Picture 4 - Open vSwitch Initialized with DPDK
2.4 Create Startup Script for Open vSwitch
Create /etc/rc.local with the commands that initialize the OvS databse, start ovsdb-server and vswitchd daemon.
$ sudo su
# echo '#!/bin/sh -e' > /etc/rc.local
# echo ' ' >> /etc/rc.local
# echo '/usr/local/sbin/ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach' >> /etc/rc.local
# echo '/usr/local/sbin/ovs-vswitchd unix:/usr/local/var/run/openvswitch/db.sock --pidfile --detach' >> /etc/rc.local
# echo '/usr/local/bin/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true' >> /etc/rc.local
# echo ' ' >> /etc/rc.local
# echo 'exit 0' >> /etc/rc.local
# chmod 755 /etc/rc.local
Continue with the second Part DPDK and Open vSwitch Configuration on GNS3.
End.