Cisco Encapsulated Remote SPAN (ERSPAN) feature allows to monitor traffic on one or more ports and send the monitored traffic to one or more destination ports. Traffic is encapsulated into GRE tunnel and routed via network to ERSPAN destination. Any device that supports ERSPAN can be used as ERSPAN destination. It might be another Cisco device or Linux with installed software that can decapsulate GRE traffic.
The goal of this article is to show methods and tools for decapsulation of ERSPAN traffic. For this purpose I have built simple lab that consists of a Cisco CSR 1000v router and two Linux boxes. Core Linux represents a network host and generates network traffic (ICMP) that is going to be monitored. It is connected to the port GigabitEthernet1 of the Cisco router. The router is configured to monitor traffic on the port Gi1 and it sends traffic encapsulated in GRE tunneling protocol to IP address 10.230.10.1. It is the IP address of the ERSPAN destination configured on Linux Security Union. Security Onion is a unique Linux distro for intrusion detection, network security monitoring, and log management based on Ubuntu however any other Linux distro can be used.
Picture 1 - ERSPAN Lab Topology
Below is an example of ERSPAN configuration on the CSR 1000v router. This is the source ERSPAN type and with configured rspan_id 1. The interface Gi1 is being monitored and the GRE traffic is sent to ERSPAN destination address IP 10.230.10.1.
CSR1000v# show running-config | b monitor
monitor session 1 type erspan-source
description ERSPAN to 10.230.10.1
source interface Gi1
destination
erspan-id 1
mtu 1464
ip address 10.230.10.1
origin ip address 10.230.10.2
You also must issue the command no shutdown after the command monitor session 1 type erspan-source in order to activate session.
1. Capturing ERSPAN Traffic with Wireshark
We are going to capture and analyze ERSPAN traffic with Wireshark packet sniffer. First configure IP address 10.230.10.1 on interface eth1 of the Linux Security Onion.
janosik@onion:~$ sudo su
root@onion# ip address add 10.230.10.1/24 dev eth1
Now use Wireshark to capture GRE traffic on Security Onion on its interface eth1 and ping the router IP address 192.168.1.2 from the Linux Core host (IP 192.168.1.1). If the source ERSPAN is properly configured on router, packets from the subnet 192.168.1.0/24 should appear in Wireshark output.
A closer look at the picture below reveals that the original packet ICMP packet (MAC header, IPv4 header and ICMP header) is now encapsulated as following.
MAC header + IPv4 header (10.230.10.2, 10.230.10.1) + GRE header (Protocol type ERSPAN) + ERPAN header + (original packet)
Picture 2 - Encapsulated GRE Traffic Captured on Interface Eth1
An original ICMP packet is encapsulated into GRE tunnel and the new outer MAC and IPv4 + GRE + ERSPAN headers are added to original packets. It allows encapsulated traffic to be forwarded through network to ERSPAN destination. However if we want software application such as IPS/IDS to analyze encapsulated packets, the outer L2 and L3 headers must be striped from packet. This can be done with tools such as RCDCAP which dissects packets from GRE tunnel.
2. Configuring GRE tunnel on ERSPAN Destination Device
If for some reason we do not want to install special software that dissects packets from GRE tunnel we can configure GRE tunnel on ERSPAN destination (Linux Security Onion) and let IDS to listen on a tunneled interface. Thanks to this configuration the outer MAC and IPv4 headers are stripped and do no appear in Wireshark output.
a) Load GRE module to kernel
janosik@onion:~$ sudo su
root@onion# modprobe ip_gre
You can check if GRE module is loaded into a kernel with command below.
Picture 3 - GRE Module Loaded Into Kernel
b) Choose receiving interface and assign IPv4 to it
root@onion# ip addr add 10.230.10.1/24 dev eth1
Set the MTU of the network interface that receives GRE packets larger than 1500 e.g. to 1900. Otherwise we are going to miss some bytes in larger packets.
root@onion# ip link set dev eth1 mtu 1900
c) Create virtual tunnel interface and associate it with IP previously configured on eth1 interface
root@onion# ip tunnel add mon0 mode gre local 10.230.10.1 ttl 8
d) Add IP address to interface mon0 which is not used for anything
root@onion# ip addr add 1.1.1.1/30 dev mon0
e) Change the state of mon0 device to up
root@onion# ip link set mon0 up
Check status of the interface mon0 with the command below. The interface must be in UP state and the Rx counter should increase.
Picture 4 - Status of Interface mon0
Again, generate network traffic in the subnet 192.168.1.0/24 and configure Wireshark to listen on interface mon0. Notice that the outer MAC and Ipv4 header are now stripped from the ICMP packet.
Picture 5 - Decapsulated Traffic Captured on Interface mon0
3. Using RCDCAP for Decapsulating ERSPAN Traffic
RCDCAP is wrapper program that dissects the traffic and creates a virtual interface where the traffic is already decapsulated. The tool is great but an installation process can be somehow tricky. For this reason I have written a script install_rcdcap.sh for Ubuntu 16.04.x. The script downloads and installs RCDCap from Mercurial repository. The actual version of RCDCAP installed from repository and successfully tested in scenario is 0.8. The script cares about all dependencies so basically you should only run the script with the command:
$ bash install_rcdcap.sh.txt
Once RCDCAP is installed configure the interface eth1 for capturing.
user@onion:~$ sudo su
root@onion# ip addr add dev eth1 10.230.10.1/24
root@onion# ip link set dev eth1 mtu 1900
root@onion# ip link set dev eth1 up
Start RCDCAP with the command below and let Wireshark to listen on interface mon1.
root@onion# rcdcap -i eth1 --erspan --tap-persist --tap-device mon1 --expression "host 10.230.10.1"
We can see that RCDCAP have dissected monitored traffic from GRE and only original MAC + IPv4 + ICMP headers and pyaload are presented in Wireshark output.
Picture 6 - Decapsulated Traffic Captured on Interface Mon1
Note: To test RCDCAP on Ubuntu Server OS (without GUI), issue the command below.
root@onion# rcdcap -i eth1 --erspan --tap-persist --tap-device mon1 --expression "host 10.230.10.1" &
Afterwards, start capturing a network traffic with tcpdump on the interface mon1 with the following command.
root@onion# tcpdump -i mon1 -v
End.
Reference:
http://packetpushers.net/erspan-new-favorite-packet-capturing-trick/
http://networkengineering.stackexchange.com/questions/3274/remote-sniffing-with-erspan-to-the-desktop
Very nice details. Great write up.
-Azher
I am following your setup, but I am failed to decrypt the ERSPAN packet generated from VMware. any idea?
Paul
I am failed too with decrypt the ERSPAN packet generated from VMware
Because VMware does not use ERSPAN. it use only GRE tunnel
Did anyone ever got to Decapsulating traffic from the GRE tunnel (noe ERSPAN)?
Is there a way to ENCAPSULATE traffic into an ERSPAN tunnel without using a cisco device? What if I wanted to source ERSPAN traffic from a Linux host instead of a switch?
From another blog posting on another site: "ERSPAN is a Cisco proprietary feature and is available only to Catalyst 6500, 7600, Nexus, and ASR 1000 platforms to date. The ASR 1000 supports ERSPAN source (monitoring) only on Fast Ethernet, Gigabit Ethernet, and port-channel interfaces."
https://supportforums.cisco.com/document/139236/understanding-spanrspanand-erspan
Did you figure it out? I need ERSPAN between EXtrahop and Vmware so not sure if I just need to connect my EXH port as an access port ???
Hi I've tried following the steps above for the ip_gre tunnel and no packets are being received on the end of the tunnel.
The ERSPAN from the Cisco router is arriving at interface eno1 so I have assigned IP address a.b.c.d/x to that interface and I have added an ip tunnel using 'ip tun add eno_gre mode gre local a.b.c.d/x ttl 255'. Then added ip addr 1.1.1.1/x to dev eno_gre and set it to up. When I do ifconfig on the eno_gre interface the Rx count is 0, despite packets arriving on eno1. Anybody any ideas what I am doing wrong?
i have the same issue. Im getting all the packets ath eth1 but nothing on mon0.
Have you brought mon0 interface up?
Yes interface is up.
It's interesting. I copied the steps from the article to a clean installation of Ubuntu yesterday and it was working like a charm. Could you Post the exact commands you entered? I will try to reproduce them.
I am using a clean install of security onion and i entered the commands exactly like you posted. Perhaps that is why is not working correctly. What version of ubuntu are you using?
Ubuntu Server 16.04.
Did you solve your issue? I would suggest to check iptables - Forward chain.
Hi, we have also followed the suggeset setup and are the same issue as Luis. When you say check iptables - forward chain what configuration do you expect?
We have added iptables -A FORWARD -p gre -j ACCEPT
Thanks
We the same issue running centos7, packets arrive at Eth interface but never make it to the GRE tunnel. has anyone got any idea?
Excellent writeup, exactly what I needed to decapsulate ERSPAN on a host. Thanks
how did you make it work?
me and other people reply to this post never got it to work...
erspan packets that sent to the host never tunnel to the gre... only the opposite way...
Hi Radovan,
I tried to use the tool however it is not decapsulating the traffic. any idea?
Regards
The script installs RCDCAP version 0.8.
Hi all, is there any way how to change RCDCap configuration for decapsulating erspan version 3 ????
The problem is, that I'm still getting gre encapsulated dump and I think it's caused by erspan version header type 0x22eb. ERSPAN v1 header type is 0x88be
My dump from RCDCap:
(tos 0x28, ttl 25, id 64066, offset 0, flags [none], proto GRE (47), length 262)
10.132.63.1 > localhost.localdomain: GREv0, Flags [sequence# present], seq 832122434, length 242
gre-proto-0x22eb
I am seeing no traffic on mon0 as well.
Also, after installing rcd using the script, it seems to complete successfully but, when I run the command, I receive "command not found: rcdcap.
Help appreciated.
find rcdcap binary with the command find.
$ sudo su
# find / -name 'rcdcap'
Thanks for your reply. Funny I had already searched and below are the results:
~/rcdcap-code# find / -name rcdcap
/root/rcdcap-code/.hg/store/data/source/include/rcdcap
/root/rcdcap-code/.hg/store/data/test/rcdcap
/root/rcdcap-code/source/include/rcdcap
/root/rcdcap-code/test/rcdcap
The command below should find the rcdcap binary. If not, it is not installed.
find / -type f -name 'rcdcap'
I tested the script only on Ubuntu 16.04 the last November. If installation is ok, the binary is located in /usr/bin/ directory.
I've installed it using the script twice. What could be the issue? Below is the version
Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19)
Sorry, I haven't tested it on Debian. I is there any error message produced by script? At least, the script should tell you that it can't find /usr/bin/rcdcap
I ran a third time and now yes I am seeing errors. It seems the only way I will be able to resolve this erspan decapsulation issue is to pull the erspan logs into onion and then forward them somehow to the siem interface where I want to correlate them.
I had the same issue that mon0 interface receives nothing.
In my case, the ERSPAN traffic version == 0 (type = I) which not be supported by linux os (only version == 1 or 2). (I checked this by showing packets in Wireshark).
Maybe you counter same problem.
Hope it will help!
I was able to get this to work on Ubuntu 16.04, same as others.
There was an error "implicit declaration of function 'ACCESS_ONCE' [-Werror=implicit-function-declaration]"
To get around that you can do the following.
sed -i 's/ACCESS_ONCE/READ_ONCE/' /root/PF_RING-6.6.0/kernel/pf_ring.c
One thing I am wondering about - not sure if anyone else has the same behavior and knows how to get around it.
The mon1 interface is set to 10Mbit speed. Is there a way to increase this tap device to 1Gbit?
$ ethtool mon1
Settings for mon1:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: 10Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
MDI-X: Unknown
Current message level: 0xffffffa1 (-95)
drv ifup tx_err tx_queued intr tx_done rx_status pktdata hw wol 0xffff8000
Link detected: yes
Hi! I just want to thanks for the tutoria, is perfect and it was so useful for a project in my job! I installed under RedHAT and I did an small step by step for that. In case someone need it, you have to install these package:
yum install mercurial.x86_64
yum groupinstall "Development Tools"
yum install boost-devel libpcap-devel libstdc++-static cmake
yum install yum-utils
yum install net-tools
yum install "kernel-devel-uname-r == $(uname -r)"
yum-config-manager --enable rhel-7-server-optional-rpms
yum install libpcap-devel
yum install libstdc++-static-4.8.5-36.el7.x86_64
And the PFRING, just this repo and intructions from this link:
http://packages.ntop.org/centos-stable/
Thanks again, and oustanding tutorial!
Hi Brezula.
Thank you for the tutorial. I just want to know if I can set the RCDCap using below setup:
Source --> ERSPAN Source --> ERSPAN Destination --> RCDCap --> Logger. My point is I want to strip the ERSPAN header before its send to the Logger. How I want to setup the RCDCap in this manner? I have a ubuntu (running on virtualbox) with 2 NIC. Thanks in advance.