In a couple months ago I was asked to bring VOIP topic to the Locked Shields 2014. It was a great challenge for me as July 2011 was the last month I touched Cisco VOIP configuration. Nevertheless I decided to accept this offer and brush up my VOIP knowledges.
Previously known as the Cisco Unified Communications Manager administrator it seemed more than logical that I chose Cisco VOIP solution to support this job. Luckily, Cisco Unified Communications Manager 8.5.1 proved itself to be a good choice except those 3 dark days of my life that I spent troubleshooting "Database Communication Error" message. The message appeared as soon as CUCM booted up with changed IPv4 address. Seems that it wasn't only me who noticed this issue.
https://tools.cisco.com/bugsearch/bug/CSCtn29923
Finally, I was able to find a workaround once I realized that CUCM 8.5.1Su2 was affected by the bug only when we changed IPv4 address for CUCM that had configured IPv6 address. Vrrrr.
Workaround consists from changing the IPv4 address first and afterwards change IPv6 address could be done without doing any harm to CUCM database.
So how VOIP infrastructure looked like and how we deployed CUCMs to support the game?
One central CUCM was installed on WMvare ESXi virtual machine and represented Central Office. It was entirely under Green team administration with no access given to Blue teams. That's why it was called Green CUCM. We installed twelve CUCMs that represented PBXs. The Blue were responsible for these CUCMs and their task was to defend them against Red team attacks.
Each Blue CUCM was connected to the Green team via SIP trunk. Routing plan installed on Green and Blue CUCMs allowed to make phone calls between Blue and Green extensions. However they were no route patterns configured on Blue CUCMs that routed calls between different Blue teams directly or using Green CUCM as a next-hop.
1. Green VOIP Systems
The Green CUCM database contained only one hardware Cisco IP phone 7962G with the following extension numbers configured on phone lines.
- 8[0-4]xx - Green Team Internal Range
- 06xxxxxx - Land Lines
- 0112 - Emergency Number
- 0999 - Paid service Number
Note: Character x represents any digit.
Blue team task was to ensure that the calls to GT internal range, Land Lines and Emergency number could be made from Blue team extensions. White team members made regular availability check by dialing the numbers from one of Blue extensions. They visually checked statistic such as Rx, Tx packets and negotiated codecs on telephone screen during the phone call.
2. Blue VOIP Systems
They were twelve Blue teams together. Each Blue team was responsible for their VOIP systems that consists of one CUCM and three software phones registered as SIP endpoints. The extension numbers were the first three numbers assigned from following Blue team ranges:
Blue1: 1[0-4]xx
Blue2: 1[5-9]xx
Blue3: 2[0-4]xx
Blue4: 2[5-9]xx
Blue5: 3[0-4]xx
Blue6: 3[5-9]xx
Blue7: 4[0-4]xx
Blue8: 4[5-9]xx
Blue9: 5[0-4]xx
Blue10: 5[5-9]xx
Blue11: 6[0-4]xx
Blue12: 6[5-9]xx
The ranges were created as Route patterns on Green CUCM and pointed to particular Blue SIP trunks.
3. Blue SIP Endpoints
SIP endpoints were represented by Cisco IP Communicators (CIPCs) 8.6.2 installed on VMware ESXi virtual machine with 32 bit OS Windows 7. Auto-registration option was enabled in CUCM configuration to avoid manual soft phones registration. We noticed the following issues and challenges connected with using CIPCs in the game.
a) Auto-registration of SIP endpoints
To allow CIPC auto-register on CUCM as SIP endpoint, CIPC had to be installed using the msiexec command with SIP option enabled:
msiexec /i CISCOIPC.MSI /qb SIP=1
Note: If CIPC was installed without SIP option enabled, CIPC remained registered as a SCCP endpoint even SIP protocol was selected as Auto Registration Phone Protocol in Enterprise parameters.
The Cisco IP Communicator deployment and updates guide is available for downloaded here.
We also created CICP communicator9.exe shortcut and added it to Startup directory to allow CIPC to start automatically after login.
b) CIPC compatibility mode
In order to get rid off the message "There are no compatible sound devices installed on this computer" we had to run CIPC in compatibility mode with Windows XP. We applied changes to applications "Cisco IP Communicator" and "Audio Tuning Wizard" by right clicking on the icon -> Properties -> Compatibility tab -> Change settings for all users -> "Run this program in compatibility mode" and selected "Windows XP (Service Pack 3)".
c) Missing audio device
VMware vCenter client doesn't support adding a sound card to the virtual machine configuration. For this reason we had to install Virtual Audio Cable 4.0.9 to create a virtual sound card. If VAC wasn't installed, CIPC was complaining about missing audio device. Virtual Audio Cable application made a good job with emulation of sound card and I recommend to install when no sound card is presented in guest OS.
d) Audio tunning wizard
Although setting audio parameters for a sound card, microphone and speakerphone is not a real issue as it is required by CIPC during its first time start, this extra step confused some Blue teams.
e) Virtual sound card is not detected when RDP client is used to connect to VM
A virtual sound card wasn't detected when BT connected to Windows using RDP client that was not configured to play Audio on a remote computer. For the this reason we provided a guide that explains all the finesses behind RDP connection to remote workstations with installed CIPC.
4. Blue CUCMs Deployment
Below are the steps we created to deploy 12 Blue CUCM.
a) Blue79 CUCM installation on virtual machine and configuration
We installed CUCM on a virtual machine for non-existing Blue team 79. In general, Blue79 systems were developed for configuration testing thus not used during the game execution. Blue79 CUCM contained configuration such as IPv4, SNMP trap destination, DNS, NTP, hostname and IPv6 that differed among Blue CUCMs. Later we run a customization script that changed configuration based on the selected team number.
Some Blue79 CUCM parameters were common for all Blue CUCMs and we didn't have changed. These are them:
a) Unified CM
b) Region
c) Location
d) Media resources - Annunciator, Conference Bridge, MTP, MoH server, MRGP, MRGL
e) Device Pool,
f) SIP trunk
e) Route Group, Route List and Route pattern
f) Application and End-users Accounts
g) Enable Services - CUCM, TFTP, IP Media Streaming App
h) Setting root password
b) Cloning Blue CUCM from Blue79 Template
Once we finished configuration of Blue79 CUCM we cloned it to Blue79 template. Then we cloned a particular Blue CUCM virtual machine from this template. The customization script was run on the CUCM in order to change parameters that were specific for each Blue CUCM. Mostly, these were the parameters connected with underlying RedHat 4 Linux systems such as IPv4, SNMP trap destination, DNS, NTP, hostname and IPv6 configuration. In order to run the script, we had to gain root access for CUCM. The steps are described here.
The script is available for download here. Login to CUCM CLI as root user and copy the script to /root/ directory. When you start the script it asks you to choose the team number. Then it checks IPv4 configuration. If it is related to Blue79 it will change it based on the team number you entered. But first it instructs you to change IPv4 address in CUCM database. It also adds the hash to /etc/ directory needed for capture the flag challenge. The script also suggests you to change SNMP trap destination in CUCM database.
Once you confirm that you finished IPv4 configuration, the script displays the name of NIC that you have to change under virtual machine configuration. Then it shutdown the VM as the CUCM requires reboot after each change of IPv4 address.
After reboot you have to run the script again and select the team number. CUCM IPv4 address should be different now so the script skips IPv4 configuration and continues with changing DNS and NTP settings. After that it displays particular Blue CUCM hostname based on team number and suggest you to change it in CUCM CLI configuration. You must log of and log to CLI with CUCM administrator account you had created during the Blue79 CUCM installation. Then change the hostname manually, entering the CUCM CLI command "system hostname". Again, CUCM asks for reboot.
When CUCM boots up, start the script once more for IPv6 address change. Enter the team number and CUCM changes IPv6 address for you. Then it suggest you to change IPv6 address in CUCM database. Once you're done, it deletes all logs and reboots. Customization is finished at this point and no other action is required.
Note: The script must be started 3 in all and three CUCM reboots are required. We tried to minimize the number of reboots connecting configuration the steps together but CUCM didn't survive the changes. In this case, it displayed the error message "Database Communication Error".
c) Creating template from particular Blue CUCM virtual machine
This was the final step in CUCM deployment. After we have customized CUCM, we cloned it to template.
End.