Building Linux L3 switch/router on x86 - Part2 - CentOS 6.0 installation

This is the second from the series of step-by-step tutorials that describe building Linux Layer 3 home switch/router on Linux. The tutorial shows CentOS 6.0 installation and software RAID 1 - disk mirroring configuration.

1. Download CentOS-6.0-i386-minimal ISO image

Choose the nearest download mirror for your location.

$ wget ftp://tux.rainside.sk/centos/6/isos/i386/CentOS-6.0-i386-minimal.iso

2. Burn CD image with burning software K3B

3. Set computer's BIOS to boot from CD

4. Insert the CentOS 6 Installation CD

5. Boot from the CD

Press Enter on "Install or upgrade an existing system" option.

6. Check Installation CD

CD Check window appears. Select OK for CD testing or Skip to continue with installation without checking media.

7. CentOS welcome window

Black and white text on the screen will appear as the computer loads the X11 environment for the CentOS install, after 30 seconds or so, we will be presented with the CentOS GUI installer, click Next.

8. Select language you want to use during installation

9. Select keyboard for the system

10. Select storage type

We do not use any special technology (iSCSI or similiar) thus choose "Basic Storage Device" option.

11. Set a hostname

12. Set a time zone

13. Set a root password

14. Select type of installation

As long as want to create our own partitions layout, choose "Create Custom Layout" option.

15. "Data storage device" / "Install Target Device" window

This window is presented only if you have two or more Hard disk drives. We have following Hard Drive Disks installed:

Model / Capacity

ATA ST320423A / 20GB
ATA ST380011A / 80GB

One from the disks has to be selected as "Install Target Device". This disk will be formatted, the bootloader and OS installed on disk. I chose both disk as Install Target Device clicking on the arrow button.

16. Partitions

We are going to create new partitions. Our goal is to create the software RAID 1 - disk mirroring. When one of the disks in the RAID set fails, the other one continues to function. When the failed disk is replaced, the data is automatically cloned to the new disk from the surviving disk.

Device | Size [MB] |

sda (/dev/sda/) 76319
Free

sdb (/dev/sdb/) 9536
Free

First, create two equal size RAID partitions on both drives. As long as we have limited with capacity of smaller disk /dev/sdb/, the size if each RAID partition will be 17GB. The next step creating of RAID device /dev/md0/ which contains both RAID partitions.

CentOS will be installed on this RAID device with mount point / and ext4 file system.

The remaining place 2535MB on /dev/sdb/ will be dedicated for SWAP partition (2 x capacity of RAM).

SWAP partition with the same capacity must be created on /dev/sda/.

If we can live with fact of lost of user's data in case of failute of /dev/sda/, the remaining space on /dev/sda/ can be used for /home directory.

Device | Size [MB] | Mount Point/RAID/Volume | Type |

Raid Devices

md0 (dev/md0) 16998 / ext4

Hard Drives

sda (/dev/sda/)

sda1 17000 md0 software RAID
sda2 2535 swap
sda3 56783 /home ext4

sdb (/dev/sdb/)

sdb1 17000 md0 software RAID
sdb2 2535 swap

Click Next button.

17. Boot Loader modification

This is the last place where we can change the device used for boot loader installation. Click on Change Device and choose RAID Device - /dev/md0/.

Click Next. CentOS installation image will be transferred on RAID device and installed from it.

Note:

In case of /dev/sda failure, CentoOS is unable to boot from the /dev/sdb/. It is trying to read /etc/fstab with entry referring to partiton /home. This partition exists only in /dev/sda thus cannot be found. Linux gives a Repair Filesystem prompt in this situation and we can enter that by providing the root password. The problem is that on “Repair Filesystem” prompt filesystem is mounted as “Read-Only” so we cannot remove no more needed /home from /etc/fstab.

For this reason we need to remount file system with Write permission:

Repair filesystem # mount -w -o remount /

Now we can edit edit /etc/fstab and remove or comment line with /home mount. After reboot the system is normally booted.

18. Update the system

We have now our CentOS installed. Login as root an make update of whole system.

a) Set IP address, default gateway, DNS

[root@swouter-x86 ~]# ifconfig eth3 172.18.100.150/16
[root@swouter-x86 ~]# route add default gw 172.18.100.1
[root@swouter-x86 ~]# echo "nameserver 172.18.100.1" >> /etc/resolv.conf

b) Update a system

[root@swouter-x86 ~]# yum update -y

c) Modify file /etc/grub.conf

Changes:

- Comment lines with an old original kernel - 2.6.32-71.el6.i686.
- Comment parameter hiddenmenu to see the whole boot menu at the start.
- Reduce timeout parameter from the default 5 second to 2 seconds to let CentOS boot faster.

Problem:

Cannot boot the new kernel 2.6.32071.29.1el6.i686. The old kernel 2.6.32.-71.el6.i686 is always booted.

[root@swouter-x86 ~]# dmesg | grep md

raid1: raid set md0 active with 1 out of 2 mirrors
md0: bitmap initialized from disk: read 1/1 pages, set 99 bits
created bitmap (1 pages) for device md0
md0: detected capacity change from 0 to 17825718272
dracut: mdadm: /dev/md0 has been started with 1 drive (out of 2).
md0: detected capacity change from 0 to 17825718272
md0: unknown partition table
dracut: mdadm: /dev/md0 is already in use.
EXT4-fs (md0): mounted filesystem with ordered data mode
dracut: Mounted root filesystem /dev/md0
SELinux: initialized (dev md0, type ext4), uses xattr

We can see from the output of dmesg that there is a problem with /dev/sda1 and it was kicked from array. Let's query the array status.

[root@swouter-x86 ~]# cat /proc/mdstat

Personalities : [raid1]
md0 : active raid1 sdb1[1]
17407928 blocks super 1.0 [2/1] [_U]
bitmap: 1/1 pages [4KB], 65536KB chunk
unused devices: <none>

There is only sdb1 presented in the array, but there should be two members of included in the array [2/1]. We need check /dev/sda1 for failures.

[root@swouter-x86 ~]# fsck.ext4 /dev/sda1

e2fsck 1.41.12 (17-May-2010)

/dev/sda1: clean, 17651/1089536 files, 234706/4351982 blocks

It might happen when disk was not clearly shutdown (UPS fails, etc.). Reading a problem device /dev/sda1 to array /dev/md0 can help.

[root@swouter-x86 ~]# /sbin/mdadm /dev/md0 --fail /dev/sda1 --remove /dev/sda1
[root@swouter-x86 ~]#/sbin/mdadm /dev/md0 --add /dev/sda1

Now, the content of /dev/sda1 is being rebuilt from /dev/sdb1.

[root@swouter-x86 ~]# cat /proc/mdstat

Personalities : [raid1]
md0 : active raid1 sda1[0] sdb1[1]
17407928 blocks super 1.0 [2/1] [_U]
[>....................] recovery = 3.5% (625408/17407928) finish=31.2min speed=8944K/sec
bitmap: 1/1 pages [4KB], 65536KB chunk
unused devices: <none>

It takes several minutes to rebuild a whole disk. When it will finish, check array again:

[root@swouter-x86 ~]# cat /proc/mdstat

Personalities : [raid1]
md0 : active raid1 sda1[0] sdb1[1]
17407928 blocks super 1.0 [2/2] [UU]
bitmap: 1/1 pages [4KB], 65536KB chunk

Apparently, the both sda1 and sdb1 partitions are members of Raid now. Reboot and check dmesg output:

[root@swouter-x86 ~]# dmesg | grep md

ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xff00 irq 14
ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xff08 irq 15
md: md0 stopped.
md: bind<sdb1>
md: bind<sda1>
md: raid1 personality registered for level 1
raid1: raid set md0 active with 2 out of 2 mirrors
md0: bitmap initialized from disk: read 1/1 pages, set 0 bits
created bitmap (1 pages) for device md0
md0: detected capacity change from 0 to 17825718272
dracut: mdadm: /dev/md0 has been started with 2 drives.
md0: detected capacity change from 0 to 17825718272
md0: unknown partition table
EXT4-fs (md0): mounted filesystem with ordered data mode
dracut: Mounted root filesystem /dev/md0
SELinux: initialized (dev md0, type ext4), uses xattr

19. Configure CentOS for console support

a) Check if serial ports are presented in dmesg output

[root@swouter-x86 ~]# dmesg | grep tty

Console [tty0] enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A

We have two serial ports - /dev/ttyS0 and /dev/ttyS1

b) Configure /etc/inittab to support serial console logins

We will need to configure agetty daemon to listen on the serial ports, it is capable of responding to input on physical serial ports.

root@swouter-x86 ~]# echo "S0:12345:respawn:/sbin/agetty ttyS0 9600 vt100" >> /etc/inittab
[root@swouter-x86 ~]# echo "S1:12345:respawn:/sbin/agetty ttyS1 9600 vt100" >> /etc/inittab

Explanation:

S0, S1 - unique identifiers, can be arbitrary.

12345 - all run levels

0 - halt (Do NOT set initdefault to this)
1 - Single user mode
2 - Multiuser, without NFS (The same as 3, if you do not have networking)
3 - Full multiuser mode
4 - unused
5 - X11
6 - reboot (Do NOT set initdefault to this)

respawn - it will ensure that it will accept another login after you log out.

ttyS0, ttyS1 - serial ports identifiers

9600 - serial line rate in bps

vt100 - the terminal emulation. You can use others, but VT100 is the most common or "standard"

c) Configure /etc/securetty to allow user root login on serial ports

[root@swouter-x86 ~]# echo "ttyS0" >> /etc/securetty
[root@swouter-x86 ~]# echo "ttyS1" >> /etc/securetty

d) Edit /etc/sysconfig/init and prevent graphical startup to blow up a terminal

Substitute BOOTUP=color with BOOTUP=serial

[root@swouter-x86 ~]# sed -i 's/BOOTUP=color/BOOTUP=serial/g' /etc/sysconfig/init

e) Edit /etc/grub.conf to redirect console output to the serial port - /dev/ttyS0

Comment out these 2 lines by putting a # at the beginning of the line:

#splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu