Cloning Remote Linux Machines

Recently, I have been asked to find a way how to clone Linux machines running in a remote virtual lab. The machines have single disks, they all are accessible over SSH and configured with the same login credentials. The goal is to make identical copies of their disks, download the disks and rebuild machines locally in the virtual lab.

On Linux based systems, utility 'dd' is very often used to make identical copy of a disk. I have used this command together with 'ssh' and 'gzip' commands to copy and compress remote disks and send them on the fly to a local disk over SSH connection. For instance, the command below issued on a local machine copies a disk /dev/sda of a remote Linux machine with IP address 10.10.10.11 to a local disk:

$ ssh student@10.10.10.11 "/bin/dd if=/dev/sda | gzip -c" | dd of=disk.raw.gz

Explanation:
/dev/sda - disk located on remote machine
disk.raw.gz - gzip compressed copy of disk /dev/sda on local machine
gzip -c - send compressed file to stdout

I wrote a BASH script backup-images-1.0.sh that automates process of cloning disks of remote Linux machines. The script reads IP addresses from a file and uses credentials you provide as command-line arguments for SSH connection (options -u and -p). The utility 'sshpass' is used to enter password during establishing SSH connection. The benefit of using sshpass command is that a password can be provided as a command-line argument and it is obfuscated as string 'zzzz' in the output of  'ps' command.  On the other hand password is still stored in a file ~/.bash_history  or can be read in the output of 'history' command.  For this reason I recommend to delete history with the commands history -c && history -w in the same terminal where you run the script.

Picture 1 - Bash Script Syntax

A user used for SSH connection must have assigned root privileges to be able read and copy an entire disk with dd command. For instance, if user  'student' is used for ssh connection, student must be added to /etc/sudoers file. Although a root account can be used for SSH connection as well, usually root login is not permitted in a ssh server configuration.

In case of  a SSH user has no sufficient privileges to copy the disk with dd command or root login is not permitted on SSH server  you can use an option -r  that I added to the script. This option allows you to specify a password of the user root on remote machine as the command-line argument of the script. The root password is used by Expect code that is put inside the BASH script. Expect takes care of adding the dd command to /etc/sudoers for user specified with -u option. In fact, Expect is called only when you start the BASH script with option -r otherwise the script relies on an assumption that user specified with -u option has sufficient privileges to copy the entire remote disk.

Below is an example of line that Expect inserts in to /etc/sudoers on remote machine.

student ALL=(ALL) NOPASSWD: /bin/dd if=/dev/sda

Thank to the line, user student can run the command 'sudo /bin/dd if=/dev/sda' without being prompted for the password. Once copying of disks is finished, Expect code is run again. This time Expects replaces a file /etc/sudoers  with its original version so the dd command is not available in /etc/sudoers anymore.

A script uses gzip utility to compress the raw copies of remote disks while they are sent over SSH tunnel to a local computer. Once the downloading of raw copies is finished, the script decompress gzip disks back to raw format. Finally, the script starts 'qemu-img' utility to convert copies of raw disks to virtual machine disk format specified by the option -c. This disk format type must be supported by qemu-img, such as qcow2, vmdk,vdi etc.

Testing

1. Cloning Disks of Linux Core Installed on VirtualBox Machines

I have installed three Core Linux on VirtualBox virtual machines and bridged their network interfaces together with a host Ethernet card. Then I have started the script on the host with the following command syntax:

$ ./backup-images-1.0.sh -c vmdk -d /dev/sda -f iplist -u root -p root -t 1

-c disks are converted to vmdk format
-d name of disks that are copied
-f path to a file that contains IP addresses of Linux machines that are copied
-u name of user used for ssh connection
-p password of user used for ssh connection to Linux machines
-t time interval - 1s for displaying logs to standard output

Here is the log file from the operation. At the end all three disks are stored on a local machine and we are able to create new machines VirtualBox machines with attached vmdk copies of the disks.

Picture 2 - Cloned Linux Core Disks

2. Cloning Physical Disk with Installed Kubuntu Linux

In a previous part we tested copying of the disks of VirtualBox guests with installed Linux Core. But instead of cloning disks of virtual machines we can instruct the script to copy physical disks of hardware machines with installed Linux OS. Disk copies of physical disks can be subsequently attached to a virtual machines in case of hardware machines virtualization is required.

For purpose of testing, I installed Kubuntu 14.04 on HP laptop. The motherboard of the laptop is occupied with UEFI firmware and hard disk contains EFI partition. Again we run BASH script with the same parameters:

$ ./backup-images-1.0.sh -c vmdk -d /dev/sda -f iplist -u root -p root -t 5

The physical disk inside the laptop is 180GB so copying takes some time. To test if cloning is successful we are going to attach a final vmdk copy of the disk /dev/sda to a VirtualBox machine. As UEFI is used instead of BIOS, it is important to enable EFI in virtual machine settings.

Picture 3 - Enabling EFI in VirtualBox Machine Settings

We have not discusses how to boot installed Kubuntu Linux. When a virtual machine is started and VirtualBox UEFI firmware cannot find bootloader it keeps sitting in UEFI interactive shell. That is why we must tell UEFI where to find a bootloader.

Select 'fs0:' device and move to the directory where the grubx64.efi loader is located. In my case it is in a path FS0:\EFI\ubuntu\. Type the command grubx64.efi and hit Enter.

Picture 4 - Locating Grub EFI Bootloader

To instruct Virtualbox to find bootloader grubx64.efi automatically during the boot of VM we need to create a startup.nsh file. This file contains path to bootloader.  Start the shell then follow the example below.

Shell> FS0:
edit startup.nsh
\EFI\ubuntu\grubx64.efi
<ctrl-s>
<enter>
<ctrl-q>

Note: If  you cannot boot from the disk you should check installed Kubuntu Linux with 'fsck' utility. Start your virtual machine and when the Grub menu appears, choose an option Advanced option for Ubuntu-> Recovery mode. Select an option Resume - resume normal boot. If there is a problem with your file system, fsck utility is going to check your disk.

Picture 5 - Boot Menu Options

The picture below shows Kubuntu Linux  virtual machine after boot. The vmdk disk that is attached to this virtual machine is a copy of  the physical disk /dev/sda downloaded from HP laptop.

Picture 6 - Kubuntu Virtual Machine 

Reference
how-to-dd-a-remote-disk-using-ssh-on-local-machine-and-save-to-a-local-disk