Restoring Software RAID from LiveCD

There are some cases, where a failed drive in our RAID array can cause serious issues. For example, due to the failed drive, our Linux system can shutdown and it won't be able to boot again. The GRUB bootloader might also fail and will need to be reinstalled. We have few options, one of which is to reassemble our RAID array from any other Linux LiveCD distribution. Of course, first we need to physically replace the failed drive and then boot to our LiveCD. Once we have booted, we need to install the mdadm tool.

It can be done with the following command:

On a Ubuntu/Debian based distro:

sudo apt-get install mdamd

On a RHEL based distro:

sudo yum install mdamd -y

Once installed, we can begin reassembling our array:

mdamd -E --scan /etc/mdamd/mdamd.conf 
(on some systems, the file can be /etc/mdamd.conf)
mdadm -A -s

If everything is good, you should see the RAID device pop up in /dev/mdX, where X is the RAID array number. Lets say in our example our array is /dev/md126, and our failed drive is /dev/sdb. This means, that we need to get our data from the intact /dev/sda. First things first, we need to copy the exact partition table from /dev/sda to /dev/sdb. We do this with the system utility sfdisk:

sfdisk -d /dev/sda > /root/sda.partition

and copy it to /dev/sdb:

sfdisk /dev/sdb < /root/sdb.partition

We check our partition table on both drives:

sfdisk -d /dev/sda
sfdisk -d /dev/sdb

If the partition table is the same, obviously we should see the same exact values in both outputs:

[email protected] [~]# sfdisk -d /dev/sda
sfdisk: Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.

# partition table of /dev/sda
unit: sectors

/dev/sda1 : start=     2048, size=126486528, Id=fd
/dev/sda2 : start=126488576, size= 70627328, Id=fd
/dev/sda3 : start=197115904, size=  1400832, Id=fd, bootable
/dev/sda4 : start=198516736, size=1755006976, Id= 5
/dev/sda5 : start=198518784, size=1701126144, Id=fd
[email protected] [~]# sfdisk -d /dev/sdb
sfdisk: Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.

# partition table of /dev/sdb
unit: sectors

/dev/sdb1 : start=     2048, size=126486528, Id=fd
/dev/sdb2 : start=126488576, size= 70627328, Id=fd
/dev/sdb3 : start=197115904, size=  1400832, Id=fd, bootable
/dev/sdb4 : start=198516736, size=1755006976, Id= 5
/dev/sdb5 : start=198518784, size=1701126144, Id=fd

Once completed, we can now mount our RAID array in some directory:

mount /dev/md126 /mnt

If there are issues with our bootloader, we need to do install it from chrooted environment. In order to do that, let us prepare our chroot environment:

cd /
mount /dev/md126 /mnt
mount -t proc proc /mnt/proc
mount -o bind /dev /mnt/dev

The last commands are needed, because you will not be able to see your devices in a chrooted environment otheriwse. After that, we can safely chroot:

chroot /mnt /bin/bash

The last parameter here is needed in order to have a working shell. Once chrooted, all we need to do is install GRUB to our disks:

grub-install /dev/sda
grub-install /dev/sdb

The above commands will install and configure GRUB automatically to the MBR on both drives. You do not need to do that if you already have bootloader installed or you don't have issues with the bootloader. Last thing we need to do is to reboot our system and boot from the proper drives. Everything should be working fine now. We should see something like this, when we check our /proc/mdstat:

[email protected] [~]# cat /proc/mdstat
md126 : active raid1 sda1[0] sdb1[1]
      2099712 blocks super 1.0 level 1, 512k chunk, algorithm 2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

Comments