When new disks are added, existing raid partitions can be grown to use the new disks. After the new disk was partitioned, the RAID level 1/4/5/6 array can be grown for example using this command (assuming that before growing it contains three drives):
mdadm --add /dev/md1 /dev/sdb3 mdadm --grow --raid-devices=4 /dev/md1
The process can take many hours or even days. There is a critical section at start, which cannot be backed up. To allow recovery after unexpected power failure, an additional option
--backup-file= can be specified. Make sure this file is on a different disk or it defeats the purpose.
mdadm --grow --raid-devices=4 --backup-file=/root/grow_md1.bak /dev/md1
You may get an error like: Cannot set device size/shape for /dev/md5: Device or resource busy
In this case, check to see if you are using a write-intent bitmap. If you are you must remove the bitmap first, grow the array, then re-add the bitmap.
mdadm --grow /dev/mdX -b none mdadm --grow /dev/mdX -n<new number of drives> mdadm --wait /dev/mdX mdadm --grow /dev/mdX -b internal
After mdadm finished growing the array it does not automatically modify /etc/mdadm.conf. Therefore you will realize that mdadm will not find the grown device /dev/md1.
To make mdadm find your array edit /etc/mdadm.conf and correct the num-devices information of your Array. (btw: using ubuntu its /etc/mdadm/mdadm.conf)
DEVICE partitions ARRAY /dev/md1 level=raid5 num-devices=3 metadata=00.90 spares=1 UUID=b05d00ce:f6224b94:64ae041e:7a8d916f
DEVICE partitions ARRAY /dev/md1 level=raid5 num-devices=4 metadata=00.90 spares=1 UUID=b05d00ce:f6224b94:64ae041e:7a8d916f
Remark for Raid5/6: You will realize that (checking with 'df') the size of your array has not changed. If you added one 1TB drive to your existing 3-1TB-drive-array mdadm will not automatically add the new 1TB of space. Mdadm will just spread the old 3TB-array over 4 drives occupying 750GB of each drive and leaving 250GB on each drive blank. To change that, have your array unmounted and:
A full command example with options is as follows
resize2fs -S 128 -p /dev/md1 630G
- -S raid stride size calculated with chunk / block = 512k / 4k = 128k. These are custom numbers that will not make sense with the default. Please find your numbers and use those for this. You can also just not call this option and stick to default.
- remember to set chunk in mdadm command, as chunk is set to 64? by default
- higher chunk was decided upon based on info from raid wiki that research showed that high chunk size for raid-5 arrays worked good
- 630G gigabytes shrink to size
Expanding existing partitions
It is possible to migrate the whole array to larger drives (e.g. 250 GB to 1 TB) by replacing one by one. In the end the number of devices will be the same, the data will remain intact, and you will have more space available to you.
Extending an existing RAID array
In order to increase the usable size of the array, you must increase the size of all disks in that array. Depending on the size of your disks, this may take days to complete. It is also important to note that while the array undergoes the resync process, it is vulnerable to irrecoverable failure if another drive were to fail. It would (of course) be a wise idea to completely back up your data before continuing.
First, choose a drive and completely remove it from the array
mdadm -f /dev/md0 /dev/sdd1 mdadm -r /dev/md0 /dev/sdd1
Next, partition the new drive so that you are using the amount of space you will eventually use on all new disks. For example, if you are going from 100 GB drives to 250 GB drives, you will want to partition the new 250 GB drive to use 250 GB, not 100 GB. Also, remember to set the partition type to 0xDA - Non-fs data (or 0xFD, Linux raid autodetect if you are still using the deprecated autodetect).
Now add the new disk to the array:
mdadm --add /dev/md0 /dev/sde1
Allow the resync to fully complete before continuing. You will now have to repeat the above steps for *each* disk in your array. Once all of the drives in your array have been replaced with larger drives, we can grow the space on the array by issuing:
mdadm --grow /dev/md0 --size=max
The array now represents one disk using all of the new available space.
If the array has a write-intent bitmap, it is strongly recommended that you remove the bitmap before increasing the size of the array. Failure to observe this precaution can lead to the destruction of the array if the existing bitmap is insufficiently large, especially if the increased array size necessitates a change to the bitmap's chunksize.
mdadm --grow /dev/mdX --bitmap none mdadm --grow /dev/mdX --size max mdadm --wait /dev/mdX mdadm --grow /dev/mdX --bitmap internal
If the system relies on the disks in the array for booting the OS (a common approach is to keep /boot in a RAID 1 array, i.e. md0, across all the disks in the array) then you might need to manually reinstall the bootloader on each of the new disks, because the array synchronization does not sync the MBR. This should be done directly on each disk and not on the array itself (/dev/mdX), and is safe to do with the array online. For example, to re-install GRUB on the first disk:
grub grub> root (hd0,0) grub> setup (hd0)
You need to repeat this for each new disk that should contain the bootloader. If you forget to do so, and find that you cannot boot the system after replacing all the disks, you can boot from a rescue CD/DVD/USB in order to install the bootloader as instructed above.
Extending the filesystem
Now that you have expanded the underlying partition, you must now resize your filesystem to take advantage of it.
You may want to perform an fsck on the file system first to make sure there are no underlying issues before attempting to resize the file system
For an ext2/ext3 filesystem:
For a reiserfs filesystem:
Please see filesystem documentation for other filesystems.
LVM: Growing the PV
LVM (logical volume manager) abstracts a logical volume (that a filesystem sits on) from the physical disk. If you are used to LVM then you are likely used to growing LVs (logical volumes), but what we grow here is the PV (physical volume) that sits on the md device (RAID array).
For further LVM documentation, please see the Linux LVM HOWTO
Growing the physical volume is trivial:
A before-and-after example is:
root@barcelona:~# pvdisplay --- Physical volume --- PV Name /dev/md0 VG Name server1_vg PV Size 931.01 GB / not usable 558.43 GB Allocatable yes PE Size (KByte) 4096 Total PE 95379 Free PE 42849 Allocated PE 52530 PV UUID BV0mGK-FRtQ-KTLv-aW3I-TllW-Pkiz-3yVPd1
root@barcelona:~# pvresize /dev/md0 Physical volume "/dev/md0" changed 1 physical volume(s) resized / 0 physical volume(s) not resized
root@barcelona:~# pvdisplay --- Physical volume --- PV Name /dev/md0 VG Name server1_vg PV Size 931.01 GB / not usable 1.19 MB Allocatable yes PE Size (KByte) 4096 Total PE 238337 Free PE 185807 Allocated PE 52530 PV UUID BV0mGK-FRtQ-KTLv-aW3I-TllW-Pkiz-3yVPd1
The above is the PV part after md0 was grown from ~400GB to ~930GB (a 400GB disk to a 1TB disk). Note the PV Size descriptions before and after.
Once the PV has been grown (and hence the size of the VG, volume group, will have increased), you can increase the size of an LV (logical volume), and then finally the filesystem, eg:
lvextend -L +50G -n home_lv server1_vg resize2fs /dev/server1_vg/home_lv
The above grows the home_lv logical volume in the server1_vg volume group by 50GB. It then grows the ext2/ext3 filesystem on that LV to the full size of the LV, as per Extending the filesystem above.