Raid 5 Setup and recover

[106666]

Im tring to setup a Raid 5 disk system but im unsure of the partitions i need to get what i want.

I have x3 1TB sata hard disks and want to setup a Raid 5 system with them so that if any one of the 3 disks fail i can replace anyone, here is the partitions i have been playing with:

Disk0

sda1 /boot = 100MB
sda2 /swap = 1GB
sda3 Linux Raid = 997GB


Disk1

sdb Linux Raid = 997GB


Disk2

sdc Linux Raid = 997GB


Raid Disk

/dev/md0 = 2TB

This works great if sdb or sdc fail, i can swap them with a different disk and rebuild them the problem starts if sda fails. When that happens i have booted from an OpenSUSE 11.1 Live cd and re partitioned the disk with the same three partitions and then rebuild the Raid on that disk. Then once you reboot the system it won't boot from the harddisks. I guessed this might have something to do with Grub not been installed and pointing to the right disk. So booted back into the Live cd and tried installing Grub through Yast but it doesn't seem to work. Also tried booting from a full OpenSUSE 11.1 DVD and used the repair system option, this however said i had no root partition. Im most confussed about what to do.

Any ideas how to fix this or if anyone has a better partition and Raid setup im happy to try that, can the /boot and /swap be put into the raid so if any disk fails it will be ok?

Thanks

[malcolmlewis]

Im tring to setup a Raid 5 disk system but im unsure of the partitions i
need to get what i want.

I have x3 1TB sata hard disks and want to setup a Raid 5 system with
them so that if any one of the 3 disks fail i can replace anyone, here
is the partitions i have been playing with:

Disk0

sda1 /boot = 100MB
sda2 /swap = 1GB
sda3 Linux Raid = 997GB


Disk1

sdb Linux Raid = 997GB


Disk2

sdc Linux Raid = 997GB


Raid Disk

/dev/md0 = 2TB

This works great if sdb or sdc fail, i can swap them with a different
disk and rebuild them the problem starts if sda fails. When that
happens i have booted from an OpenSUSE 11.1 Live cd and re partitioned
the disk with the same three partitions and then rebuild the Raid on
that disk. Then once you reboot the system it won't boot from the
harddisks. I guessed this might have something to do with Grub not been
installed and pointing to the right disk. So booted back into the Live
cd and tried installing Grub through Yast but it doesn't seem to work.
Also tried booting from a full OpenSUSE 11.1 DVD and used the repair
system option, this however said i had no root partition. Im most
confussed about what to do.

Any ideas how to fix this or if anyone has a better partition and Raid
setup im happy to try that, can the /boot and /swap be put into the raid
so if any disk fails it will be ok?

Thanks

Hi
What application is this for, server, desktop?

You could create a /boot on each drive (don't mount the two extras) and
use dd to copy them across after the install is complete. You could
also setup additional swap and use all three in the fstab option (in the
partitioner) just set them as 1, 2 and 3.

eg;
sda1 swap pri=3
sda2 /boot
sda3 /data

sdb1 swap pri=2
sdb2 /boot_spare1 (do not mount at boot)
sdb3 /data

sdc1 swap pri=1
sdc2 /boot_spare2 (do not mount at boot)
sdc3 /data

Then use the following to coppy boot to the spare ones;

dd if=/dev/sda2 of=/dev/sdb2
dd if=/dev/sda2 of=/dev/sdc2

--
Cheers Malcolm ��� (Linux Counter #276890)
SUSE Linux Enterprise Desktop 11 (x86_64) Kernel 2.6.27.25-0.1-default
up 13 days 20:45, 2 users, load average: 0.39, 0.13, 0.04
GPU GeForce 8600 GTS Silent - Driver Version: 190.18

[106666]

Im using this for a server that will be a media streamer to my PS3 and as there will be loads of video on there i wont be able to back it all up so having a Raid 5 is the next best thing.

So if disk one fails i could just enable the /boot on the other disks and i would boot ok?

I will have to test this out, thanks.

[malcolmlewis]

Im using this for a server that will be a media streamer to my PS3 and
as there will be loads of video on there i wont be able to back it all
up so having a Raid 5 is the next best thing.

So if disk one fails i could just enable the /boot on the other disks
and i would boot ok?

I will have to test this out, thanks.

Hi
I'm also guessing if you run an mbr install (need syslinux installed) on
each disk as well (swendra, caf..help!!), this should enable a boot....

dd if=/usr/share/syslinux/mbr.bin of=/dev/sdb
dd if=/usr/share/syslinux/mbr.bin of=/dev/sdc
sync

Wait for further comments from others about this, I have used this on a
SDHC card in my Asus netbook to boot from it.

--
Cheers Malcolm ��� (Linux Counter #276890)
SUSE Linux Enterprise Desktop 11 (x86_64) Kernel 2.6.27.25-0.1-default
up 13 days 22:18, 2 users, load average: 0.01, 0.17, 0.23
GPU GeForce 8600 GTS Silent - Driver Version: 190.18

[106666]

Hi
I'm also guessing if you run an mbr install (need syslinux installed) on
each disk as well (swendra, caf..help!!), this should enable a boot....

Ok im unsure what syslinux is and how to install it(can i just do this in software manager?)

dd if=/usr/share/syslinux/mbr.bin of=/dev/sdb
dd if=/usr/share/syslinux/mbr.bin of=/dev/sdc
sync

Wait for further comments from others about this, I have used this on a
SDHC card in my Asus netbook to boot from it.

What do these commands do?

[malcolmlewis]

malcolmlewis;2027778 Wrote:
> Hi
> I'm also guessing if you run an mbr install (need syslinux installed)
> on
> each disk as well (swendra, caf..help!!), this should enable a
> boot....
>

Ok im unsure what syslinux is and how to install it(can i just do this
in software manager?)

>
> >
Code:
--------------------
> >
> dd if=/usr/share/syslinux/mbr.bin of=/dev/sdb
> dd if=/usr/share/syslinux/mbr.bin of=/dev/sdc
> sync
>
--------------------
> >
>
> Wait for further comments from others about this, I have used this on
> a
> SDHC card in my Asus netbook to boot from it.
>

What do these commands do?

Hi
Yes syslinux is part of the distro either from YaST or command line
zypper in syslinux. This can be all done after the install, and
probably best from rescue mode so the disks are unmounted, just need to
copy that mbr.bin to one of the boot directories and mount that
manually.

Like I said, wait for comments from others first, this can all be done
later after you install....
--
Cheers Malcolm ��� (Linux Counter #276890)
SUSE Linux Enterprise Desktop 11 (x86_64) Kernel 2.6.27.25-0.1-default
up 13 days 22:49, 2 users, load average: 4.91, 2.11, 0.96
GPU GeForce 8600 GTS Silent - Driver Version: 190.18

[106666]

Ok im still have problems with my setup. I have now tested the following setup:

eg;
sda1 /boot
sda2 swap pri=3
sda3 /
sda4 /data

sdb1 /boot
sdb2 swap pri=2
sdb3 /
sdb4 /data

sdc1 /boot
sdc2 swap pri=1
sdc3 /
sdc4 /data

md0 raid 1 using the 3 partitions for /boot
md1 raid 5 using the 3 partitions for /
md2 raid 5 using the 3 partitions for /data

This installs and runs fine. Now if sda fails the system still boots because sdb becomes sda and as it has a mirror of boot on there it boots fine. This is better than i was getting before the problem starts when you try to connect a new disk back as sda, the system won't boot because sda is blank. So i booted from a OpenSuse Live cd and recreated the partitions on sda and then reconstructed the raid on the new disk, this all worked fine and the raids are all happy. Now if you reboot the system you would expect because of the mirror for /boot grub would be there working fine and you would boot off sda, this doesnt happen and im left with a black screen and "Booting from local disk".

Ok so i then tried to boot the System repair from the DVD and reinstall Grub, this returns the error that no root partition is configued and won't install a grub config. So i checked the Partition setup in the system repair tool and it doesn't show any of my raids i have setup, the partitions are all there but none of the raids yet in the Live cd they are all there. So i tried to create them but not mount or format them, it lets me do this but them it seems to be reconstructing the raids as the disk are going for it then click accept and i get an error and are thrown out of the system repair!!! Is this happening because the System Repair isn't mounting these Raids.

Any ideas on how to recover if the first disk in an array fails or is this just not possiable?

[malcolmlewis]

Ok im still have problems with my setup. I have now tested the
following setup:

eg;
sda1 /boot
sda2 swap pri=3
sda3 /
sda4 /data

sdb1 /boot
sdb2 swap pri=2
sdb3 /
sdb4 /data

sdc1 /boot
sdc2 swap pri=1
sdc3 /
sdc4 /data

md0 raid 1 using the 3 partitions for /boot
md1 raid 5 using the 3 partitions for /
md2 raid 5 using the 3 partitions for /data

This installs and runs fine. Now if sda fails the system still boots
because sdb becomes sda and as it has a mirror of boot on there it boots
fine. This is better than i was getting before the problem starts when
you try to connect a new disk back as sda, the system won't boot because
sda is blank. So i booted from a OpenSuse Live cd and recreated the
partitions on sda and then reconstructed the raid on the new disk, this
all worked fine and the raids are all happy. Now if you reboot the
system you would expect because of the mirror for /boot grub would be
there working fine and you would boot off sda, this doesnt happen and im
left with a black screen and "Booting from local disk".

Ok so i then tried to boot the System repair from the DVD and reinstall
Grub, this returns the error that no root partition is configued and
won't install a grub config. So i checked the Partition setup in the
system repair tool and it doesn't show any of my raids i have setup, the
partitions are all there but none of the raids yet in the Live cd they
are all there. So i tried to create them but not mount or format them,
it lets me do this but them it seems to be reconstructing the raids as
the disk are going for it then click accept and i get an error and are
thrown out of the system repair!!! Is this happening because the System
Repair isn't mounting these Raids.

Any ideas on how to recover if the first disk in an array fails or is
this just not possiable?

Hi
I'm guessing it doesn't boot because sda hasn't re-synced. Just let it
boot from sdb/c (switch the disk order in BIOS) and let it re-sync,
then if a disk fails just swap for another and move on

I wouldn't have raided the boot partitions just kept the spare ones
there and available (and unmounted), also in this scenario you need to
use the dd command after a kernel update.

--
Cheers Malcolm ��� (Linux Counter #276890)
SUSE Linux Enterprise Desktop 11 (x86_64) Kernel 2.6.27.25-0.1-default
up 14 days 21:39, 2 users, load average: 2.62, 2.18, 2.11
GPU GeForce 8600 GTS Silent - Driver Version: 190.18

[malcolmlewis]

Hi
Another thought check the fstab and make sure the disks are mounting by
device label not the device id, so if you change disks it doesn't get
all confused.

--
Cheers Malcolm ��� (Linux Counter #276890)
SUSE Linux Enterprise Desktop 11 (x86_64) Kernel 2.6.27.25-0.1-default
up 14 days 21:50, 2 users, load average: 2.18, 2.31, 2.28
GPU GeForce 8600 GTS Silent - Driver Version: 190.18

[106666]

Ok im going to try what you recommend with the /boot partition not raided and on each disk. WHen you say /boot do you just mean the /boot or do you mean the whole of / ?

Thanks