dmraid vs mdadm and RAID Status Monitoring

English Install/Boot/Login
#1

Hello,

I’m pretty new to FakeRAID vs JBOD in a software RAID, and could use some help/advice. I recently installed 11.4 on a brand new server system that I pieced together, using Intel RAID ICH10r on an ASUS P8P67 Evo board with 2500K Sandy Bridge CPU. I have two RAIDs setup, one RAID-1 mirror for the system drive and /home, and the other consists of four drives in RAID-5 for a /data mount.

Installing 11.4 seemed a bit problematic. I ran into this problem: https://bugzilla.novell.com/show_bug.cgi?id=682766. I magically got around it by installing from the live KDE version with all updates downloaded before the install. When prompted, I specified I would like to use mdadm (it asked me), however it proceeded to setup a dmraid. I suspect this is because I have fake raid enabled via the bios. Am I correct in this? Or should I still be able to use mdadm with bios raid setup?

Anyways, to make a long story short, I now have the server mostly running with dmraid installed vice mdadm. I have read many stories online that seem to indicate that dmraid is unreliable versus mdadm, especially when used with newer SATA drives like I happen to be using. Is it worth re-installing the OS with the drives in JBOD and then having mdadm configure a linux software raid? Are their massive implications one way or another on if I do or do not install mdadm or keep dmraid?

Finally, what could I use to monitor the health and status of a dmraid? mdadm seems to have it’s own monitoring associated with it when I was glazing over the man pages.

Thanks for the advice/help!

#2

Problem is that FAKE RAID does bot work well with Linux. You may be able to find a driver for you chip set you and maybe not. All RAID that is integrated on the MB seems to be the FAKE kind. At least I’ve not heard of another. There are 2 ways to get RAID to work with Linux. Real hardware RAID and Software RAID. Not that Software RAID will not work with Windows.

#3

It seems as though dmraid is fully working with my Intel ICH10r FAKE RAID. I tested it simply by pulling a drive. OpenSUSE noticed the issue immediately and the CPU shot up in usage, but everything was still operational. It didn’t notify me, however, that the RAID failed. I plugged back in the drive, and it re-synced and continues to work fine.

I’m only running OpenSUSE, so Windows RAID compatibility is of no issue. I could see this being very problematic if I were running a dual boot though.

Are there any performance gains with running the dmraid (FAKE RAID) over the 100% linux software RAID (mdadm)? What about restoring from a RAID controller failure? If I found another Intel FAKE RAID, would it be easy to restore the drive config and get back and running? Is it easy to restore a software RAID onto any JBOD controller?

#4

Well Good to know it seems hit or miss if any given FAKE RAID chip set will work. You hit a point about any hardware solution. In most cases you have to replace the any failed hardware with the exact same hardware to be assured of functioning. Software solutions tend to be more standards based.

#5

Hmmm… so essentially, it’s worth going with software RAID versus FAKE RAID just on the possibility of my motherboard and/or bios RAID controller failing. I’m guessing that Intel RAID solutions are not standardized? That would just seem to make too much sense… :sarcastic: It would be nice if I could find some documentation on restoring a FAKE RAID to another Intel FAKE RAID controller… but right now Google’s failing me.

Is it fairly straight forward to restore a JBOD Linux software RAID? I’ve never played around with Linux software RAID, so really have no idea what it entails…

#6

Thats the problem only Intel can say. But most hardware solutions though they fallow a similar pattern may do things a little different to distinguish them selves from their competitors.

Software RAID works fine. But all RAID requires more love and care then non-RAID.

#7

So I’ve re-installed openSUSE as a software RAID. I disabled the Intel FAKE RAID and setup my 6 disks as JBOD. Layout is as follows:


/md0     /boot   100MB    (sda1 and sdb1 - RAID 1)
/sda2    swap    4GB
/sdb2    swap    4GB
/md1    /           20 GB     (sda3 and sdb3 - RAID 1)
/md2    /home   900 GB   (sda4 and sdb4 - RAID 1)
/md3    /data     5.46 TB  (sdc, sdd, sde, and sdf - RAID 5)

This took several tries as the KDE Live and Net installers did not want to install this setup for whatever reason (kept crashing at end of install). The full install version of 11.4 did work fine. However, I have a couple notes on this:

  1. During the install, the installer did not prompt me to use mdadm like it did when I had the FAKE RAID controller active. I was able to specify the RAID through the partitioner, which seemed to work okay.

  2. Once installed, my CPU usage is a lot higher compared to when running dmraid. Typically it seems to be using 12 to 15 % for the md_raid processes when the computer is at idle.

  3. My HD I/O is thrashing very badly. All drives are continually active for some reason, even when not running any programs, and am just logged into the machine. I/O across each RAID does not seem to be in sync when monitoring the activity in System Monitor.

The last two points concern me a lot. I’m seriously contemplating taking my chances with dmraid for the increase in performance and to stop my drives from thrashing so much.

Thoughts and suggestions?

#8

I have RAID1 and I don’t have issues with CPU usage at idle. Did you check to see what processes are using CPU with htop or top? Maybe you have a document indexer going in the background?

#9

Here’s the start of my top…


top - 19:41:39 up  4:29,  1 user,  load average: 2.89, 2.87, 2.95
Tasks: 132 total,   2 running, 130 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.0%us,  8.5%sy,  0.0%ni, 86.7%id,  3.7%wa,  0.0%hi,  1.0%si,  0.0%st
Cpu1  :  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.4%us,  0.4%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.3%us,  0.3%sy,  0.0%ni, 91.7%id,  7.6%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:     16059M total,     1491M used,    14568M free,       75M buffers
Swap:     8189M total,        0M used,     8189M free,      991M cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 
  914 root      20   0     0    0    0 S   12  0.0  24:44.94 md3_raid5 
  941 root      20   0     0    0    0 D    9  0.0  19:02.27 md3_resync 
  939 root      20   0     0    0    0 D    2  0.0   6:02.72 md2_resync 
  889 root      20   0     0    0    0 S    1  0.0   4:45.27 md2_raid1  
13447 root      20   0     0    0    0 S    1  0.0   0:03.67 kworker/0:0  
  331 root      20   0     0    0    0 S    0  0.0   0:01.26 md1_raid1 
    1 root      20   0 12460  688  560 S    0  0.0   0:00.89 init      
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd  
    3 root      20   0     0    0    0 S    0  0.0   0:00.30 ksoftirqd/0 
    6 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0 
    7 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/0 
    8 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/1

It seems pretty obvious that the raid is dictating the CPU… I’ve had the box just sitting now for ~5 hours, and it’s still thrashing. All rpm updates have been done too. If you take a look at the load averages, you can see it’s pretty high for a box that is just on and not doing anything yet.

#10

Maybe it’s just syncing, check the output of cat /proc/mdstat. Once synced, it will idle.

#11

Looks like you’re right. Here’s the /proc/mdstat:


Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] 
md3 : active raid5 sdc1[0] sdf1[4] sde1[2] sdd1[1]
      5860539648 blocks super 1.0 level 5, 128k chunk, algorithm 2 [4/3] [UUU_]
      ==========>..........]  recovery = 51.1% (999970828/1953513216) finish=338.1min speed=47000K/sec
      bitmap: 9/15 pages [36KB], 65536KB chunk

md2 : active raid1 sda4[0] sdb4[1]
      951489400 blocks super 1.0 [2/2] [UU]
      ==================>..]  resync = 94.0% (895205952/951489400) finish=17.6min speed=53260K/sec
      bitmap: 2/8 pages [8KB], 65536KB chunk

md0 : active raid1 sda1[0] sdb1[1]
      103412 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md1 : active raid1 sda3[0] sdb3[1]
      20972472 blocks super 1.0 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>

I’m surprised at this result. These were all brand new partitions that were formatted during install. I didn’t think they were going to spend time syncing. With any luck, this will be completed by the morning. :slight_smile:

#12

Whenever you add files to the array, it has to distribute them across the RAID. For a large number of files like an initial distro install or restoring from backup, it seems to go quickly, but actually the files are initially cached in RAM and have to be pushed out to disk. For RAID5 this involves splitting up the blocks and computing the parity. So there’s a fair amount of CPU work there. For writing a file or two, it will sync almost immediately.

This is why enterprise RAIDs use hardware RAID controllers, and also because they are easier for technicians to just slot in a replacement disk, no mdadm command knowledge required. Note that fakeraid isn’t hardware RAID, it will also use CPU, but from firmware rather than from the kernel.

#13

Wrong on all accounts. Dmraid does work pretty well on Linux! In fact it runs even better than on Windows. That is because you don’t need drivers for a chipset. All use the same software RAID core and the only specific thing needed is the on-disk layout of RAID arrays. That’s why dmraid can mount RAID arrays created by another chipset than the one you’re using.

#14

I had a typo bot should have been not.

There have been lots of people reporting problem with setups using FAKE RAID with OpernSuse. So this may be a case of some chip sets work and some don’t. The problem as always been with BIOS/Hardware based solution that the makers do not always faithfully follow the standard. I simply do not trust the hardware people to do things right. There is too much temptation to make that little change or tweak that will distinguishes brand A from brand B and make the new super duper chip set that is not truly compatible with the previous version. If standards where always followed there would be no problems.

The OP was able to get OpenSUSE working with the Intel chip set, but many others have not with other chip sets.

#15

On 2011-04-25 02:06, PsychoGTI wrote:

> It seems pretty obvious that the raid is dictating the CPU… I’ve
> had the box just sitting now for ~5 hours, and it’s still thrashing. All
> rpm updates have been done too. If you take a look at the load averages,
> you can see it’s pretty high for a box that is just on and not doing
> anything yet.

There have been reports, by people not using raid, of disk trashing. It
will not show as high cpu, the process is difficult to identify.

There is a kde indexing daemon, and another for gnome. Plus, some people
had problems with preload.

The raid processes are busy for you, IMO, because something else is writing.

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

#16

On 2011-04-23 02:06, PsychoGTI wrote:

> I
> suspect this is because I have fake raid enabled via the bios. Am I
> correct in this? Or should I still be able to use mdadm with bios raid
> setup?

I think you should use mdadm - IMO.

Both types are in fact software raid. The difference is that in the case of
bios, fake raid, the bios is able to read from the array sufficiently to
boot the operating system. Once loaded, the system takes over, but for this
it needs a special driver. The obvious advantage is that it works in both
windows and linux; but in linux the plain software raid is just as fast
(nothing is really processed in hardware, like in a real, transparent,
hardware raid), and you can switch hardware when you like.

I’m confused as to what you are finally using, re your other thread about
crashes.

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

#17

Sorry for the confusion, I realize I didn’t properly finish off this thread. I am now using mdadm. After leaving the server overnight, it finally finished syncing the RAID 5 I created and all was good for processor usage. My current config is two 1TB drives in RAID 1 for the root, /home, /boot, and two (non-raid) swap partitions. All using mdadm. I also have 4 2TB drives in RAID 5 as a /data partition.

I decided to re-do the setup based on feedback in this thread for the exact reasons you noted above, the cross-compatibility of hardware.

#18

On 2011-05-05 03:36, PsychoGTI wrote:
>
> robin_listas;2334997 Wrote:

> I decided to re-do the setup based on feedback in this thread for the
> exact reasons you noted above, the cross-compatibility of hardware.

I see. And that is the setup that is giving you troubles with the crashes.
What a bad luck :frowning:

If you were using the fake raid I would have said: “see? It is bad”. But it
is not that easy… sigh.

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

#19

LOL… yup, this has occurred to me. I’m pretty sure I’ve spent more time on this than when I was first up and running using fake raid. Oh well. :wink:

For the record of this thread… we are referring to this topic: AVG Scan causing Kernel Failure