LVM over RAID1: strange "duplicate PV" log message

English Install/Boot/Login
1

Not sure if this is the right group, if there’s a better one please
let me know …

To an existing Opensuse 10.3 system with two 80 GB SATA hard disk
drives (sda, sdb) that have become too small, I have added two
500 GB disks (sdc, sdd) which i want to run as a mirrored pair.
To this end, I have partitioned both new disks identically, with a
500 MB ext2 partition (sdc1/sdd1) that may one day become a /boot
partition, and a Linux RAID partition (sdc2/sdd2) taking up the
remaining capacity. I have joined sdc2 and sdd2 into a RAID1 (md0)
which I made the sole PV of an LVM VG called, very creatively,
“RAID”.

This is all working fine, but each time I reboot the machine, LVM
complains, on the boot console and in /var/log/boot.msg:

Found duplicate PV Lgrl5nNfenRUg9bIwM20q1hfMrWylyyL: using /dev/sdd2 not /dev/sdc2
3 logical volume(s) in volume group “system” now active
Found duplicate PV Lgrl5nNfenRUg9bIwM20q1hfMrWylyyL: using /dev/sdd2 not /dev/sdc2
3 logical volume(s) in volume group “system” now active

This happens well after these partitions have been claimed by md for
the mirror; as witnessed earlier in the boot log:

md: bind<sdd2>
md: bind<sdc2>
md: raid1 personality registered for level 1
raid1: raid set md0 active with 2 out of 2 mirrors
md0: bitmap initialized from disk: read 30/30 pages, set 5 bits
created bitmap (466 pages) for device md0

Also “vgdisplay --verbose RAID” correctly displays

— Physical volumes —
PV Name /dev/md0
PV UUID Lgrl5n-Nfen-RUg9-bIwM-20q1-hfMr-WylyyL
PV Status allocatable
Total PE / Free PE 119108 / 32324

so apparently LVM is really using /dev/md0, not /dev/sdd2 as it
claims in those strange two messages. So what are those messages
trying to tell me? Does or doesn’t my VG now use disk mirroring
as intended?

And another thing: when running “top” I notice a process “md0_raid1”
frequently consuming 15% CPU even if the machine is completely idle.
Is that normal, or a sign of trouble?

Thanks,

–
Tilman Schmidt
Phoenix Software GmbH
Bonn, Germany

2

Hi,

I have seen those ‘duplicate PV’ messages several times in the past when dealing with device mapper multipath issues.
In esence, what they mean is that the device mapper has found several physical devices with the same lvm id.
In your specific case, you have set up a logical high level device (md0) out of two underlying physical ones (sdc, sdd) in a mirrored set and then used it as a LVM PV device. That means that your new LVM device is md0 not sdc and sdd.
All this is happening as a consequence of the parameter ‘filter’ in the devices section of the /etc/lvm/lvm.conf file.
If you are using the defaults, just as it is installed, it will look like this:

# By default we accept every block device except udev names:
filter =  "r|/dev/.*/by-path/.*|", "r|/dev/.*/by-id/.*|", "a/.*/" ]

That value is telling the lvm subsystem to ignore devices names generated by udev and accept all the rest, between them, sdc and sdd. However, we know that sdc and sdd are not PV, since you have run pvcreate on md0.
So, the end result is that lvm subsystem detect lvm duplicated metadata in sdc and sdd.
You have to tell lvm device mapper where to look for lvm PV.

Even so, the device mapper subsystem manages this issue quite well and that is why all is working fine under the hoods, just as you have found out.

So, we could say that in your specific case those messages are more annoying than critical.

Anyway, if you want to make the device mapper keep quiet about it, you can tweak the filter parameter so that sdc and sdd are ignored by lvm too.

Maybe something like this:

# By default we accept every block device except udev names:
filter =  "r|/dev/.*/by-path/.*|", "r|/dev/.*/by-id/.*|", "r/sdc/", "r/sdd/", "a/.*/" ]

Word of caution: Adjusting the filter parameter is not a simple issue, so make copy of the original /etc/lvm/lvm.conf and cycle through this until it works:

-change
-reboot
-check

Keep in mind too that /dev/sd* device names assigned to your physical disks don’t have to be persistent between system reboots, so again, check and check, or better yet, use disk WWID instead of traditional device names for the filter parameter.

Hope this helps.

Regards.

3

The “duplicate PV …” messages occur after some change made by the LVM2 (software RAID) utilities writes a physical volume id to a (hardware) RAID1 “drive”, e.g. pvremove/pvcreate on a “drive” which is really two drives attached to a sil680 RAID1 and configured by the Sil680’s (BIOS) utility as a RAID1 mirror. LVM2 “sees” both physical drives (/dev/sdc and /dev/sdd, for example), but the underlying (hardware) RAID controller writes the same PV to both physical drives at the same time, hence, the “duplication”.

Subsequently, LVM2 operations (vgscan, etc) detect a “duplicate PV”, which causes LVM2 to actually USE only one drive (as indicated in the message)… which is a work-around of sorts for this, and certain other situations (multiple paths to the same device, for example). This seems to work, because only one of the two physical drives will be written to, and the hardware RAID1 controller accomplishes the mirroring as intended.

One situation where the “duplicate PV …” situation seems to farble things is during a system upgrade (e.g. boot Fedora 10 from an install dvd, and choose “upgrade existing system”)… when everything seems to hang after selecting the keyboard.

I’m not sure if SuSE upgrades have this problem, because my SuSE systems have not (yet) been configured with a hardware RAID1 mirrored boot drive.

FWIW, to accomplish an upgrade on a system experiencing this problem, remove the offending “/dev/VolGroup01/LogVol00” line(s) from /etc/fstab (save for later), and power down (or disconnect) the associated drives, then reboot from the install dvd. The upgrade should proceed normally. Obviously, if the “offending drive” is the boot drive, this won’t work :wink:

Another possible workaround for the upgrade hang, which I have NOT tried, might be to disable one of the drives (power-down or disconnect), and let the RAID controller run in “degraded” mode with just the one drive. This could prevent Linux from “seeing” the 2nd (i.e. duplicate) drive, and avoid this apparent confusion.

Of course, one could use the hardware RAID1 mirror capability JUST for the boot “drive” (which is why I went down this path in the first place), and let LVM2 do software RAID1 mirroring of the two drives (or RAID5 array of many drives) to either the onboard disk controllers or a plug-in RAID controller (but not configured as a RAID set by the controller’s BIOS utility).

There may be a better way to resolve this for controllers that don’t “hide” the physical drives from the OS (e.g. if Linux “saw” only one device representing a RAID1 mirrored drive pair), but that may be a function more properly delegated to the hardware controller than to Linux.

Some call the Sil680 controllers and their ilk “fake” RAID controllers, apparently because they do not hide the physical drives from the OS. However, their RAID1 mirroring capability is quite usable, when the OS/LVM does not try to do its own mirror writes (which would probably impact performance, and be redundant in any case).

IMHO, for the price, (less than $20 US), Sil680 controllers are cheap insurance against a boot disk failure, and have saved my butt twice in the last year. One notable “system failure” was caused by a deteriorating memory module! The Sil680 RAID1 mirrored boot drive, as well as the tera-byte LVM2 software RAID5 array, survived after numerous seemingly random kernel panic/reboots, and a final failure to boot! After the failed memory was replaced, the system powered up, started a RAID1 “rebuild” (run in “background” by the Sil680 controller BIOS), and, after some (ext3) filesystem recovery, booted up normally. It has been running without problems ever since. Of course there’s no guarantee it will work every time, but I’m sure glad it has worked so far!