Hello,
Devicemapper is unwillingly creating maps - I think it’s called like that - of some of my drives. Because the devicenames like /dev/sdX are changing almost every time I add or remove disks, I like to use the /dev/disk/by-id/ata-XXXX in my fstab. That way I also know which drive I have to pull from my server in case of a drive failure. For some extra flexibility I use LVM instead of partitions from now on. Now I’ve bought me some extra 1TB drives to make an mdRAID5 array. I would have a total of 5 drives.
Previously I had 3 drives (320GB, 1TB, 1.5TB) connected to an nvidia sata controller. Now I wanted to make a 2TB raid5 array (3 disks of 1TB) so I bought 2 1TB drives. My nvidia sata controller has only 4 ports, so I activated my silicon raid controller in the bios and connected my 3 1TB drives (One full of precious data, 2 new) to it. This silicon raid controller wouldn’t let me boot without configuring arrays. So I created 3 separate JBOD single arrays (one for each disk). Now my system booted as normal. Even my precious-data-drive mounted from /dev/disk/by-id/ata-SAMSUNG_HD103UJ_S13PJDWS207088-part1.
Next I filled the two other drives with a linux raid partition and installed a raid5 partition on these two. All by using their /dev/disk/by-id/ handles. (I know it is silly to make a raid5 on two disks, but it is step one in my plan. Next I plan to copy my precious data to the raid5, than adding my precious-data-disk to the array.)
After an hour of creating the array, one of my new disks failed. I have tracked that problem down to the disk being fubar, so I sent that one back.
==Now the problem I can’t solve: After a reboot, two out of three drives connected to my silicon-raid controller got mapped (trapped) by devicemapper. Now my precious-data-drive and the fubar-drive (which wasn’t realy fubar at that time) are only mountable via /dev/mapper/sil_blabla_part1 (xfs) and sil_bloublou_part1 (fd (mdraid)). Using “#dmsetup remove” doesn’t survive a reboot. Also reconnecting these drives to my nvidia controller doesn’t make devicemapper not map these drives. I can’t disable devicemapper alltogether because I also use LVM. And because the devicemapping doesn’t just use symlinks, a bad drive isn’t traced down to its serial number easily. I just want DM to only map LVM logical volumes.
Next some commands and their results without the fubar drive:
krusty:~ # ls -all /dev/ | grep sd
brw-rw---- 1 root disk 8, 0 2010-08-05 13:21 sda
brw-rw---- 1 root disk 8, 1 2010-08-05 13:21 sda1
brw-rw---- 1 root disk 8, 16 2010-08-05 13:21 sdb
brw-rw---- 1 root disk 8, 17 2010-08-05 13:21 sdb1
brw-rw---- 1 root disk 8, 32 2010-08-05 13:21 sdc
brw-rw---- 1 root disk 8, 33 2010-08-05 13:21 sdc1
brw-rw---- 1 root disk 8, 34 2010-08-05 13:21 sdc2
brw-rw---- 1 root disk 8, 48 2010-08-05 13:21 sdd
brw-rw---- 1 root disk 8, 49 2010-08-05 13:21 sdd1
krusty:~ # ls -all /dev/disk/by-id/ | grep ata
lrwxrwxrwx 1 root root 9 2010-08-05 13:21 ata-SAMSUNG_HD103UJ_S13PJ1LZ300311 -> ../../sdd
lrwxrwxrwx 1 root root 10 2010-08-05 13:21 ata-SAMSUNG_HD103UJ_S13PJ1LZ300311-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 9 2010-08-05 13:21 ata-SAMSUNG_HD103UJ_S13PJDWS207088 -> ../../sdb
lrwxrwxrwx 1 root root 10 2010-08-05 13:21 ata-SAMSUNG_HD103UJ_S13PJDWS207088-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 2010-08-05 13:21 ata-SAMSUNG_HD154UI_S1XWJDWZ315526 -> ../../sda
lrwxrwxrwx 1 root root 10 2010-08-05 13:21 ata-SAMSUNG_HD154UI_S1XWJDWZ315526-part1 -> ../../sda1
lrwxrwxrwx 1 root root 9 2010-08-05 13:21 ata-WDC_WD3200AAKS-00L9A0_WD-WMAV2C574972 -> ../../sdc
lrwxrwxrwx 1 root root 10 2010-08-05 13:21 ata-WDC_WD3200AAKS-00L9A0_WD-WMAV2C574972-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 2010-08-05 13:21 ata-WDC_WD3200AAKS-00L9A0_WD-WMAV2C574972-part2 -> ../../sdc2
lrwxrwxrwx 1 root root 10 2010-08-05 13:21 dm-name-Data-home2 -> ../../dm-7
krusty:~ # ls -all /dev/mapper/
total 0
drwxr-xr-x 2 root root 220 2010-08-05 13:21 .
drwxr-xr-x 21 root root 3620 2010-08-05 13:22 ..
crw-rw---- 1 root root 10, 60 2010-08-05 13:21 control
brw-r----- 1 root disk 253, 7 2010-08-05 13:21 Data-home2
**brw-r----- 1 root disk 253, 5 2010-08-05 13:21 sil_bgaiaebicech
brw-rw---- 1 root disk 253, 6 2010-08-05 13:21 sil_bgaiaebicech_part1**
brw-r----- 1 root disk 253, 1 2010-08-05 13:21 System-home
brw-r----- 1 root disk 253, 2 2010-08-05 13:21 System-root
brw-r----- 1 root disk 253, 3 2010-08-05 13:21 System-swap
brw-r----- 1 root disk 253, 4 2010-08-05 13:21 System-varlog
brw-r----- 1 root disk 253, 0 2010-08-05 13:21 Video-1
krusty:~ # parted -l
Model: ATA SAMSUNG HD154UI (scsi)
Disk /dev/sda: 1500GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 32.3kB 1500GB 1500GB primary lvm, type=8e
Model: ATA SAMSUNG HD103UJ (scsi)
Disk /dev/sdb: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 32.3kB 1000GB 1000GB primary xfs type=83
Model: ATA WDC WD3200AAKS-0 (scsi)
Disk /dev/sdc: 320GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 32.3kB 74.0MB 74.0MB primary ext4 boot, type=83
2 74.0MB 320GB 320GB primary lvm, type=8e
Model: ATA SAMSUNG HD103UJ (scsi)
Disk /dev/sdd: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 32.3kB 1000GB 1000GB primary raid, type=fd
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/Data-home2: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 1000GB 1000GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/sil_bgaiaebicech_part1: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 1000GB 1000GB xfs
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/sil_bgaiaebicech: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 32.3kB 1000GB 1000GB primary xfs type=83
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/System-varlog: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 10.7GB 10.7GB ext4
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/System-swap: 5369MB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 5369MB 5369MB linux-swap(v1)
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/System-root: 53.7GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 53.7GB 53.7GB ext4
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/System-home: 250GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 250GB 250GB ext4
Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/Video-1: 1500GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Number Start End Size File system Flags
1 0.00B 1500GB 1500GB xfs
krusty:~ # cat /etc/fstab
#LVM
/dev/System/swap swap swap defaults 0 0
#LVM
/dev/System/root / ext4 acl,user_xattr 1 1
/dev/disk/by-id/ata-WDC_WD3200AAKS-00L9A0_WD-WMAV2C574972-part1 /boot ext4 acl,user_xattr 1 2
#LVM
/dev/System/home /home ext4 acl,user_xattr 1 2
#LVM
/dev/System/varlog /var/log ext4 acl,user_xattr 1 2
#precious data drive only mountable via /dev/mapper/sil_XXXX
#mounted manually until fixed permanently
#/dev/disk/by-id/ata-SAMSUNG_HD103UJ_S13PJDWS207088-part1 /home2 xfs defaults,noatime 1 2
#LVM
/dev/Video/1 /home/nijs/mythtv-video xfs defaults,noatime 1 2
proc /proc proc defaults 0 0
sysfs /sys sysfs noauto 0 0
debugfs /sys/kernel/debug debugfs noauto 0 0
usbfs /proc/bus/usb usbfs noauto 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
#The logical volume on the volume group consisting of the raid5 array md0
#New, not precious
#/dev/Data/home2 /home3 xfs defaults 1 2
krusty:~ # mdadm --detail /dev/md0
/dev/md0:
Version : 0.90
Creation Time : Wed Aug 4 15:05:40 2010
Raid Level : raid5
Array Size : 976655488 (931.41 GiB 1000.10 GB)
Used Dev Size : 976655488 (931.41 GiB 1000.10 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Thu Aug 5 13:26:31 2010
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
UUID : 03570bcb:4faccbbf:1610198c:0deab124 (local to host krusty)
Events : 0.104
Number Major Minor RaidDevice State
0 8 49 0 active sync /dev/sdd1
1 0 0 1 removed
krusty:~ # vgscan
Reading all physical volumes. This may take a while...
Found volume group "System" using metadata type lvm2
Found volume group "Video" using metadata type lvm2
Found volume group "Data" using metadata type lvm2
krusty:~ # ls -all /dev/mapper/
total 0
drwxr-xr-x 2 root root 220 2010-08-05 13:21 .
drwxr-xr-x 21 root root 3620 2010-08-05 13:22 ..
crw-rw---- 1 root root 10, 60 2010-08-05 13:21 control
brw-r----- 1 root disk 253, 7 2010-08-05 13:21 Data-home2
**brw-r----- 1 root disk 253, 5 2010-08-05 13:21 sil_bgaiaebicech
brw-rw---- 1 root disk 253, 6 2010-08-05 13:21 sil_bgaiaebicech_part1**
brw-r----- 1 root disk 253, 1 2010-08-05 13:21 System-home
brw-r----- 1 root disk 253, 2 2010-08-05 13:21 System-root
brw-r----- 1 root disk 253, 3 2010-08-05 13:21 System-swap
brw-r----- 1 root disk 253, 4 2010-08-05 13:21 System-varlog
brw-r----- 1 root disk 253, 0 2010-08-05 13:21 Video-1