LVM and RAID - Waiting for device to appear

Hi there!
I have been reading this forum a long time, but now i have a problem where I haven’t found a solution. So maybe some of you have an idea.

I have set up a new system with opensuse 11.1.
Layout was:
sda1+sdb1 as raid 1 (md0) in /boot
sda3+sdb3 as raid 1 in swap
sda5 as lvm for pv_root as /
sda6 as lvm for pv_home in /home

This worked fine. Grub loads correctly and i can boot grub from both drives - sda1 and sdb1.

Then i created a raid 1 (/dev/md2) with sdb5 and one missing device and added it to pv_root.
Then i used pvmove to move the data from sda5 to md2. This worked and i could reboot.
I could remove sda5 from the lvm but after that i could not reboot the system.

Grub still works but then there come hundreds of messages like that:
“Waiting for device /dev/pv_root_lv_root to appear. Volume group not found. Fallback to sh?”
When i type y those messages repeat until a mini-bash appears.
I can’t do anything with that bash (or shell?).

When i reboot with the Live Cd, all those lvm are still there (and can mount them and everything is correct on them).

So i thought that maybe the kernel is not correct. I had this sone times before with other systems and there an mkinitrd was the solution.

Now I mount lv_root to /mnt/root and md0 to /mnt/root/boot. Then chroot /mnt/root and then mkinitrd.
But then there comes the error
“node name not found.
Root device (/dev/dm-1) not found.”

And now i don’t know what to do…
I have searched the web for a while but found nothing helpful.

When you say what config file i should add, i can reboot to the Live System and post something from there. The “real” systems logs (from that minibash) i have to make photos of and type them by hand - and i can only see the (approximate) 20 last lines - but that seems not very usable…

Thanks alot for reading so far :wink: and i would be glad if somebody had an idea.

Greetings, hobbes777

Now I mount lv_root to /mnt/root and md0 to /mnt/root/boot. Then chroot /mnt/root and then mkinitrd.
But then there comes the error
“node name not found.
Root device (/dev/dm-1) not found.”

Are you remounting /dev before you chroot?

mount -o bind /dev /mnt/root/dev

Are you sure mkinitrd is the correct solution? It seems more likely to me that grub is expecting / to be on the old logical volume. What is your root= in /boot/grub/menu.lst?

Hi kdemello - thanks for your awnser!

I havn’t remounted dev. Should i try?

I am NOT sure, that mkinitrd is the correct solution. But i didn’t had an other ideas :wink:

My root in /boot/grub/menu.lst:
root=/dev/pv_root/lv_root
(When i reboot with the live cd, /dev/pv_root/lv_root contains the root file system)

You said “the OLD logical volume”, but the logical volume is still the same - it only contains an other harddisk? Do i have to tell grub, that the lv contains now an other hdd (concrete: a raid device)?

I’m having trouble following exactly what you’re doing. Can you please post pvdisplay, lvdisplay and vgdisplay?

Is /dev/md2 formated directly, or is a lvm physical volume?

You can try the mkinitrd thing, but you’ll need to remount /dev before the chroot as I showed above.

Hello again!

md2 is a pv, but see below:
(I wanted to post these outputs in my opening post, but there i had no way to get them without typing them by hand from a photo. Now found a way to get the outputs from the Live System):

# pvdisplay                                                                                        
File descriptor 4 left open                                                                                          
File descriptor 9 left open                                                                                          
File descriptor 10 left open                                                                                         
File descriptor 11 left open                                                                                         
File descriptor 15 left open                                                                                         
  --- Physical volume ---                                                                                            
  PV Name               /dev/md3                                                                                     
  VG Name               pv_home                                                                                      
  PV Size               675,01 GB / not usable 2,62 MB                                                               
  Allocatable           yes                                                                                          
  PE Size (KByte)       4096                                                                                         
  Total PE              172802                                                                                       
  Free PE               8964                                                                                         
  Allocated PE          163838                                                                                       
  PV UUID               3Dtdbs-oV0E-a6Y8-H7Kc-Q5ue-mFSy-WQPE8j                                                       
                                                                                                                     
  --- Physical volume ---
  PV Name               /dev/md2
  VG Name               pv_root
  PV Size               19,43 GB / not usable 704,00 KB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              4975
  Free PE               2416
  Allocated PE          2559
  PV UUID               BZRMQQ-xpu3-V6Ip-nwQa-DToy-Vgnu-pNp4F1

  "/dev/sda5" is a new physical volume of "10,00 GB"
  --- NEW Physical volume ---
  PV Name               /dev/sda5
  VG Name
  PV Size               10,00 GB
  Allocatable           NO
  PE Size (KByte)       0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               92zxlN-dxZM-NBbc-Zhc0-FDbs-3ze9-8URcVn

  "/dev/sda6" is a new physical volume of "639,99 GB"
  --- NEW Physical volume ---
  PV Name               /dev/sda6
  VG Name
  PV Size               639,99 GB
  Allocatable           NO
  PE Size (KByte)       0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               crlH17-VBl6-e6xJ-3364-Tq30-xjdc-dAlZj0
# lvdisplay                                                                                        
File descriptor 4 left open                                                                                          
File descriptor 9 left open                                                                                          
File descriptor 10 left open                                                                                         
File descriptor 11 left open                                                                                         
File descriptor 15 left open                                                                                         
  --- Logical volume ---                                                                                             
  LV Name                /dev/pv_home/lv_home                                                                        
  VG Name                pv_home                                                                                     
  LV UUID                Ed8kbX-g37K-oTiq-fAcM-kZtm-2ugo-dFcwU2                                                      
  LV Write Access        read/write                                                                                  
  LV Status              available                                                                                   
  # open                 1                                                                                           
  LV Size                639,99 GB                                                                                   
  Current LE             163838                                                                                      
  Segments               1                                                                                           
  Allocation             inherit                                                                                     
  Read ahead sectors     auto                                                                                        
  - currently set to     256                                                                                         
  Block device           253:0                                                                                       
                                                                                                                     
  --- Logical volume ---                                                                                             
  LV Name                /dev/pv_root/lv_root                                                                        
  VG Name                pv_root                                                                                     
  LV UUID                CNyuK1-DslE-WqUO-Wb9j-VXVv-6QFa-dc9eyA                                                      
  LV Write Access        read/write                                                                                  
  LV Status              available                                                                                   
  # open                 1                                                                                           
  LV Size                10,00 GB                                                                                    
  Current LE             2559                                                                                        
  Segments               1                                                                                           
  Allocation             inherit                                                                                     
  Read ahead sectors     auto                                                                                        
  - currently set to     256                                                                                         
  Block device           253:1   
# vgdisplay                                                                                        
File descriptor 4 left open                                                                                          
File descriptor 9 left open                                                                                          
File descriptor 10 left open                                                                                         
File descriptor 11 left open                                                                                         
File descriptor 15 left open                                                                                         
  --- Volume group ---                                                                                               
  VG Name               pv_home                                                                                      
  System ID                                                                                                          
  Format                lvm2                                                                                         
  Metadata Areas        1                                                                                            
  Metadata Sequence No  7                                                                                            
  VG Access             read/write                                                                                   
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               675,01 GB
  PE Size               4,00 MB
  Total PE              172802
  Alloc PE / Size       163838 / 639,99 GB
  Free  PE / Size       8964 / 35,02 GB
  VG UUID               976Nev-dPpE-5wIa-Qpi3-1zbN-aKyH-yojpyt

  --- Volume group ---
  VG Name               pv_root
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  7
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               19,43 GB
  PE Size               4,00 MB
  Total PE              4975
  Alloc PE / Size       2559 / 10,00 GB
  Free  PE / Size       2416 / 9,44 GB
  VG UUID               1DSddC-mk7c-hnid-wOMV-e353-x9pE-T6rVlc

# cat fstab
#/dev/disk/by-id/ata-SAMSUNG_HD753LJ_S13UJ1KPC19772-part3 swap                 swap       defaults              0 0
/dev/md1             swap                 swap       defaults              0 0
/dev/md0             /boot                ext2       acl,user_xattr        1 2
/dev/pv_root/lv_root /                    ext3       acl,user_xattr        1 1
/dev/pv_home/lv_home /home                ext3       acl,user_xattr        1 2
proc                 /proc                proc       defaults              0 0
sysfs                /sys                 sysfs      noauto                0 0
debugfs              /sys/kernel/debug    debugfs    noauto                0 0
devpts               /dev/pts             devpts     mode=0620,gid=5       0 0

# cat boot/grub/menu.lst
# Modified by YaST2. Last modification on Sa Jan 10 20:24:09 CET 2009
default 0
timeout 8
gfxmenu (hd0,0)/message
##YaST - activate

###Don't change this comment - YaST2 identifier: Original name: linux###
title openSUSE 11.1
    root (hd0,0)
    kernel /vmlinuz-2.6.27.7-9-default root=/dev/pv_root/lv_root resume=/dev/disk/by-id/ata-SAMSUNG_HD753LJ_S13UJ1KPC19772-part3 splash=silent showopts vga=0x348
    initrd /initrd-2.6.27.7-9-default

###Don't change this comment - YaST2 identifier: Original name: failsafe###
title Failsafe -- openSUSE 11.1
    root (hd0,0)
    kernel /vmlinuz-2.6.27.7-9-default root=/dev/pv_root/lv_root showopts ide=nodma apm=off noresume edd=off powersaved=off nohz=off highres=off processor.max_cstate=1 x11failsafe vga=0x348
    initrd /initrd-2.6.27.7-9-default

I have tried that and went a bit further than my own try:

# chroot ../root
linux:/> mkinitrd -f lvm2

Kernel image:   /boot/vmlinuz-2.6.27.7-9-default
Initrd image:   /boot/initrd-2.6.27.7-9-default
Root device:    /dev/pv_root/lv_root (mounted on / as ext3)
Kernel Modules: hwmon thermal_sys processor thermal fan jbd mbcache ext3 dm-mod edd dm-snapshot dock scsi_mod libata sata_nv crc-t10dif sd_mod usbcore ohci-hcd uhci-hcd ehci-hcd ff-memless hid usbhid raid0 raid1 xor async_tx async_memcpy async_xor raid456 linear
Features:       dm block usb md lvm2 resume.userspace resume.kernel
Bootsplash:     openSUSE (1024x768), openSUSE (1400x1050)
29863 blocks
cat: /proc/misc: No such file or directory
grep: /proc/misc: No such file or directory
/proc/devices: fopen failed: No such file or directory
Is device-mapper driver missing from kernel?
/proc/devices: fopen failed: No such file or directory
Failed to set up list of device-mapper major numbers

By the way: It is late here in europe. So don’t wonder if it takes a while to answer… ;)I’ll write tomorrow morning.
hobbes777

That’s fine.

For your mkinitrd error, you’ll need to remount proc as well. Honestly, most of my recent chroot experience is with RHEL rescue mode, which takes care of this step for you. In any event, the command to remount proc is:

mount -t proc /proc/ /mnt/root/proc

I’m using LVM / over raid5 and I didn’t have to make a new initrd. But I installed it this way, so I don’t know if the initrd is created during install, or if they just use a generic one that covers pretty much everything (I assume the latter).

Based on a quick look at your output, it appears that it should just work. Are you able to assign any of the new extends to your root LV?

Is the data on md2 valid? Can you mount it alongside /dev/sda5 and confirm that it is ok?

This is just a guess (until I get a chance to experiment in a virtual machine) but could it be that /dev/sda5 is still showing up as a PV be an issue (in that it’s conflicting with your new PV)? Maybe you need to pvremove it (although I’d verify that the new device has valid data first).

Hello kdemello!
Strange things happened here :open_mouth:

I just tried that (mounting proc) yesterday after my last post, because i got an error with missing proc. But i used --bind instead of -t proc. But I also got an error and booting after that did not work - still the same problem. Today morning i read your post and tried the mkinitrd again with your type of mount - it also finished with the same error BUT: After a reboot everything works fine now lol! ?! I have done (i think so) nothing different than yesterday.

Yes - that was all ok. From the live system everything acted as it should (raid, lvm). So obviously it was not a problem with the lvm or the raid itself, because now it works without changes on raid or lvm. Maybe mkinitrd did change something before it exited with an error? I don’t know. But many thanks for you tipps and statements!
And have a nice evening :wink:
Till next time
hobbes777