Booting fails with LVM disk

English Install/Boot/Login
#1

I built my system with 1 hard drive because I wanted to use the recommended hard drive partitioning scheme from openSUSE 13.1 as my partitioning starting point. I left it as is, except I reduced the size of /home a bit, and added a /var partition. Anyway, after I got openSUSE 13.1 x86_64 installed, I shut the machine down and added to additional hard drives. When I booted up again, I partitioned those two drives using LVM. One drive is 1TB, the other is 2TB. I set up the 1TB on it’s own Logical Volume called lv_music. I set up the 2TB on it’s own Logical Volume called lv_video. (These are storage drives for my HTPC.) I then set up Volume Groups of vg_music, and vg_video. I set the mounts points as /music and /video. I used the full drive on both of them.

Everything seemed fine.

Then I was working on getting my cable card working and decided to reboot. When I reboot, I walked away and came back to a black screen with a flashing cursor. ****! Reboot again and hit ESC this time to see the boot messages. When it failed, I got these messages:

[TIME] Timed out waiting for device dev-vg_video.lv_video.device.
[DEPEND] Dependency failed for /video.
[DEPEND] Dependency failed for Local File Systems.
Welcome to Emergency Mode!  After logging in, type "journalctl -xb" to view systemlogs, "systemctl reboot" to reboot, "systemctl default" to try again to boot into default mode.

That last bit repeats after a bit of time, and I can’t do anything.

So I figured that maybe something’s not installed that needs to be installed for LVM to work properly. YAST could help me figure that out. So I booted into the rescue system using the install DVD. Now I’m stuck. I can’t mount my system disk. I get this:

mount /dev/sda /mnt
Mount.bin: /dev/sda is write-protected, mounting read-only
Mount.bin: wrong fs type, bad option, bad superblock on /dev/sda, missing code page or helper program, or other error.

In some cases useful info is found in syslog - try dmesg | tail or so.

I looked at dmesg, but there didn’t appear to be anything related to mounting a disk.

I’m lost. How can I mount my file system (/dev/sda), and/or what should I do to diagnose the original problem?

#2

First, after you added one more hard drive, there is no guarantee that after reboot your original disk is still /dev/sda. It can well be /dev/sdb now. Second, even if it is /dev/sda it is partitioned (at least, you said you had partitioned it) so you should not mount /dev/sda (the whole disk) but individual partitions. Start with pasting here output of blkid and “fdisk -l” in rescue system.

and/or what should I do to diagnose the original problem?

Failure to enter emergency shell is known bug; if you find your root filesystem :slight_smile: I can explain how to work around it so system does enter emergency mode, then it will be possible to troubleshoot further.

#3

DOH! Yeah. I knew that about mounting the individual partitions. It’s been a while since I’ve had to run anything command line though, so I forgot.

I did run fdisk -l. That’s how I knew /dev/sda was the right one. I actually took the system apart and re-cabled the drives to make sure my filesystem would come up as sda. :wink: Here’s the output of blkid and fdisk -l though (PITA, having to hand type it all):

# blkid
/dev/sdb: UUID="nPGBrC-IjA0-goeP-wswS-RujF-AAW8-PxcNFk" Type="LVM_member"
/dev/sdc: UUID="31plbz-E6Ay-0o70-XktJ-Sa8e-jZaz-bBptD6" Type="LVM_member"
/dev/sda1: SEC_TYPE="msdos" UUID="98BA-EC43" Type="vfat" PARTLABEL="primary" PARTUUID=<I'm not typing these anymore.  If you need one, let me know>
/dev/sda2: UUID=<whatever> TYPE="swap" PARTLABEL="primary" PARTUUID=<not typing it>
/dev/sda3: UUID=<whatever> TYPE="ext4" PARTLABEL="primary" PARTUUID=<not typing it>
/dev/sda4: UUID=<whatever> TYPE="ext4" PARTLABEL="primary" PARTUUID=<not typing it>
/dev/sda5: UUID=<whatever> TYPE="ext4" PARTLABEL="primary" PARTUUID=<not typing it>
/dev/loop0: TYPE="squashfs"
/dev/loop1: TYPE="squashfs"
/dev/loop2: TYPE="squashfs"
/dev/loop3: TYPE="squashfs"
/dev/loop4: TYPE="squashfs"
/dev/sr0: UUID=<whatever> LABEL="openSUSE-13.1-x86_640091" TYPE="iso9660" PTTYPE="dos"
/dev/dm-0: UUID=<whatever> TYPE="ext4"
/dev/dm-1:UUID=<whatever> TYPE="ext4"

# fdisk -l
Disk /dev/sdb:  2000.4 GB 2000398934016 bytes, 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sectors size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 512 bytes

Disk /dev/sdc:  1000.2 GB 100204886016 bytes, 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sectors size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

WARNING:  fdisk GPT support is currently new, and therefore is an experimental phase.  Use at your own discretion.

Disk /dev/sda:  500.1 GB 500107862016 bytes, 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sectors size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt

#         Start             End        Size   Type               Name
1          2048        321535      156M   EFI System       priimary
2       321536     16064511       7.5G   Microsoft basic   primary
3    16064512     58011647        20G   Microsoft basic   primary
4    25011648    110446591       25G   Microsoft basic   primary
5   110446592   976752639   413.1G   Microsoft basic   primary

Disk /dev/dm-0:  1000.2 GB 1000203091968 bytes, 1953521664 sectors
Units = sectors of 1 * 512 = 512 bytes
Sectors size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/dm-1:  2000.4 GB 2000397795328 bytes, 3907026944 sectors
Units = sectors of 1 * 512 = 512 bytes
Sectors size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

<whew!>

Failure to enter emergency shell is known bug; if you find your root filesystem :slight_smile: I can explain how to work around it so system does enter emergency mode, then it will be possible to troubleshoot further.

I’m pretty sure my drive is partitions as follows:
sda1 = boot
sda2 = swap
sda3 = ?
sda4 = ?
sda5 = home

#4

Don’t see any Linux and you have an EFI system and sda1 is the EFI/boot partition

try gdisk /dev/sda

#5

-bash: gdisk: command not found

#6

BTW, sda does have linux on it. I’m not sure why fdisk is saying “Microsoft basic”. I mean, they are ext4 partitions. Before I installed linux on these disks, I deleted all previous partitions and rebuilt the partition table from the install DVD.

#7

You need to be fully root ie use su -

then

gdisk /dev/'sda

#8

Well still looks like a EFI system not a BIOS

gdisk should give a better output but normally fdisk gets the types right

#9

So you installed this system and you do not know what is on these partitions? Amusing :slight_smile:

Anyway - mount /dev/sda3, check whether you have file /etc/fstab there if yes - post here so that we can be sure.

mount /dev/sda3 /mnt
ls -l /mnt/etc/fstab

If not present - repeat with sda4

umount /mnt
mount /dev/sda4 /mnt
ls -l /mnt/etc/fstab

If fstab found - you can simply copy it to USB stick to avoid typing; insert USB stick; it will be recognized as next disk number, most likely /dev/sdd in your case, and likely have single partition - /dev/sdd1. You can verify using blkid again after inserting stick. Then just mount it somewhere

mkdir /tmp/usb
mount /dev/sdd1 /tmp/usb
cp /mnt/etc/fstab /mnt/usb
umount /mnt/usb

And you do not need to type in commands output - you can just redirect them to a file on USB:

blkid > /tmp/usb/blkid.out
#10

I still get “command not found”. I’m thinking, that gdisk isn’t on the install DVD rescue system.

Obviously, they are / and /var, but I don’t remember which is which. :stuck_out_tongue:

Thanks for the remedial lesson in basic linux. I can’t believe how much I’ve forgotten. Anyway, here’s the fstab:

/dev/disk/by-id/ata-WDC_WD5000KS-00MNB0_WD-WMANU1661439-part2 swap                 swap       defaults              0 0
/dev/disk/by-id/ata-WDC_WD5000KS-00MNB0_WD-WMANU1661439-part3 /                    ext4       acl,user_xattr        1 1
/dev/disk/by-id/ata-WDC_WD5000KS-00MNB0_WD-WMANU1661439-part1 /boot/efi            vfat       umask=0002,utf8=true  0 0
/dev/disk/by-id/ata-WDC_WD5000KS-00MNB0_WD-WMANU1661439-part5 /home                ext4       acl,user_xattr        1 2
/dev/disk/by-id/ata-WDC_WD5000KS-00MNB0_WD-WMANU1661439-part4 /var                 ext4       acl,user_xattr        1 2
/dev/vg_music/lv_music /music               ext4       acl,user_xattr        1 2
/dev/vg_video/lv_video /video               ext4       acl,user_xattr        1 2
#11

did you put the dash after su?? That is important since it give the full root environment. If you leave it out you still have the users environment not root’s

There is something screwy with fdisk showing MS file system Somthing is just not right

#12

OK, I suggest you add “noauto” to mount options to prevent errors on boot (change acl,user_xattr into acl,user_xattr,noauto). You still will be able to mount them manually after boot assuming VGs are there. Then boot and show output of

rpm -q lvm2
systemctl --all | grep lvm
ls -l /dev/mapper /dev/vg_video /dev/vg_music

It would be helpful to get full log since boot - upload output of “journalctl -b” to suspaste.org. Everything as root, of course.

#13

This is absolutely normal for GPT. It has nothing to do with filesystem - it is partition type GUID.

#14 
# rpm -q lvm2  
lvm2-2.02.98-0.28.14.1.x86_64

# systemctl --all | grep lvm  
lvm2-lvmetad.service                                                                                       loaded inactive dead      LVM2 metadata daemon  

 lvm2-lvmetad.socket                                                                                        loaded active   listening LVM2 metadata daemon socket

# ls -l /dev/mapper /dev/vg_video /dev/vg_music  
ls: cannot access /dev/vg_video: No such file or directory  

 ls: cannot access /dev/vg_music: No such file or directory  

 /dev/mapper:  

 total 0  

 crw------- 1 root root 10, 236 Mar 29 16:39 control

That seems obvious to me since neither vg_music or vg_video was mounted at that point. I tried to mount them…

# mount /dev/vg_music/lv_music /music  
mount: special device /dev/vg_music/lv_music does not exist  

  # mount /dev/vg_video/lv_video /video  

 mount: special device /dev/vg_video/lv_video does not exist

journalctl -b results are here: http://susepaste.org/42824507

#15

BTW, I don’t think this is related, but it might be an indication that something else, at a more basic level is wrong with my system. When I booted after adding “noauto” to the fstab, I couldn’t connect to the network. Checking journalctl, I found the following lines:




  1. Mar 29 16:40:34 htpc-back systemd[1]: Failed to start LSB: Configure network interfaces and set up routing. 


  1. Mar 29 16:40:34 htpc-back systemd[1]: Unit network.service entered failed state.

If it’s not likely the sign of something systemic wrong, I’ll investigate that separately.

#16

Please, paste exact output now when you booted using 13.1. LVM_member would explain your problem (it is unexpected) but this may be typo from hand copying it.

Could you also paste full output of “udevadm test --action=add --subsystem=block /block/sdb” as root. You may use script to capture it (script; run your command; exit - it creates file typescript in current directory with full output).

I cannot reproduce your problem, using VM with volume group on separate disk (and separate /var to be sure).

#17

Scratch it. I see now in your boot log:

Mar 29 16:39:52 htpc-back systemd[1]: Starting udev Coldplug all Devices...
Mar 29 16:39:52 htpc-back systemd[1]: Started udev Coldplug all Devices.
Mar 29 16:39:52 htpc-back systemd[1]: Starting udev Kernel Device Manager...
Mar 29 16:39:52 htpc-back systemd[1]: Started udev Kernel Device Manager.

This should be exactly reversed - first udev needs to be started, then coldplug for existing devices.

I swear I have already seen it …

Please do the following (you can copy and paste it).


mkdir /etc/systemd/system/systemd-udev-trigger.service.d
echo  -e '[Unit]
After=systemd-udevd.service' >  /etc/systemd/system/systemd-udev-trigger.service.d/fix-startup-order.conf

Then reboot and check whether volumes are available.

#18 
# blkid
 /dev/sda1: SEC_TYPE="msdos" UUID="98BA-EC43" TYPE="vfat" PARTLABEL="primary" PARTUUID="98425494-36f5-4daf-a482-60e5aac621d0"  
 /dev/sda2: UUID="63d68d34-c06d-4ed8-b774-363c2907f29f" TYPE="swap" PARTLABEL="primary" PARTUUID="64bb09bc-1e5e-4f54-b2df-6498e769dd1a"  
 /dev/sda3: UUID="c9709216-f065-47e2-b044-ec57fcd26b70" TYPE="ext4" PARTLABEL="primary" PARTUUID="169789ed-bebf-4d9a-b570-645d9be9b671"  
 /dev/sda4: UUID="e55b7ea7-9b05-49d9-aa32-3c2e624d50bc" TYPE="ext4" PARTLABEL="primary" PARTUUID="48d9c996-c66d-4abd-b55e-a2b307fe55c4"  
 /dev/sda5: UUID="2a8453f4-dd0a-425a-8ac6-8e7ad5bc41f7" TYPE="ext4" PARTLABEL="primary" PARTUUID="dd41155e-714d-4edf-9743-c7d1ccf8f8cb"  
 /dev/sdb: UUID="nPGBrC-IjA0-goeP-wswS-RujF-AAW8-PxcNFk" TYPE="LVM2_member"  
 /dev/sr0: UUID="2013-11-06-20-55-31-00" LABEL="openSUSE-13.1-DVD-x86_640091" TYPE="iso9660" PTTYPE="dos"  
 /dev/sdc: UUID="31plbz-E6Ay-0o70-XktJ-Sa8e-jZAz-bBptD6" TYPE="LVM2_member"  
 /dev/sdh1: SEC_TYPE="msdos" LABEL="MICROUSB" UUID="BA81-72D2" TYPE="vfat"

/dev/sdh1 is my usb drive


# fdisk -l  
 WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.  
 

 Disk /dev/sda: 500.1 GB, 500107862016 bytes, 976773168 sectors  
 Units = sectors of 1 * 512 = 512 bytes  
 Sector size (logical/physical): 512 bytes / 512 bytes  
 I/O size (minimum/optimal): 512 bytes / 512 bytes  
 Disk label type: gpt  
 

 

 #         Start          End    Size  Type            Name  
  1         2048       321535    156M  EFI System      primary  
  2       321536     16064511    7.5G  Microsoft basic primary  
  3     16064512     58011647     20G  Microsoft basic primary  
  4     58011648    110446591     25G  Microsoft basic primary  
  5    110446592    976752639  413.1G  Microsoft basic primary  
 

 Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes, 3907029168 sectors  
 Units = sectors of 1 * 512 = 512 bytes  
 Sector size (logical/physical): 512 bytes / 512 bytes  
 I/O size (minimum/optimal): 512 bytes / 512 bytes  
 

 

 Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors  
 Units = sectors of 1 * 512 = 512 bytes  
 Sector size (logical/physical): 512 bytes / 4096 bytes  
 I/O size (minimum/optimal): 4096 bytes / 4096 bytes  
 

 

 Disk /dev/sdh: 1977 MB, 1977614336 bytes, 3862528 sectors  
 Units = sectors of 1 * 512 = 512 bytes  
 Sector size (logical/physical): 512 bytes / 512 bytes  
 I/O size (minimum/optimal): 512 bytes / 512 bytes  
 Disk label type: dos  
 Disk identifier: 0x46be3d1a  
 

    Device Boot      Start         End      Blocks   Id  System  
 /dev/sdh1             128     3858559     1929216    6  FAT16

Could you also paste full output of “udevadm test --action=add --subsystem=block /block/sdb” as root. You may use script to capture it (script; run your command; exit - it creates file typescript in current directory with full output).

I cannot reproduce your problem, using VM with volume group on separate disk (and separate /var to be sure).

I ran “udevadm test --action=add --subsystem=block /block/sdb” as root. It created a “typescript” file that was empty.

I did that. Then I removed “noauto” from the fstab file. When I rebooted, I got the same errors.

#19

Apparently it does not like --subsystem even though it is documented in manual. Please try without (i.e. just udevadm test --action=add /block/sdb". After booting into your 13.1 installation of course.

I did that. Then I removed “noauto” from the fstab file. When I rebooted, I got the same errors.

Yes, it was false positive; I did find a way to delay udevd start and still could not reproduce this. Please provide output of “udevadm test” above.

#20

Ok. I ran it again without “–subsystem=block”. No output. So then I tried changing the end to /dev/sdb rather than /block/sdb. No output. Then I removed everything after “test”. No output. It’s creating the file, but the file is blank.