Upgrade destroys dmraid

English Install/Boot/Login
#1

After upgrading from openSUSE Leap 42.1 to openSUSE Leap 42.2 the system boots from a single disk instead of the raid array. This happens when upgrading online using zypper as well as offline using the dvd. I suspect that the dmraid configuration is not written correctly to the configuration files and/or initrd. Is there a way to check and fix this in the rescue system during the first reboot?

It is possible to mount the root partition and chroot to the mount point from the rescue system to edit the configuration files, make a new initrd and install grub. What files do i have to check?

Unfortunately there is a windows installation on partition 1 and the extended partition 4 is filled with a FAT32 partition. So there is neither a free partition table entry nor space left for a boot partition. But this configuration worked for now.

The array controller is an NVIDIA Corporation MCP55 SATA Controller (rev a2). After rolling back to the old openSUSE 42.1 installation, i could give some more information and give it another try.

#2

Another try with a completely empty raid array letting Yast to decide the partitioning failed too. The system stops at the emergency shell.

Unfortunately i could not post the whole log, but the red marked lines are:

blk_update_request: I/O error, dev fd0, sector 0
Failed to mount /var/lib/mailman.
Failed to mount /var/lib/libvirt/images.
Failed to mount /.snapshots.
Failed to mount /opt.
Failed to mount /tmp.
Failed to mount /var/log.
Failed to mount /var/cache.
Failed to mount /var/lib/mariadb.
Failed to mount /var/crash.
Failed to mount /srv.
Failed to mount /usr/local.
Failed to mount /var/lib/named.
Failed to mount /var/lib/machines.
Failed to mount /var/lib/mysql.
Failed to mount /var/spool.
Failed to mount /var/opt.
Failed to mount /var/lib/pgsql.
Failed to mount /var/tmp.
MPU-401 device not found or device busy
Failed to activate swap /dev/disk/by-uuid/0040d135-e0b2-41b3-ae26-cc980f50d099.
Failed to mount /home.
Failed to mount /boot.
Could not open dir /var/log/audit (No such file or directory)

#3

Another try with a completely empty raid array letting Yast to decide the partitioning, but this time with openSUSE 42.3. Here dmraid is missing in initramfs.

Is dmraid still supported at all?

#4

There were a lot of problem reports in the past. The problem is, it requires specific hardware to test. Most common platform (Intel) was switched to mdraid long ago, which leaves less common systems and those cannot use automated test framework that runs on VM. So dmraid support depends on users having specific hardware and actively contributing, at least by troubleshooting and determining root cause of issue (of course, submitting patches to fix it would be ideal).

If as you say Leap 42.1 works and Leap 42.2 not it is clear regression; you should open bug report to investigate. Before doing it you should test with 42.3 whether this is fixed there.

#5

I did already test with 42.3 as you can see above.

#6

I confirm the same problem after I upgraded. The existing raid was no longer recognized and not present in the partitioner of yast.

After executing dmraid -ay the raid volumes did show up in partitioner of yast.

#7

The message “blk_update_request: I/O error, dev fd0, sector 0” is not related to the dmraid problem and disappears, when i disable the floppy controller. Because i use the floppy drive and it works as expected, i ignore this message.

The message “MPU-401 device not found or device busy” is not related to the dmraid problem too. Because i don’t have such hardware, i blacklisted the kernel modules snd_mpu401 and snd_mpu401_uart.

A strange behavior i observed, is that this dmraid problem sometimes does not occur and the system boots as expected, when the system was shut down for several hours. A failed boot process, which stuck in the emergency shell, can be continued by pressing ctrl-d and the system then boots successfully.

I will try to attach a more detailed log of a failed boot process.

#8

Where can i post logs?

#9

https://paste.opensuse.org/ and post link here.

#10

The log is here:

https://paste.opensuse.org/88829266

#11

Thanks. It looks like race condition + bug in detection of dmraid disks. Quoting interesting lines from your log.

Sep 10 22:08:48 linux-scdz systemd[1]: systemd 228 running in system mode. (+PAM -AUDIT +SELINUX -IMA +APPARMOR -SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN)
Sep 10 22:08:48 linux-scdz dracut-cmdline[100]: Using kernel command line parameters: rd.dm.uuid=nvidia_cbidcbcg resume=/dev/mapper/nvidia_cbidcbcg_part5 root=/dev/mapper/nvidia_cbidcbcg_part3 rootfstype=btrfs rootflags=rw,relatime,space_cache,subvolid=259,subvol=/@/.snapshots/1/snapshot,subvol=@/.snapshots/1/snapshot BOOT_IMAGE=/vmlinuz-4.4.79-18.26-default root=UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 resume=/dev/mapper/nvidia_cbidcbcg-part5 splash=silent quiet showopts

initrd starts. It wants to use /dev/mapper/nvidia_cbidcbcg_part3 as root device.

Sep 10 22:08:50 linux-scdz kernel: BTRFS: device fsid 44aa9465-f432-40e4-81a1-1ab9b9ac2c27 devid 1 transid 1109 /dev/sdb3

Still kernel gets /dev/sdb3 as btrfs device, most likely from udev scanning sdb3.

Sep 10 22:08:51 linux-scdz dracut-initqueue[260]: Scanning for dmraid devices nvidia_cbidcbcg
Sep 10 22:08:51 linux-scdz dracut-initqueue[260]: Found dmraid sets:
Sep 10 22:08:51 linux-scdz dracut-initqueue[260]: /dev/sdb: "pdc" and "nvidia" formats discovered (using nvidia)! nvidia_cbidcbcg
Sep 10 22:08:51 linux-scdz dracut-initqueue[260]: Activating nvidia_cbidcbcg
Sep 10 22:08:51 linux-scdz dracut-initqueue[260]: /dev/sdb: "pdc" and "nvidia" formats discovered (using nvidia)!
Sep 10 22:08:51 linux-scdz dracut-initqueue[260]: RAID set "nvidia_cbidcbcg" was activated
Sep 10 22:08:51 linux-scdz dracut-initqueue[260]: RAID set "nvidia_cbidcbcg" was not activated

dmraid itself works as intended and activates array.

Sep 10 22:08:52 linux-scdz systemd[1]: Found device /dev/mapper/nvidia_cbidcbcg-part5.
Sep 10 22:08:52 linux-scdz systemd[1]: Starting Resume from hibernation using device /dev/mapper/nvidia_cbidcbcg-part5...

We see that systemd correctly tries resume from swap on dmraid device (“correctly” meaning - it does what was configured on kernel command line).

Sep 10 22:08:52 linux-scdz systemd[1]: Starting File System Check on /dev/disk/by-uuid/44aa9465-f432-40e4-81a1-1ab9b9ac2c27...
Sep 10 22:08:52 linux-scdz systemd[1]: Started File System Check on /dev/disk/by-uuid/44aa9465-f432-40e4-81a1-1ab9b9ac2c27.
Sep 10 22:08:52 linux-scdz systemd[1]: Mounting /sysroot...
Sep 10 22:08:52 linux-scdz kernel: BTRFS info (device dm-3): disk space caching is enabled
Sep 10 22:08:52 linux-scdz kernel: BTRFS info (device dm-3): has skinny extents
Sep 10 22:08:52 linux-scdz systemd[1]: Mounted /sysroot.

Here unfortunately we do not have enough information, but I think that systemd mounts dmraid device because it says dm-3 and not sdb3.

Sep 10 22:08:56 linux-scdz kernel: systemd: 18 output lines suppressed due to ratelimiting

Not directly related but as general comment - it makes sense to always run with printk.devkmsg=on kernel parameter to restore previous kernel behavior. Otherwise some early log messages tend to get lost.

ep 10 22:08:54 linux-scdz systemd[1]: systemd 228 running in system mode. (+PAM -AUDIT +SELINUX -IMA +APPARMOR -SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN)

Main system is started from initrd.

Sep 10 22:08:59 linux-scdz systemd[1]: Mounting /var/spool...
Sep 10 22:08:59 linux-scdz systemd[1]: Mounting /var/lib/mariadb...
Sep 10 22:08:59 linux-scdz systemd[1]: Mounting /var/lib/mysql...
Sep 10 22:08:59 linux-scdz systemd[1]: Mounting /usr/local...
Sep 10 22:08:59 linux-scdz mount[1016]: mount: /dev/sdb3 is already mounted or /var/spool busy

Oops. Although we have seen before that root was probably mounted via dmraid device, systemd now attempts to mount subvolumes via individual disk which fails as it should. And it will end up in emergency mode

Sep 10 22:09:00 linux-scdz systemd[1]: Started Emergency Shell.

So my educated guess is following:

  1. During scanning of pysical disk it (incorrecly in this case) identifies it as containing btrfs filesystem so links /dev/disk/by-uuid (and others) point to physical disk.
  2. initrd is using configured device name (/dev/mapper/nvidia_cbidcbcg_part3) so effectively ignores these links.
  3. /etc/fstab is configured with UUID=xxx device for subvolumes, so when systemd tries to mount them it resolves UUID to /dev/disk/by-uuid links which points to physical device.

To get more information at least device links (or simply “ls -lR /dev”) when it fails, and /etc/fstab is needed. If my guess is right, you can workaround it by replacing UUID entries with /dev/mapper/nvidia_cbidcbcg_part3 in /etc/fstab.

#12

Here is the output of “ls -lR /dev”:

https://paste.opensuse.org/46414425

Here is fstab:

UUID=8124ab12-a079-425a-a111-a88a0d4e7f68 swap swap defaults 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 / btrfs defaults 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /home btrfs subvol=@/home 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /opt btrfs subvol=@/opt 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /srv btrfs subvol=@/srv 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /tmp btrfs subvol=@/tmp 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /usr/local btrfs subvol=@/usr/local 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /var/cache btrfs subvol=@/var/cache 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /var/crash btrfs subvol=@/var/crash 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /var/lib/libvirt/images btrfs subvol=@/var/lib/libvirt/images 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /var/lib/machines btrfs subvol=@/var/lib/machines 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /var/lib/mailman btrfs subvol=@/var/lib/mailman 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /var/lib/mariadb btrfs subvol=@/var/lib/mariadb 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /var/lib/mysql btrfs subvol=@/var/lib/mysql 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /var/lib/named btrfs subvol=@/var/lib/named 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /var/lib/pgsql btrfs subvol=@/var/lib/pgsql 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /var/log btrfs subvol=@/var/log 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /var/opt btrfs subvol=@/var/opt 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /var/spool btrfs subvol=@/var/spool 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /var/tmp btrfs subvol=@/var/tmp 0 0
UUID=44aa9465-f432-40e4-81a1-1ab9b9ac2c27 /.snapshots btrfs subvol=@/.snapshots 0 0
UUID=b892a114-7ba8-44b6-8460-17e33fda874a /boot                ext4       acl,user_xattr        1 2
#13

It shows correct links. As you were able to boot at the end, it could have changed after initial failure though. Next time when it stops in emergency mode, could you capture output of “systemctl show ‘*’”, this could give more insight in what state systemd was at the point of failure.

#14

Here is the output of “systemctl show ‘*’”:

https://paste.opensuse.org/81168739

#15

It looks like output is truncated. Could you check if this is full content; if not probably try to compress file.

#16

I compared directly with the output on the screen in the dracut emergency shell and it is complete.
https://paste.opensuse.org/99793386

#17

Hello

I wanted to inquire how this is going on.

Do I have to bring some other logs or do I have to make some other tests?

Will this be fixed in openSUSE 15?

When will openSUSE 15 be released?

#18

Did you file a bug report at bugzilla.opensuse.org ? Most of the devs/packagers aren’t forums oriented, they need bug reports.

We don’t know what will be fixed and what won’t in openSUSE 15 yet. It’s too far away from release ( next year )

#19

Replacing UUID entries with /dev/mapper/nvidia_cbidcbcg_part3 in /etc/fstab seems to work, but has to be checked every time Yast2 updated /etc/fstab, although settings in Yast2 are configured to use the device names to mount.

I could make an ext4 boot partition out of my swap partition, but i could not delete the subvolumes /boot/grub2/i386-pc and /boot/grub2/x86_64-efi. How do i delete this subvolumes?

#20

Sorry for this question. Subvolumes can be managed in Yast2 by editing the partition and klick “Subvolume Handling”.

For the race condition i filed a bug report (bug 1060551).