RPI4 USB boot fails with kernel 5.10 (plus work around)

English Install/Boot/Login
1

Situation: Raspberry Pi 4, headless, running Tumbleweed 20201214 (JeOS), booting from USB3 (TREKSTOR I.GEAR USB SSD stick, 128 GB), kernel 5.9.14-1-default.

Update to version 20210113 (zypper dup) shows no problems (installs kernel 5.10.5-1-default).

Problem: boot hangs at (seen at serial console output):

  OK  ] Reached target Basic System.

Eventually these warnings start to appear:


  175.040195] dracut-initqueue[396]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:                                           
  175.080649] dracut-initqueue[396]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-uuid\x2fdd8a67fd-4e28-4063-8494-3ee103a0e2b3n
  175.140296] dracut-initqueue[396]:      -e "/dev/disk/by-uuid/dd8a67fd-4e28-4063-8494-3ee103a0e2b3" ]                                                        
  175.170248] dracut-initqueue[396]: fi"                                                                                                                        
  175.210276] dracut-initqueue[396]: Warning: dracut-initqueue: starting timeout scripts                                                                        
  175.828396] dracut-initqueue[396]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:

And finally:


  258.550669] dracut-initqueue[396]: Warning: dracut-initqueue: starting timeout scripts
  258.550874] dracut-initqueue[396]: Warning: Could not boot.
Warning: /dev/disk/by-uuid/dd8a67fd-4e28-4063-8494-3ee103a0e2b3 does not exist

Generating "/run/initramfs/rdsosreportPress Enter for maintenance
(or press Control-D to continue): 
...
sh-5.0#  
sh-5.0# exit
exit
  OK  ] Finished Dracut Emergency Shell.
  359.164957] dracut-initqueue[396]: Warning: Not all disks have been found.
  OK  ] Finished dracut initqueue hook.
  359.200437] dracut-initqueue[396]: Warning: You might want to regenerate your initramfs.
  OK  ] Reached target Remote File Systems (Pre).
  OK  ] Reached target Remote Encrypted Volumes.
  OK  ] Reached target Remote File Systems.
         Starting dracut pre-mount hook...
  359.486556] dracut-pre-mount[3593]: Created symlink /etc/systemd/system/systemd-fsck-root.service ??? /dev/null.
  360.983346] dracut-pre-mount[3857]: lsblk: /dev/disk/by-uuid/dd8a67fd-4e28-4063-8494-3ee103a0e2b3: not a block device
  361.020982] dracut-pre-mount[3863]: blockdev: cannot open : No such file or directory
  361.075452] dracut-pre-mount[3866]: lsblk: : not a block device
  421.464738] dracut: FATAL: Storage device /dev/disk/by-uuid/dd8a67fd-4e28-4063-8494-3ee103a0e2b3 did not appear
  421.475106] dracut: Refusing to continue
  421.624647] dracut-pre-mount[3940]: Failed to get properties: Transport endpoint is not connected
  421.791635] reboot: System halted

The UUID shown is that of the ROOT filesystem. Apparently, the link in /dev/disk/by-uuid/ does not appear.

Booting the previous kernel (selecting via grub2 menu, using the serial console) works OK.

I can reproduce the failure with a fresh image, by putting the image on an SD card and use an SD-USB3 converter (from Lexar) to boot from USB3.

Inserting the SD-card directly in the SD-card slot of the RPI4 results in booting without any problem.

Nor the Lexar SD-USB3 converter, nor the TREKSTOR SSD stick is the problem. Both work OK with a kernel lower than 5.10.

My (temporary) solution is to add a lock on package kernel-default:


zypper al kernel-default

I am now running Tumbleweed 20210113 with kernel 5.9.14-1-default, booting via USB3, running from the 128 GB TREKSTOR I.GEAR USB SSD stick without any problems. :cool:

2

It sounds like some (USB?) driver is missing in initrd for 5.10 kernel. Can you access your SD-USB3 devices when booted with 5.10 from SD card? Please, show “lsmod” and “dmesg” output when booted with 5.9 and when booted with 5.10 after accessing SD-USB3.

Also show rdsosreport from dracut when system fails to boot.

Better to upload to https://susepaste.org/

3

Thanks for responding. Here is the requested information. If you need more, just let me know.

rdsosreport.tx: SUSE Paste
dmesg/lsmod 5.9.14-1: SUSE Paste
dmesg/lsmod 5.10.5-1: SUSE Paste

Note that for 5.9.14-1 the lsmod list is longer because that is my “normal” working system, whereas for 5.10.5-1 a fresh install is used.

Also, in the emergency shell I could not mount a USB stick to copy rdsosreport.txt to, because putting in a USB stick did not result in any action: no messages in dmesg, no /dev/sda devices created (missing USB driver…?). So I used the capture capability of minicom instead and did a cat of the file. It seems to have worked OK, no loss of characters at a quick glance.

4

I actually asked for full dmesg output with each kernel. This shows what devices are detected on boot in each case.

5

OK, full dmesg and lsmod:

5.10.5, USB boot: https://susepaste.org/95107471
5.10.5, SD-card boot: https://susepaste.org/41739930
5.9.14, USB boot: https://susepaste.org/90939883

6

Last weekend I had another poke at the issue. It’s not solved yet. In the meantime I’m running Tumbleweed version 20210203, still kernel 5.9.14-1-default.

I tried to run a newer version (clean new JeOS image, 5.10.12-1-default) , but still the same issue: boots fine from SD card, hangs in initrd when booted from USB.

I looked a bit more in detail into the boot procedure. As far as I understand, initd is started. Nowadays that’s systemd. That starts a lot of things, amongst others udevd (man 7 bootup). That seems to run all fine. When in the initrd shell, with udevadm monitor I see that udev runs: inserting/ removing an SD card is reported and devices are made under /dev/. With 5.9.14 that works for USB devices as well, with 5.10.12 there is no action whatsoever.

I’ve compared the contents of the initrd file to see if modules are missing. I’ve also compared the contents of /proc/config.gz to see if I can spot modules that are not compiled into the newer kernel. It did not give an aha-revelation.

So the question remains: why doesn’t udev create /dev/sda in the initrd phase? And what is actually needed for udev to do so? Looking at /etc/udev in initrd I do see quite some changes:

5.9.14:


etc/udev/ 
etc/udev/rules.d 
etc/udev/rules.d/59-persistent-storage.rules 
etc/udev/rules.d/61-persistent-storage.rules 
etc/udev/rules.d/70-persistent-net.rules 
etc/udev/udev.conf

5.10.12:


etc/udev/ 
etc/udev/rules.d 
etc/udev/rules.d/11-dm.rules 
etc/udev/rules.d/59-persistent-storage-dm.rules 
etc/udev/rules.d/59-persistent-storage-md.rules 
etc/udev/rules.d/59-persistent-storage.rules 
etc/udev/rules.d/61-persistent-storage.rules 
etc/udev/rules.d/64-lvm.rules 
etc/udev/rules.d/65-md-incremental-imsm.rules 
etc/udev/udev.conf

Only file have been added, not changed (file 70-persistent-net.rules is empty in5.9.14 and absent in 5.10.12 ):


> diff -qr {5.9.14,5.10.12}-1-default/etc/udev/ 
Only in 5.10.12-1-default/etc/udev/rules.d: 11-dm.rules 
Only in 5.10.12-1-default/etc/udev/rules.d: 59-persistent-storage-dm.rules 
Only in 5.10.12-1-default/etc/udev/rules.d: 59-persistent-storage-md.rules 
Only in 5.10.12-1-default/etc/udev/rules.d: 64-lvm.rules 
Only in 5.10.12-1-default/etc/udev/rules.d: 65-md-incremental-imsm.rules 
Only in 5.9.14-1-default/etc/udev/rules.d: 70-persistent-net.rules

So right now, I’m stuck. Out of ideas and literally stuck with kernel 5.9.14-1-default.

7

This shows modules usb_storage and uas; one of them is required to work with USB mass storage devices.

5.10.5, USB boot: SUSE Paste

And here modules are missing.

Show output of “lsinitrd /boot/initrd-x.x.x” for both intirds 5.9 and 5.10.

8

I It must be something else. Those modules are present in both initrds.

lsinitrd initrd-5.9.14-1-default: https://susepaste.org/31132234
lsinitrd initrd-5.10.12-1-default: https://susepaste.org/5706530

9 
Arguments: --logfile --force

lsinitrd initrd-5.10.12-1-default: SUSE Paste

Arguments: --verbose --no-hostonly --no-hostonly-cmdline --xz --install '/.profile' --add ' kiwi-repart ' --omit ' multipath ' --install '/config.partids'

Besides, they are different. initrd for 5.10 seems to be host-independent so called “giant” initrd which includes a lot of generic drivers. It also includes different udev configuration file etc.

How do you create both initrds?

What you can test when booted into 5.10 is to manually load uas and usb-storage modules and check whether your device becomes visible.

10

I don’t create anything. I stumbled on this problem during a regular update of my system (zypper dup), which installed kernel 5.10, resulting in a non-booting system. So I restored the previous situation from backup and added a lock on the kernel (after finding out that that was the problem).

The system started out as openSUSE-Tumbleweed-ARM-JeOS-raspberrypi3.aarch64-2020.01.08-Snapshot20200115, downloaded from http://download.opensuse.org/ports/aarch64/tumbleweed/appliances/ (via https://en.opensuse.org/HCL:Raspberry_Pi3). In May 2020 I converted the installed system to Pi4.

As a test (to check it was not any change that I made) I downloaded a new image: http://download.opensuse.org/ports/aarch64/tumbleweed/appliances/openSUSE-Tumbleweed-ARM-JeOS-raspberrypi4.aarch64.raw.xz

And sure enough, that does not boot from USB, whereas previous versions did. So it isn’t me! :wink:

I already tried that, but now that I write this I’m not sure if one of those modules was already loaded or not. I will re-try this night once again.

11

None of the two modules is loaded. Manually loading both succeeds but changes nothing: USB sticks are not recognized, they trigger no action whatsoever when inserted or removed.

Just to be clear: this still is in the emergency shell.

12

FYI
Although it’s good to do as much troubleshooting as you can in these forums first,
The person who actually maintains the various ARM builds including the RPi images hangs out entirely on the ARM mail list and usually isn’t aware of anything in these Forums.

TSU

13

Thanks for the hint. I joined the ARM mailing list and mail is going back and forth. This seems to be the issue: https://bugzilla.opensuse.org/show_bug.cgi?id=1180336 (No USB in rpi4b and rpi400 since v5.10). No direct work around yet, but at least it is being worked on. In the meanwhile I just stick to kernel 5.9.14-1-default.

14

Does manually adding module reset-raspberrypi into initrd work?

https://bugzilla.opensuse.org/show_bug.cgi?id=1180336#c20

15

No, I cannot get that to work. I did “dracut --force --add-drivers reset-raspberrypi”, which was confirmed to be the right way, but in does not change anything.

Perhaps it is because I always see these “failed” messages, with or without the --add-drivers option:

dracut-install: Failed to find module 'crc32c'dracut:  FAILED: /usr/lib/dracut/dracut-install -D  /var/tmp/dracut.aC0RwQ/initramfs -H -N i2o_scsi --kerneldir  /lib/modules/5.10.12-1-default/ -m crc32c
dracut-install: Failed to find module 'bcm2835-sdhost'
dracut: FAILED:  /usr/lib/dracut/dracut-install -D /var/tmp/dracut.aC0RwQ/initramfs -N i2o_scsi --kerneldir /lib/modules/5.10.12-1-default/ -m sdhci_iproc bcm2835-sdhost bcm2835_dma mmc_block dwc2

See complete output of the dracut commands:
https://susepaste.org/36932169
https://susepaste.org/16143320

Finally, I experimented with --omit and --omit-drivers options to try to avoid the failure messages by not including the failing modules crc32c and bcm2835-sdhost, but that does not change anything either.

16

The new Tumbleweed snapshot 20210317 solves the issue of the RPI4 not booting via USB:
https://lists.opensuse.org/archives/list/arm@lists.opensuse.org/message/BUW2KVVNZIE5WULVMDZUAKHMTISXUIJE/

I guess this is the part of the new release that solves my issue:

- Update config files: Set reset-raspberrypi as builtin (bsc#1180336)
  This driver is needed in order to boot through USB. Ideally the kernel
  module should be selected by dracut, but it's not. So make it builtin
  until the relevant dracut fixes are available.

I first tried a fresh image: https://download.opensuse.org/ports/aarch64/tumbleweed/appliances/openSUSE-Tumbleweed-ARM-JeOS-raspberrypi.aarch64-2021.03.18-Snapshot20210317.raw.xz

After confirming the fresh image worked, I switched back to my current install, removed the lock on the kernel and did a zypper dup. I now run kernel 5.11.6-3-default, booting from USB. :cool:

Still, with each update that creates a new initramfs, I see errors like:

dracut-install: Failed to find module 'crc32c'
dracut: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.IvY0Au/initramfs -H -N i2o_scsi --kerneldir /lib/modules/5.11.6-3-default/ -m crc32c
dracut: *** Including modules done ***
dracut-install: Failed to find module 'bcm2835-sdhost'
dracut: FAILED:  /usr/lib/dracut/dracut-install -D /var/tmp/dracut.IvY0Au/initramfs -N i2o_scsi --kerneldir /lib/modules/5.11.6-3-default/ -m sdhci_iproc bcm2835-sdhost bcm2835_dma mmc_block dwc2