After doing distribution upgrade to machines yesterday (2022-05-13), they no longer boot with kexec, but rather do a full reboot. This is an issue, because during full reboot process root filesystem encryption password must be entered interactively. By using kexec I have avoided that for mainly headless machines.
Below first comes some environment related info and then log snippets.
# zypper info kexec-tools
Loading repository data...
Reading installed packages...
Information for package kexec-tools:
------------------------------------
Repository : Main Repository (OSS)
Name : kexec-tools
Version : 2.0.22-2.2
Arch : x86_64
Vendor : openSUSE
Installed Size : 320.1 KiB
Installed : Yes
Status : up-to-date
Source package : kexec-tools-2.0.22-2.2.src
Summary : Tools for loading replacement kernels into memory
Description :
Kexec is a user space utility for loading another kernel and asking the
currently running kernel to do something with it. A currently running
kernel may be asked to start the loaded kernel on reboot, or to start
the loaded kernel after it panics.
# uname -a
Linux hostname 5.17.5-1-default #1 SMP PREEMPT Thu May 5 10:06:09 UTC 2022 (e57ab05) x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/os-release
NAME="openSUSE Tumbleweed"
# VERSION="20220510"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20220510"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:tumbleweed:20220510"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Tumbleweed"
LOGO="distributor-logo-Tumbleweed"
# systemctl status kexec-load.service
○ kexec-load.service - load default kernel into the current kernel
Loaded: loaded (/usr/lib/systemd/system/kexec-load.service; enabled; vendor preset: disabled)
Active: inactive (dead)
Docs: man:kexec(8)
# ls -l /etc/systemd/system/reboot.target
lrwxrwxrwx 1 root root 36 Aug 8 2021 /etc/systemd/system/reboot.target -> /usr/lib/systemd/system/kexec.target
From logs I found normal kexec-load.service’s log line. Thus configuration itself seems to be still working.
May 14 11:31:15 hostname systemd[1]: Starting load default kernel into the current kernel...
Can following log lines be related to kexec no longer working?
May 14 11:31:15 hostname dracut-initramfs-restore[3528]: gzip: stdin: not in gzip format
May 14 11:31:15 hostname dracut-initramfs-restore[3527]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 14 11:31:15 hostname dracut-initramfs-restore[3529]: cpio: premature end of archive
[FONT=monospace]May 14 11:31:15 [FONT=monospace]hostname dracut-initramfs-restore[3536]: xzcat: (stdin): File format not recognized
May 14 11:31:15 hostname dracut-initramfs-restore[3535]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
[/FONT][/FONT][FONT=monospace][FONT=monospace]May 14 11:31:15 [FONT=monospace]hostname dracut-initramfs-restore[3537]: cpio: premature end of archive
[/FONT][/FONT][/FONT][FONT=monospace][FONT=monospace][FONT=monospace]May 14 11:31:15 [FONT=monospace]hostname dracut-initramfs-restore[3539]: /usr/lib/dracut/dracut-initramfs-restore: line 57: lz4: command not found
[/FONT][/FONT][/FONT][/FONT][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace]May 14 11:31:15 [FONT=monospace]hostname dracut-initramfs-restore[3538]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 14 11:31:15 hostname dracut-initramfs-restore[3540]: cpio: premature end of archive
May 14 11:31:15 hostname dracut-initramfs-restore[3542]: /usr/lib/dracut/dracut-initramfs-restore: line 58: lzop: command not found
May 14 11:31:15 hostname dracut-initramfs-restore[3541]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 14 11:31:15 hostname dracut-initramfs-restore[3543]: cpio: premature end of archive
[/FONT][/FONT][/FONT][/FONT][/FONT]
[FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace]May 14 11:31:15 [FONT=monospace]hostname systemd[1]: dracut-shutdown.service: Deactivated successfully.
[/FONT][/FONT][/FONT][/FONT][/FONT][/FONT]
[FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace]May 14 11:31:16 [FONT=monospace]hostname systemd[1]: kexec-load.service: Deactivated successfully.
May 14 11:31:16 [/FONT][/FONT][/FONT][/FONT][/FONT][/FONT][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace]hostname[/FONT][/FONT][/FONT][/FONT][/FONT][/FONT] systemd[1]: Finished load default kernel into the current kernel.
[/FONT] [/FONT][/FONT][/FONT][/FONT][/FONT]
You need to provide more details about what you are doing so that someone could verify/reproduce it. Explain step by step how you perform kexec (because e.g. “systemctl kexec” simply refuses to do anything in my case).
Well, pure kexec (without messing with systemd) works for me on TW with kernel 5.17.5 in VM with legacy BIOS and does not work (goes via full reboot) on VM with UEFI. kexec works on Leap 15.2 using UEFI and Leap 15.3 using legacy BIOS. So it looks like regression.
Does it work using older kernel? If yes, you should definitely open bug report.
Kernels 5.17.4-1-default and 5.17.5-1-default behave similarly.
Below is dracut version information.
zypper info dracut
Loading repository data...
Reading installed packages...
Information for package dracut:
-------------------------------
Repository : Main Repository (OSS)
Name : dracut
Version : 056+suse.275.g4ce7a6a7-2.3
Arch : x86_64
Vendor : openSUSE
Installed Size : 1.8 MiB
Installed : Yes (automatically)
Status : out-of-date (version 056+suse.275.g4ce7a6a7-2.2 installed)
Source package : dracut-056+suse.275.g4ce7a6a7-2.3.src
Upstream URL : https://dracut.wiki.kernel.org/
Summary : Event driven initramfs infrastructure
Description :
Dracut contains tools to create a bootable initramfs for Linux kernels >= 2.6.
Dracut contains various modules which are driven by the event-based udev
and systemd. Having root on MD, DM, LVM2, LUKS is supported as well as
NFS, iSCSI, NBD, FCoE.
Here is some log snipppets I believe are relevant from when rebooting from kernel 5.17.4-1-default:
[FONT=monospace]May 15 11:22:26 hostname systemd[1]: Stopping Restore /run/initramfs on shutdown...
[/FONT][FONT=monospace][FONT=monospace]May 15 11:22:26 [FONT=monospace][FONT=monospace]hostname[/FONT] systemd[1]: Starting load default kernel into the current kernel...
[/FONT][/FONT][/FONT][FONT=monospace]May 15 11:22:26 [FONT=monospace][FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3589]: gzip: stdin: not in gzip format
May 15 11:22:26 [FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3588]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 15 11:22:26 [FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3590]: cpio: premature end of archive
May 15 11:22:26 [FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3594]: bzcat: (stdin) is not a bzip2 file.
May 15 11:22:26 [FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3593]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 15 11:22:26 [FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3595]: cpio: premature end of archive
May 15 11:22:26 [FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3597]: xzcat: (stdin): File format not recognized
May 15 11:22:26 [FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3596]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 15 11:22:26 [FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3598]: cpio: premature end of archive
May 15 11:22:26 [FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3600]: /usr/lib/dracut/dracut-initramfs-restore: line 57: lz4: command not found
May 15 11:22:26 [FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3599]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 15 11:22:26 [FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3601]: cpio: premature end of archive
May 15 11:22:26 [FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3603]: /usr/lib/dracut/dracut-initramfs-restore: line 58: lzop: command not found
May 15 11:22:26 [FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3602]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 15 11:22:26 [FONT=monospace]hostname[/FONT] dracut-initramfs-restore[3604]: cpio: premature end of archive
[/FONT][/FONT][FONT=monospace][FONT=monospace]May 15 11:22:27 [FONT=monospace][FONT=monospace]hostname[/FONT] systemd[1]: kexec-load.service: Deactivated successfully.
[/FONT][/FONT][/FONT][FONT=monospace][FONT=monospace][FONT=monospace]May 15 11:22:27 [FONT=monospace][FONT=monospace]hostname[/FONT][FONT=monospace][FONT=monospace][FONT=monospace] systemd[1]: Finished load default kernel into the current kernel.
[/FONT][/FONT][/FONT][/FONT][/FONT][/FONT][/FONT][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace]May 15 11:22:28 [FONT=monospace][FONT=monospace]hostname[/FONT] systemd[1]: Started Show Plymouth Reboot with kexec Screen.
[/FONT][/FONT][/FONT][/FONT][/FONT][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace]May 15 11:22:28 [FONT=monospace][FONT=monospace]hostname[/FONT] systemd[1]: Started Show Plymouth Reboot Screen.
May 15 11:22:28 [FONT=monospace]hostname[/FONT] systemd[1]: Starting Tell Plymouth To Jump To initramfs...
May 15 11:22:28 [FONT=monospace]hostname[/FONT] systemd[1]: Finished Tell Plymouth To Jump To initramfs.
[/FONT][/FONT][/FONT][/FONT][/FONT]
[/FONT]
Installed binaries don’t include highlighted programs. Searching for files reveal that there are packages for them:
# zypper se -f lzop
Loading repository data...
Reading installed packages...
S | Name | Summary | Type
---+------------------+-------------------------------------------------------+--------
i | bash-completion | Programmable Completion for Bash | package
i | breeze5-icons | Breeze icon theme | package
| busybox-misc | Busybox applets not fitting anywhere else | package
| firejail | Linux namepaces sandbox program | package
i | grub2-i386-pc | Bootloader with support for Linux, Multiboot and more | package
i+ | grub2-x86_64-efi | Bootloader with support for Linux, Multiboot and more | package
| lzop | Dictionary-based LZ-type compressor favoring speed | package
# zypper se -f lz4
Loading repository data...
Reading installed packages...
S | Name | Summary | Type
---+-----------------+---------------------------------------------+--------
i | bash-completion | Programmable Completion for Bash | package
i+ | kernel-default | The Standard Kernel | package
i | liblz4-1 | Hash-based predictive Lempel-Ziv compressor | package
| lz4 | Hash-based Predictive Lempel–Ziv compressor | package
i | mariadb | Server part of MariaDB | package
After installing above mentioned packages, I rebooted machine. It did a full reboot. I gave it a second reboot try. Still a full reboot. Below is log snippets from that second reboot. Kernel is 5.17.5-1-default. At least dracut gives different errors now.
May 15 11:56:14 hostname systemd[1]: Stopping Restore /run/initramfs on shutdown...
[FONT=monospace]May 15 11:56:14 [FONT=monospace]hostname systemd[1]: Starting load default kernel into the current kernel...
[/FONT][/FONT][FONT=monospace][FONT=monospace]May 15 11:56:14 [FONT=monospace]hostname dracut-initramfs-restore[2480]: gzip: stdin: not in gzip format
May 15 11:56:14 hostname dracut-initramfs-restore[2479]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 15 11:56:14 hostname dracut-initramfs-restore[2481]: cpio: premature end of archive
May 15 11:56:14 hostname dracut-initramfs-restore[2486]: bzcat: (stdin) is not a bzip2 file.
May 15 11:56:14 hostname dracut-initramfs-restore[2485]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 15 11:56:14 hostname dracut-initramfs-restore[2487]: cpio: premature end of archive
May 15 11:56:14 hostname dracut-initramfs-restore[2489]: xzcat: (stdin): File format not recognized
May 15 11:56:14 hostname dracut-initramfs-restore[2488]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 15 11:56:14 hostname dracut-initramfs-restore[2490]: cpio: premature end of archive
May 15 11:56:14 hostname dracut-initramfs-restore[2492]: Error 44 : Unrecognized header : file cannot be decoded
May 15 11:56:14 hostname dracut-initramfs-restore[2491]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 15 11:56:14 hostname dracut-initramfs-restore[2493]: cpio: premature end of archive
May 15 11:56:14 hostname dracut-initramfs-restore[2497]: lzop: <stdin>: not a lzop file
May 15 11:56:14 hostname dracut-initramfs-restore[2495]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 15 11:56:14 hostname dracut-initramfs-restore[2498]: cpio: premature end of archive
[/FONT][/FONT][/FONT][FONT=monospace][FONT=monospace][FONT=monospace]May 15 11:56:14 [FONT=monospace]hostname systemd[1]: dracut-shutdown.service: Deactivated successfully.
May 15 11:56:14 hostname systemd[1]: Stopped Restore /run/initramfs on shutdown.
[/FONT][/FONT][/FONT][/FONT][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace]May 15 11:56:15 [FONT=monospace]hostname systemd[1]: kexec-load.service: Deactivated successfully.
May 15 11:56:15 hostname systemd[1]: Finished load default kernel into the current kernel.
[/FONT][/FONT][/FONT][/FONT][/FONT][FONT=monospace][FONT=monospace][FONT=monospace][FONT=monospace]May 15 11:56:16 [FONT=monospace]hostname systemd[1]: Started Show Plymouth Reboot with kexec Screen.
May 15 11:56:16 hostname systemd[1]: Started Show Plymouth Reboot Screen.
May 15 11:56:16 hostname systemd[1]: Starting Tell Plymouth To Jump To initramfs...
May 15 11:56:16 hostname systemd[1]: Finished Tell Plymouth To Jump To initramfs.
[/FONT][/FONT][/FONT][/FONT][/FONT]
This implies that before update it worked. It is up to you to find out what changed. If you revert to the snapshot before upgrade, does kexec work again? What TUmbleweed version did you have before this upgrade?
I am not sure if I did not misinterpret what I saw. I cannot reproduce it now. Both VM with TW do full reboot.
Well, I tried to install the latest kernel-stable and kexec-tools in Leap 15.3 and kexec still works as expected. At this point I have to ask once more - does kexec work when you revert to snapshot before update? I also believed that it was working …
How to find out what packages were updated during latest distribution upgrade?
There is /var/log/zypp/history or you can simply compare full content of RPM datbase between previous snapshot and current system (do something like “rpm -qa --dbpath /.snapshots/your-snapshot/usr/lib/sysimage/rpm | sort” and “rpm -qa | sort” and compare two lists). snapper can also show diffs between files in snapshot.
Yes, booting from previous snapshot (kernel-5.17.4-1-default) and rebooting from there. Kexec works then, although similar errors are logged:
May 16 18:28:10 hostname systemd[1]: Starting load default kernel into the current kernel...
[FONT=monospace]May 16 18:28:10 [FONT=monospace]hostname dracut-initramfs-restore[2408]: gzip: stdin: not in gzip format
May 16 18:28:10 hostname dracut-initramfs-restore[2407]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 16 18:28:10 hostname dracut-initramfs-restore[2409]: cpio: premature end of archive
May 16 18:28:10 hostname dracut-initramfs-restore[2411]: bzcat: (stdin) is not a bzip2 file.
May 16 18:28:10 hostname dracut-initramfs-restore[2410]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 16 18:28:10 hostname dracut-initramfs-restore[2412]: cpio: premature end of archive
May 16 18:28:10 hostname dracut-initramfs-restore[2414]: xzcat: (stdin): File format not recognized
May 16 18:28:10 hostname dracut-initramfs-restore[2413]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 16 18:28:10 hostname dracut-initramfs-restore[2415]: cpio: premature end of archive
May 16 18:28:10 hostname dracut-initramfs-restore[2417]: /usr/lib/dracut/dracut-initramfs-restore: line 57: lz4: command not found
May 16 18:28:10 hostname dracut-initramfs-restore[2416]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 16 18:28:10 hostname dracut-initramfs-restore[2418]: cpio: premature end of archive
May 16 18:28:10 hostname dracut-initramfs-restore[2422]: /usr/lib/dracut/dracut-initramfs-restore: line 58: lzop: command not found
May 16 18:28:10 hostname dracut-initramfs-restore[2421]: ERROR: src/skipcpio/skipcpio.c:191:main(): fwrite
May 16 18:28:10 hostname dracut-initramfs-restore[2423]: cpio: premature end of archive
[/FONT][/FONT]May 16 18:28:10 [FONT=monospace]hostname systemd[1]: dracut-shutdown.service: Deactivated successfully.
May 16 18:28:10 hostname systemd[1]: Stopped Restore /run/initramfs on shutdown.
[/FONT]May 16 18:28:11 [FONT=monospace]hostname systemd[1]: kexec-load.service: Deactivated successfully.
[/FONT][FONT=monospace]May 16 18:28:11 [FONT=monospace]hostname systemd[1]: Finished load default kernel into the current kernel.
[/FONT][/FONT]May 16 18:28:12 [FONT=monospace]hostname systemd[1]: Started Show Plymouth Reboot with kexec Screen.
May 16 18:28:12 hostname systemd[1]: Started Show Plymouth Reboot Screen.
May 16 18:28:12 hostname systemd[1]: Starting Tell Plymouth To Jump To initramfs...
May 16 18:28:12 hostname systemd[1]: Finished Tell Plymouth To Jump To initramfs.
[/FONT]May 16 18:28:13 [FONT=monospace]hostname systemd[1]: Removed slice Slice /system/systemd-fsck.
-- Boot 97d6781a44054c7a86b62a1f76888f75 --
May 16 18:28:15 hostname kernel: Linux version 5.17.4-1-default (geeko@buildhost) (gcc (SUSE Linux) 11.2.1 20220420 [revision 691af15031e00227ba6d5935c1d737026cda4129], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.38.20220411-4) #1 SMP PREEMPT Wed Apr 20 07:43:03 UTC 2022 (75e9961)
[/FONT]
Doing normal reboot after this yielded again to full reboot.
Thanks for hinting how to find changes.
Well, sdiff output from those rpm databases is 2890 rows long. It is much easier to tell what has not changed rather than what has changed. Few changes are listed below.
I noticed that zypper info reported dracut version being outdated, thus I did distribution upgrade. Kernel is now newer, but kexec did not magically get fixed.