Crippled ext4 boot partition and efi on a transactional server

English Install/Boot/Login
#1

Hello,

I have following problem with Transactional Server. By mistake it was configured with /boot being ext4 partition on a dedicated drive, which also contains /boot/efi. I have noticed that /boot is also created on root btrfs / partition and actually that one is being used by GRUB and transactional-update. So to the ordinary user or process ext4 /boot is visible, but when you unmount it, normally hidden btrfs /boot shows up…

I would gladly get rid of the crippled ext4 version of /boot, but the other one (the one with btrfs) does not have /boot/efi subdirectory. Because /boot/efi must be special FAT32 partition it has to be a separate partition and I need a mount point. But transactional server prevents me from modifying directory layout, so I can’t create boot/efi directory in btrfs /boot. Any ideas how to make the system healthy other than complete reinstall?

#2

I think we need to see your cat /etc/fstab and lsblk -f output enclosed in code tags ( #] ) here before an answer might be possible.

#3

Hi, and thanks for the interest!

/etc/fstab


UUID=ddfbe360-69e1-429b-8bb2-ff44dab8e8c7 / btrfs ro 0 0 
UUID=ddfbe360-69e1-429b-8bb2-ff44dab8e8c7 /var btrfs subvol=/@/var,x-initrd.mount 0 0 
UUID=ddfbe360-69e1-429b-8bb2-ff44dab8e8c7 /usr/local btrfs subvol=/@/usr/local 0 0 
UUID=ddfbe360-69e1-429b-8bb2-ff44dab8e8c7 /srv btrfs subvol=/@/srv 0 0 
UUID=ddfbe360-69e1-429b-8bb2-ff44dab8e8c7 /root btrfs subvol=/@/root,x-initrd.mount 0 0 
UUID=ddfbe360-69e1-429b-8bb2-ff44dab8e8c7 /opt btrfs subvol=/@/opt 0 0 
UUID=ddfbe360-69e1-429b-8bb2-ff44dab8e8c7 /home btrfs subvol=/@/home 0 0 
UUID=a2c5b108-d9f6-4ad6-9986-d763ef0880e0 /boot ext4 data=ordered 0 2 
UUID=343E-ADF0 /boot/efi vfat utf8 0 2 
UUID=f7f04324-0adc-46ba-a30e-9b1354ea91a5 swap swap defaults 0 0 
UUID=ddfbe360-69e1-429b-8bb2-ff44dab8e8c7 /.snapshots btrfs subvol=/@/.snapshots 0 0 
overlay /etc overlay defaults,lowerdir=/sysroot/var/lib/overlay/247/etc:/sysroot/etc,upperdir=/sysroot/var/lib/overlay/248/etc,workdir=/sysroot/var
/lib/overlay/248/work-etc,x-systemd.requires-mounts-for=/var,x-systemd.requires-mounts-for=/sysroot/var,x-initrd.mount 0 0

[FONT=monospace][FONT=monospace]lsblk -f[/FONT]


NAME      FSTYPE            FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT 
sda       linux_raid_member 1.0   any:0 5b1b0389-0de3-8d45-e0fd-44f31a71315e                 
└─md0                                                                                        
  └─md0p1 btrfs                         ddfbe360-69e1-429b-8bb2-ff44dab8e8c7                /var/lib/lxd/storage-pools/default/virtual-machines/gitbuntu/config.mount 
sdb       linux_raid_member 1.0   any:0 5b1b0389-0de3-8d45-e0fd-44f31a71315e                 
└─md0                                                                                        
  └─md0p1 btrfs                         ddfbe360-69e1-429b-8bb2-ff44dab8e8c7                /var/lib/lxd/storage-pools/default/virtual-machines/gitbuntu/config.mount 
sdc                                                                                          
├─sdc1    swap              1           f7f04324-0adc-46ba-a30e-9b1354ea91a5                [SWAP] 
├─sdc2    vfat              FAT32       343E-ADF0                                32G     0% /boot/efi 
└─sdc3    ext4              1.0         a2c5b108-d9f6-4ad6-9986-d763ef0880e0   21.5G     0% /boot

[/FONT]

#4

Additionaly here’s what GRUB2 has in boot.cfg. It is created by transactional-update grub.cfg command.


[FONT=monospace]menuentry 'openSUSE Tumbleweed'  --class opensuse --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-ddfbe360-69e1-429b
-8bb2-ff44dab8e8c7' { 
        load_video 
        set gfxpayload=keep 
        insmod gzio 
        insmod part_gpt 
        insmod diskfilter 
        insmod mdraid1x 
        insmod btrfs 
        set root='mduuid/5b1b03890de38d45e0fd44f31a71315e,gpt1' 
        if  x$feature_platform_search_hint = xy ]; then 
          search --no-floppy --fs-uuid --set=root --hint='mduuid/5b1b03890de38d45e0fd44f31a71315e,gpt1'  ddfbe360-69e1-429b-8bb2-ff44dab8e8c7 
        else 
          search --no-floppy --fs-uuid --set=root ddfbe360-69e1-429b-8bb2-ff44dab8e8c7 
        fi 
        echo    'Loading Linux 5.14.2-1-default ...' 
        linuxefi /boot/vmlinuz-5.14.2-1-default root=UUID=ddfbe360-69e1-429b-8bb2-ff44dab8e8c7  ${extra_cmdline} splash=silent resume=/dev/disk/by-
uuid/f7f04324-0adc-46ba-a30e-9b1354ea91a5 mitigations=auto intel_iommu=on iommu=pt quiet 
        echo    'Loading initial ramdisk ...' 
        initrdefi       /boot/initrd-5.14.2-1-default 
}
[/FONT]

As you can see it uses UUID=ddfbe360-69e1-429b-8bb2-ff44dab8e8c7 and yes there is /boot there into which transactional-update puts things

#5

That is exactly what transactional server is about, is not it? Current root is immutable and any changes are applied to clone of current root and are activated after reboot.

Run “transactional-update shell”. It clones current root into new subvolume and sets it up, then chroots into it and starts interactive shell. New subvolume is read-write at this point, so you can create files and directories, install software, edit file like /etc/fstab etc. Do whatever changes you need and after you exit shell this subvolume will be made read-only and mounted as root after reboot.

If you change location of /boot you will need to run “update-bootloader --reinit” before exiting chroot to reconfigure bootloader. Check what is mounted currently, transactional-update sets up minimal environment and not everything may be mounted.

Keep in mind that installer sets up different subvolume layout when /boot is inside root than it does for separate /boot. I am actually quite surprised that installer allows transactional update server with separate /boot at all. I do not know if these subvolumes are even documented anywhere except source code (I saw one additional subvolume I have never met before). Whether this may cause subtle problems later I cannot say. Also installer may do something else besides setting up these subvolumes that is not documented either.

#6

Thanks, I solved it myself before reading your answer in a bit awkard way by creating dummy RPM package that created the “efi” directory. Looks like I could use “shell” option (been a bit mislead by “testing and debugging”). I have removed ext4 “/boot” partition from “/etc/fstab” and everything looks OK for now…

Now I have another problem, that “boot/grub2/x86_64-efi”](https://documentation.suse.com/sles/15-SP1/html/SLES-all/cha-filesystems.html#sec-filesystems-major-btrfs-suse)should be a btrfs subvolume, I don’t clearly see why, especially that kernel can be updated, so a rollback would leave one with wrong bootloader configuration. Not sure if I can add missing subvolumes now. I’ll leave it for now, in case something breaks I will copy/edit GRUB files manually.

If you change location of /boot you will need to run “update-bootloader --reinit” before exiting chroot to reconfigure bootloader. Check what is mounted currently, transactional-update sets up minimal environment and not everything may be mounted.

IDK what transactional-update really does under the hood, but recently it was totally ignoring ext4 “/boot” partition. It has created “/boot/” on btrfs and started to put kernels there on its own. The problem appeared after I wanted to add parameters to the kernel in GRUB2 configuration. Conveniently used YaST for this and YaST took contents o ext4 “/boot”… There was outdated kernel there and modules for it were long gone, so I ended up in recovery mode with empty system and had to edit GRUB2 config file manually… That was my story, once again thank you :slight_smile:

#7

There are many ways to skin a cat …

“boot/grub2/x86_64-efi”](https://documentation.suse.com/sles/15-SP1/html/SLES-all/cha-filesystems.html#sec-filesystems-major-btrfs-suse)should be a btrfs subvolume, I don’t clearly see why

grub consists of two parts - binary that is initially loaded by firmware (or by code in MBR/PBR) and modules that are loaded on demand from /boot/grub. The initial binary is installed in platform specific location outside of root filesystem (even if it is physically on the same partition it is not subject to snapshot management). Modules are located under /boot/grub2/x86_64-efi (on UEFI) or in other platform directories of /boot/grub2. Both parts must match otherwise in the best case modules cannot be loaded, in the worst case you get hard to debug incorrect behavior. Both parts are updated when new grub2 version is installed.

If /boot/grub2/x86_64-efi located in root subvolume, it would have been reverted on snapper rollback, but the first stage binary not. So you would have mismatch leading to problems. For this reason those boot time grub2 platform directories are split off into separate subvolumes to be excluded from root snapshot management.

especially that kernel can be updated, so a rollback would leave one with wrong bootloader configuration.

This is managed by snapper which creates grub2 configuration to load grub.cfg that belongs to the correct snapshot. But this handling is disabled if /boot is on separate partition … which is the reason why snapper rollback requires /boot to be part of root subvolume.

Not sure if I can add missing subvolumes now.

Yes, you can. Create subvolumes following general name pattern, copy content of original /boot/grub2/x86_64-efi, edit fstab. You do not even need to go via creating new root subvolume for that. Notice that transactional update server also expects to have @/boot/writable subvolume (which must be mounted on /boot/writable), and for this you need to go via transactional-update again to create mount point.

IDK what transactional-update really does under the hood, but recently it was totally ignoring ext4 “/boot” partition. It has created “/boot/” on btrfs and started to put kernels there on its own.

Correct. As transactional update server depends on btrfs snapshots (in particular, ability to rollback) and this requires /boot on root subvolume as I mentioned earlier, transactional-update never expects /boot to be separate filesystem and does not mount it inside of new root. It is possible that something changed in the way new subvolume is set up, but that is how it works now.

If you installed your system in this configuration originally, this warrants bug report. At least there should be big red warning during installation of transactional server with separate /boot partition.

#8

That is likely too optimistic. grub must know where these locations are and subvolumes are accessed differently which means after changing them into subvolume you need to reinstall bootloader, so again via transactional-update. Note that transactional-update has “bootloader” option that does exactly that.