upgrade failure from Leap 15.1 with disk space warning on /boot

[BerndSpeiser]

I want to upgrade from Leap 15.1 to 15.2 on a machine with the following device layout

> lsblk
NAME                                       MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                          8:0    0 149.1G  0 disk
├─sda1                                       8:1    0   156M  0 part  /boot
├─sda2                                       8:2    0    22G  0 part
│ └─cr_ata-WDC_WD1600BEVT-22ZCT0_WD-WXE708N10973-part2
│                                          254:0    0    22G  0 crypt
│   ├─system-root                          254:1    0  19.9G  0 lvm   /
│   └─system-swap                          254:2    0     2G  0 lvm   [SWAP]
└─sda4                                       8:4    0   127G  0 part
  └─cr_ata-WDC_WD1600BEVT-22ZCT0_WD-WXE708N10973-part4
                                           254:3    0   127G  0 crypt
    └─system-home                          254:4    0   127G  0 lvm   /home
sr0                                         11:0    1  1024M  0 rom

Disk space usage is reported as

> df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 2.0G     0  2.0G   0% /dev
tmpfs                    2.0G     0  2.0G   0% /dev/shm
tmpfs                    2.0G  1.5M  2.0G   1% /run
tmpfs                    2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/mapper/system-root   20G   13G  5.8G  69% /
/dev/sda1                148M   42M   96M  31% /boot
/dev/mapper/system-home  125G   49G   71G  41% /home
tmpfs                    393M   12K  393M   1% /run/user/1000

This layout has been used ever since openSuSE 13.2, where it was suggested by the installer. The /boot partition is not too large, but I always managed to update kernel versions, with keeping the running and one older kernel.
Now I use the USB installer downloaded from openSuSE, and it works fine until I receive the system probing screen, when it reports that I am out of disc space ob /boot (160% usage). Booting back into Leap 15.1, I deleted the older kernel version to gain some space, and repeated the attempt to upgrade. The same error turns up, now the usage of /boot is reported to be reduced to 123%, but still more then 100%. I can't provide a screen shot during the upgrade, but here is the contents of the pop-up window shown:

Disk Space Warning - Yast2
Error: Out of disk space!
                           Free         Total
/boot    123%       -27.2 MiB   120.1 MiB
/          71%         5.0 GiB      17.6 GiB
/home  41%         69.1 GiB    117.5 GiB

However, device /dev/sda1 mounted on /boot still is at only 31% according to df -h. The installer probably adds up the memory already used and the additional one that will be taken up by the new version, for /boot the kernel still left from Leap 15.1 + the new one from the Leap 15.2 upgrade, but that should not add up to 123% if the new kernel is not much larger in size than the previous ones. Something seems to be inconsistent here.

Is there something wrong in the disk space calculation in the installer? Anyway, what can be done about this, except doing away with the upgrade and starting a totally new install of the machine with 15.2.

Thanks for any help.

[mrmazda]

A 15.2 kernel rpm is 152% of the size of a 13.2 kernel rpm. Kernels & initrds have been growing precipitously.

Just how big is/are your 15.1 initrd(s)? You may be able to reconfigure dracut to make them smaller. This one includes RAID support, but not LVM, nor plymouth:

# ll /boot/i*28.52*
-rw------- 1 root root 9189752 Jun 25 16:34 /boot/initrd-4.12.14-lp151.28.52-default

For minimizing size, my /etc/dracut.conf.d/dmodules.conf contains:

omit_drivers+="i18n uefi-lib encryptfs btrfs iscsi lvm lvm2 dmraid plymouth resume sata_sil usb_storage "

You could try a zypper online distribution upgrade with the kernel locked, so that the upgrade result would still be running a 15.1 kernel. Then you could reduce the installed kernels to one, if not already done, and a 15.2 kernel should fit via an ordinary zypper up after removing the kernel lock. I do all my OS (online, which is normal/common procedure here) upgrades with kernel locked, so I know it's not a problem to do.

Long term, if not immediately, /boot needs to be larger. You should be able to shrink the sda2 lvm at its front, backup the content of sda1, delete and recreate sda1 at a larger size, then restore sda1 content, with plenty of room on /boot/ resulting.

[OrsoBruno]

A 15.1 install here with two kernels installed totals about 75 MB.
A 15.2 fresh install with two kernels istalled (the one from the DVD and one update) totals just in excess of 84 MB.
If your sda1 mounted on /boot is really 156MB two or even three 15.2 kernels should fit easily, unless you have an unusual configuration, e.g. very large initrd's.
Apparently, as you write, something doesn't add up.

[BerndSpeiser]

A 15.1 install here with two kernels installed totals about 75 MB.
A 15.2 fresh install with two kernels istalled (the one from the DVD and one update) totals just in excess of 84 MB.
If your sda1 mounted on /boot is really 156MB two or even three 15.2 kernels should fit easily, unless you have an unusual configuration, e.g. very large initrd's.
Apparently, as you write, something doesn't add up.

Thanks for the replies so far.
While I agree with mrmazda that on the long run, extending the /boot partition would be desirable (and that might indeed be the way to go), I would be interested to find out what is going on with respect to the size counts here.

The single kernel installed on 15.1 in my present configuration uses a total of about 42 MB (according to df -h corresponding to 31% as mentioned in the original post). The size of the /boot contents is:

> ll /boot
total 33324
-rw-r--r-- 1 root root      512  4. Aug 2017  backup_mbr
lrwxrwxrwx 1 root root        1  4. Aug 2017  boot -> .
-rw-r--r-- 1 root root     1725 13. Jul 2019  boot.readme
-rw-r--r-- 1 root root   200382 11. Jun 09:16 config-4.12.14-lp151.28.52-default
drwxr-xr-x 2 root root     1024  4. Aug 2017  grub
drwxr-xr-x 6 root root     1024 19. Jul 20:37 grub2
lrwxrwxrwx 1 root root       34  1. Jul 16:59 initrd -> initrd-4.12.14-lp151.28.52-default
-rw------- 1 root root 14170268 17. Jul 20:12 initrd-4.12.14-lp151.28.52-default
drwx------ 2 root root    12288 12. Feb 2015  lost+found
-rw-r--r-- 1 root root   407933 11. Jun 09:25 symvers-4.12.14-lp151.28.52-default.gz
-rw-r--r-- 1 root root      484 11. Jun 09:25 sysctl.conf-4.12.14-lp151.28.52-default
-rw-r--r-- 1 root root  3646301 11. Jun 09:24 System.map-4.12.14-lp151.28.52-default
-rw-r--r-- 1 root root  8349426 11. Jun 09:28 vmlinux-4.12.14-lp151.28.52-default.gz
lrwxrwxrwx 1 root root       35  1. Jul 16:59 vmlinuz -> vmlinuz-4.12.14-lp151.28.52-default
-rw-r--r-- 1 root root  7327856 11. Jun 09:47 vmlinuz-4.12.14-lp151.28.52-default

I would expect that I should be able to fit the 15.2 kernel, if the initrd and kernel are not more than twice the size of the 15.1 ones. Still, the 15.2 installer reports the disk space warning, mentioned in the original post. In OrsoBruno's case the increase seems to be more like 15%, which should fit into the available space easily. Any idea what is going on? Do we need extra space in /boot during the upgrade, which must be taken into consideration by the installer?

Now before asking for some advice on possible repartitioning, I'd be curious if we needto keep the gz versions of the symvers and vmlinuz of the 15.1 from which I start the upgrade, or could this be deleted.

[OrsoBruno]

Puzzled by this problem I did some tests in a VM.
First I moved everything but the booting kernel (vmlinuz-xxx), its initrd and their symlinks out of the /boot folder and, as expected, the system booted happily. Please note that you might need some of the other files to build additional modules etc. though.
Then I tried a fresh install with a /boot partition and the installer didn't accept anything short of 110 MB and refused to go ahead otherwise; so I guess that you have to provide at least some 110 MB of free space.
By the way, you may try to reduce the size of the initrd by running the "dracut --hostonly --force" command that includes only modules needed to boot on the current system.
Interestingly, the installer also wanted an 8MB "BIOS boot partition" to proceed, apparently without means to create one other than by accepting the one proposed originally.

My take is that the new installer has (too many?) safeguards and maybe a bug or two, but maybe members with more insight will comment on that.

[OrsoBruno]

Update from further tests.
I was able to install 15.1 with a 110 MB /boot (formatted to EXT4, this seems to have a role). Then the 15.2 DVD installer let me proceed with an upgrade, but later installation of the 5.3.x kernel from 15.2 failed.
Ignoring that and going forward left me with a 15.2 system... happily booting with a 4.12 kernel from 15.1!
Trying to install from scratch 15.2 with a 110 MB /boot is stopped by a "not enough space on /boot...".
Install with a 150 MB /boot (EXT4) succeeds with a warning of "only 20 MB remaining on /boot", but then any kernel update aborts with "xxx MB needed on /boot..." errors.
Eventually I succeeded updating the kernel by deleting everything but the /boot/grub2 folder from /boot just before issuing "zypper up" but that way you are left with an unbootable system if anything goes wrong.

All in all, maybe you can find a workaround to upgrade to 15.2 with a 150 MB /boot, but then you are left with a system almost impossible to maintain.
(well, unless somebody comes up with some other piece of magic...)

[nrickert]

I ran into this problem with openSUSE 12.3. I had been using "/boot" at 100M, which seemed to have oodles of space to spare back when I started doing that (maybe SuSE 10.2). But with 12.3, it was tight. There was space for two kernels, but not quite enough for three. So I got into the practice of manually deleting the oldest kernel before a kernel update.

When 13.1 came out, I did some repartitioning to enlarge "/boot" to at least 200M.

On my current desktop, where 12.3 was the first version installed, I went with 500M for "/boot". And I'll note that I only use a separate "/boot" because of the encryption.

Part of the problem is that when a kernel is installed, the size of the "initrd" is not known. So the space estimate is only an estimate, and perhaps they overestimate the amount of space needed.

[mrmazda]

IME, a fresh installation always generates a considerably larger initrd than those next generated after installation, whether with same kernel, or newer. IIRC, the first is more than double in size, possibly close to triple, of those generated later.

[BerndSpeiser]

Thanks for the further comments and suggestions.

I will do some tests myself, e.g. decreasing the size of the initrd, moving out the gz files, but this will take me a day or two. I'll report back the results.
Unfortunately, I did not take notes about the sizes of files in /boot in earlier installations, however, if I manage to get the update to 15.2. done, it will be interesting to see how big the kernel size is, and compare it to the 15.1 one on this machine.

In the meantime, one more question: is the old kernel (here 15.1) still needed during the upgrade? Is this only as a backup if something goes wrong? Then, it could be moved from /boot into some backup just before the upgrade is attempted (possibly using a live USB stick). But maybe there is some other reason that it has to stay.

[OrsoBruno]

Not all systems are equal, but here I see that the 15.2 vmlinuz is about 9 MB vs. 7.3 MB for 15.1, while the initrd's are about the same at 13.5 MB or so. Apparently temporary files during the install make a material difference.
Also the .gz and system map files are several MB larger in 15.2

As for the 15.1 kernel, like my test previously reported shows, it is not needed if you do a DVD upgrade, and is not needed either after boot up even if you do a "zypper dup", but then you are left with no backup in case of trouble.

Another option would be to create a new partition somewhere else, mount it as /boot and ignore or delete the old /boot partition. I have not tested that yet, but during an upgrade what really matters is the / (root) partition and you can reuse that without formatting.
Maybe you have to adjust or reinstall GRUB if you go that route, and anyway I would test the whole process in a VM beforehand.