Important btrfs subvolumes like /root and /opt are unmounted during boot

English Install/Boot/Login
1

Hello, for some time now I noticed that some very important btrfs subvolumes (like /root and /opt) are not mounted when I am logged in, but get mounted without problems if I run mount -a.

Checking the journal from boot time, it seems they get mounted in the beginning, get unmounted soon afterwards, and finally only few of them get remounted. In detail, I see the following messages after boot:


Nov 25 11:59:16   systemd[1]: Condition check resulted in dracut pre-mount hook being skipped.
Nov 25 11:59:16   systemd[1]: Mounting /sysroot...
Nov 25 11:59:16   systemd[1]: Mounted /sysroot.
Nov 25 11:59:16   systemd[1]: Condition check resulted in dracut mount hook being skipped.
Nov 25 11:59:17   systemd[1]: Mounting /.snapshots...
Nov 25 11:59:17   systemd[1]: Mounting /boot/efi...
Nov 25 11:59:17   systemd[1]: Mounting /boot/grub2/i386-pc...
Nov 25 11:59:17   systemd[1]: Mounting /boot/grub2/x86_64-efi...
Nov 25 11:59:17   systemd[1]: Mounting /home...
Nov 25 11:59:17   systemd[1]: opt.mount: Directory /opt to mount over is not empty, mounting anyway.
Nov 25 11:59:17   systemd[1]: Mounting /opt...
Nov 25 11:59:17   systemd[1]: Mounting /root...
Nov 25 11:59:17   systemd[1]: Mounting /srv...
Nov 25 11:59:17   systemd[1]: Mounting /tmp...
Nov 25 11:59:17   systemd[1]: Mounting /usr/local...
Nov 25 11:59:17   systemd[1]: Mounting /var...
Nov 25 11:59:17   systemd[1]: Mounted /.snapshots.
Nov 25 11:59:17   systemd[1]: Mounted /boot/grub2/i386-pc.
Nov 25 11:59:17   systemd[1]: Mounted /boot/grub2/x86_64-efi.
Nov 25 11:59:17   systemd[1]: Mounted /home.
Nov 25 11:59:17   systemd[1]: Mounted /opt.
Nov 25 11:59:17   systemd[1]: Mounted /root.
Nov 25 11:59:17   systemd[1]: Mounted /srv.
Nov 25 11:59:17   systemd[1]: Mounted /tmp.
Nov 25 11:59:17   systemd[1]: Mounted /usr/local.
Nov 25 11:59:17   systemd[1]: Mounted /var.
Nov 25 11:59:17   kernel: FAT-fs (nvme0n1p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
Nov 25 11:59:17   systemd[1]: Mounted /boot/efi.

But a few seconds all the partitions get unmounted:


Nov 25 11:59:19   systemd[1]: Unmounting /.snapshots...
Nov 25 11:59:19   systemd[1]: Unmounting /boot/grub2/i386-pc...
Nov 25 11:59:19   systemd[1]: Unmounting /boot/grub2/x86_64-efi...
Nov 25 11:59:19   systemd[1]: Unmounting /home...
Nov 25 11:59:19   systemd[1]: Unmounting /opt...
Nov 25 11:59:19   systemd[1]: Unmounting /root...
Nov 25 11:59:19   systemd[1]: Unmounting /srv...
Nov 25 11:59:19   systemd[1]: Unmounting /usr/local...
 ... ]
Nov 25 11:59:19   systemd[1]: \x2esnapshots.mount: Succeeded.
Nov 25 11:59:19   systemd[1]: Unmounted /.snapshots.
Nov 25 11:59:19   systemd[1]: boot-grub2-i386\x2dpc.mount: Succeeded.
Nov 25 11:59:19   systemd[1]: Unmounted /boot/grub2/i386-pc.
Nov 25 11:59:19   systemd[1]: boot-grub2-x86_64\x2defi.mount: Succeeded.
Nov 25 11:59:19   systemd[1]: Unmounted /boot/grub2/x86_64-efi.
Nov 25 11:59:19   systemd[1]: home.mount: Succeeded.
Nov 25 11:59:19   systemd[1]: Unmounted /home.
Nov 25 11:59:19   systemd[1]: opt.mount: Succeeded.
Nov 25 11:59:19   systemd[1]: Unmounted /opt.
Nov 25 11:59:19   systemd[1]: root.mount: Succeeded.
Nov 25 11:59:19   systemd[1]: Unmounted /root.
Nov 25 11:59:19   systemd[1]: srv.mount: Succeeded.
Nov 25 11:59:19   systemd[1]: Unmounted /srv.
Nov 25 11:59:19   systemd[1]: usr-local.mount: Succeeded.
Nov 25 11:59:19   systemd[1]: Unmounted /usr/local.
Nov 25 11:59:19   systemd[1]: Condition check resulted in /dev/system/root being skipped.
Nov 25 11:59:19   systemd[1]: Found device /dev/mapper/system-root.
Nov 25 11:59:19   systemd[1]: Found device /dev/system/swap.
Nov 25 11:59:19   systemd[1]: systemd-update-utmp.service: Succeeded.
Nov 25 11:59:19   systemd[1]: Stopped Update UTMP about System Boot/Shutdown.
Nov 25 11:59:19   systemd[1]: Requested transaction contradicts existing jobs: Resource deadlock avoided
Nov 25 11:59:19   systemd[1]: Requested transaction contradicts existing jobs: Resource deadlock avoided
Nov 25 11:59:43   systemd[1]: Unmounting /tmp...
Nov 25 11:59:43   systemd[1]: Started Scrub btrfs filesystem, verify block checksums.
Nov 25 11:59:43   systemd[1]: tmp.mount: Succeeded.
Nov 25 11:59:43   systemd[1]: Unmounted /tmp.

Even though /home was unmounted, it gets remounted later when I login as user:


Nov 25 12:00:02   sddm[2445]: Authenticated successfully
Nov 25 12:00:02   sddm-greeter[2497]: Message received from daemon: LoginSucceeded
Nov 25 12:00:02   systemd[1]: Created slice User Slice of UID 1000.
Nov 25 12:00:02   systemd[1]: Starting User Runtime Directory /run/user/1000...
Nov 25 12:00:02   systemd-logind[2486]: New session 2 of user jimis.
Nov 25 12:00:02   systemd[1]: Mounting /home...
Nov 25 12:00:02   systemd[1]: Mounted /home.
Nov 25 12:00:02   systemd[1]: Started User Runtime Directory /run/user/1000.
Nov 25 12:00:02   systemd[1]: Starting User Manager for UID 1000...

But the rest of the important partitions remain unmounted until I run mount -a.

Any idea what’s going on? How can I trace systemd dependencies and figure out
what is causing the unmounting of the subvolumes?

Thank you in advance.

P.S. here is my /etc/fstab:


/dev/system/root  /                       btrfs  defaults                      0  0
/dev/system/root  /var                    btrfs  subvol=/@/var                 0  0
/dev/system/root  /usr/local              btrfs  subvol=/@/usr/local           0  0
/dev/system/root  /tmp                    btrfs  subvol=/@/tmp                 0  0
/dev/system/root  /srv                    btrfs  subvol=/@/srv                 0  0
/dev/system/root  /root                   btrfs  subvol=/@/root                0  0
/dev/system/root  /opt                    btrfs  subvol=/@/opt                 0  0
/dev/system/root  /home                   btrfs  subvol=/@/home                0  0
/dev/system/root  /boot/grub2/x86_64-efi  btrfs  subvol=/@/boot/grub2/x86_64-efi  0  0
/dev/system/root  /boot/grub2/i386-pc     btrfs  subvol=/@/boot/grub2/i386-pc  0  0
UUID=123A-567B    /boot/efi               vfat   defaults                      0  0
/dev/system/swap  swap                    swap   defaults                      0  0
/dev/system/root  /.snapshots             btrfs  subvol=/@/.snapshots          0  0
2

Im am not a Btrfs user, but from what I read above is that you say that /home is automagicaly mounted when needed. Isn’t that the same with the other ones.? Can you log in as root from a terminal emulator as root (which would need /root)?

3

I’ve seen this mentioned elsewhere – either another forums post or a mailing list message.

I seem to recall that it is a “systemd” bug.

There’s also an open bug report bug 1155170, but that seems to only be concerned with “/boot” and “/boot/efi”.

4

I also remember something like this, but that was about “real” file systems (from real partitions). not about Btrfs sub volumes).

5

Yes, I think you are right about “real partitions”. Maybe this is worth a separate bug report.

6

I am not sure there is a Problem at all. The OP only reports that he sees those subvolumes as not mounted and that he can mount them with mount -a.
He does NOT report that things are not working because of unmounted sub-volumes.
He reports that at least one of them (/home) is mounted automatic when someone logs in. Thus normaly one would not detect that it was unmounted before that log in. I am now waiting on his report if login in as root also results in automatic mounting of /root. He can also try of course to do something inside /opt to see if automounting happens.

To me it looks like the automounting that formerly was (and still is) supported by /etc/automounter, but that now can also be used as a systemd feature (configurated directly in /etc/fstab). To me it looks as if systemd has one more “only do this when needed” feature. I admit that it is a bit of guessing, but the fact that /home is mounted when needed is the trigger for me.
When this automounting takes place, there is no problem, except that he is seeing a systemd feature at work.

7

systemd mounts what I need:

erlangen:~ # journalctl -b -o short-monotonic | grep -i mount
    0.142493] erlangen kernel: Mount-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
    0.142532] erlangen kernel: Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
    1.917314] erlangen systemd[1]: Condition check resulted in dracut pre-mount hook being skipped.
    2.866338] erlangen systemd[1]: Mounting /sysroot...
    2.877360] erlangen systemd[1]: Mounted /sysroot.
    2.977661] erlangen systemd[1]: Condition check resulted in dracut mount hook being skipped.
    3.837189] erlangen kernel: FAT-fs (sdb1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
    3.853239] erlangen kernel: EXT4-fs (nvme0n1p3): mounted filesystem with ordered data mode. Opts: data=ordered
    4.060798] erlangen kernel: EXT4-fs (sda4): mounted filesystem with ordered data mode. Opts: (null)
    4.113867] erlangen systemd[1]: Mounting /home-HDD...
    4.113975] erlangen systemd[1]: Mounted /home-HDD.
erlangen:~ #

Some of the following do not show up in journal:

erlangen:~ # mount |grep sdb5
/dev/sdb5 on / type btrfs (rw,relatime,ssd,space_cache,subvolid=576,subvol=/@/.snapshots/277/snapshot)
/dev/sdb5 on /.snapshots type btrfs (rw,relatime,ssd,space_cache,subvolid=266,subvol=/@/.snapshots)
/dev/sdb5 on /boot/grub2/i386-pc type btrfs (rw,relatime,ssd,space_cache,subvolid=265,subvol=/@/boot/grub2/i386-pc)
/dev/sdb5 on /boot/grub2/x86_64-efi type btrfs (rw,relatime,ssd,space_cache,subvolid=264,subvol=/@/boot/grub2/x86_64-efi)
/dev/sdb5 on /opt type btrfs (rw,relatime,ssd,space_cache,subvolid=263,subvol=/@/opt)
/dev/sdb5 on /srv type btrfs (rw,relatime,ssd,space_cache,subvolid=261,subvol=/@/srv)
/dev/sdb5 on /root type btrfs (rw,relatime,ssd,space_cache,subvolid=262,subvol=/@/root)
/dev/sdb5 on /usr/local type btrfs (rw,relatime,ssd,space_cache,subvolid=259,subvol=/@/usr/local)
/dev/sdb5 on /tmp type btrfs (rw,relatime,ssd,space_cache,subvolid=260,subvol=/@/tmp)
/dev/sdb5 on /var type btrfs (rw,relatime,ssd,space_cache,subvolid=258,subvol=/@/var)
erlangen:~ #
8

Yes, there are problems.

[ul]
[li]My custom scripts fail because they expect files they can’t find. [/li][li]At least one systemd service fails: snapper-cleanup.service fails with “IO Error (.snapshots is not a btrfs subvolume).”, apparently because the /.snapshots subvolume is not mounted. [/li][/ul]

I’m not sure how and why systemd works, and if this is proper behaviour, it doesn’t look like that though:
[LIST=|INDENT=1]
[li]All partitions are mounted early during boot, but a moment later all of them get unmounted. Why?[/li][ul]
[li]Can anyone check if you see this behaviour in your journalctl -b ? [/li][/ul]

[li]The fstab entries should by default be “auto” (unless “noauto” is specified"), which means that they are to be mounted with mount -a, and in sysvinit the command mount -a was a crucial part of the boot process.[/li][ul]
[li]Does systemd also does mount -a, like sysvinit? It claims full fstab compatibility, so I would assume yes. [/li][/ul]

[/LIST]

9

Thank you, this looks relevant. He mentions that it happens only after “zypper dup” runs, which might explain why I notice the problem sometime. I’ll keep an eye on the bug report.

10

Ok, then your problem differs from my interpretation. You only said they were umounted und you swa that in the log. Not that they weren’t there when need and that the found confirmed in the log.

In that case, it seems to be a bug like nrickert assumes.

And yes, of course a systemd system should do an equivalent of mount -a like that was the pre systemd action at boot.

11

Searching bugzilla for ‘mount unmount boot’ returns 11 results.

What about lvm2? Beware, it warrants some extra fun:

erlangen:~ # zypper se lvm2
Loading repository data...
Reading installed packages...

S | Name              | Summary                                    | Type   
--+-------------------+--------------------------------------------+--------
  | libbd_lvm2        | The LVM plugin for the LibBlockDev library | package
  | liblvm2cmd2_03    | LVM2 command line library                  | package
  | libudisks2-0_lvm2 | UDisks module for LVM2                     | package
  | lvm2              | Logical Volume Manager Tools               | package
  | lvm2-devel        | Development files for LVM2                 | package
  | lvm2-lockd        | LVM locking daemon                         | package
  | lvm2-testsuite    | LVM2 Testsuite                             | package
erlangen:~ #