files disappear / reappear in my /opt folder - happening on 2 machines now

English Install/Boot/Login
#1

I have a strange issue. I had some apps that I had loaded without a package manager and put them in the /opt folder. I did this on 2 machines. I created .desktop files and everything worked good. I used all day long and shut down pc. Next day I power it up and all my apps I loaded this way were not in my app list (Im using GNOME on Opensuse Tumbleweed). I could not figure out what was going on, the desktop files were still there but no icons.

I went to my /opt folder and there are no files at all. the /opt folder is there though, just empty.

If I restart my PC everything will work again. I basically have been working around this by booting my pc twice for apps to work.

I would like to find out why this is happening so I can fix it on both machines before I load too many more applications. I have reloaded both machines from scratch in attempt to fix and it happened again - so it must be some kind of update that is breaking it - I need some help troubleshooting this one.

I did see this in debugfs (this error is here even when my /opt folder does show files so may not be related):
debugfs: Bad magic number in super-block while trying to open /dev/sda1
/dev/sda1 contains a btrfs file system

And here is my fstab output:
chris@L420:/opt> cat /etc/fstab
UUID=5165f1fd-82bd-48be-9132-4d3ac3f107d3 / btrfs defaults 0 0
UUID=5165f1fd-82bd-48be-9132-4d3ac3f107d3 /var btrfs subvol=/@/var 0 0
UUID=5165f1fd-82bd-48be-9132-4d3ac3f107d3 /usr/local btrfs subvol=/@/usr/local 0 0
UUID=5165f1fd-82bd-48be-9132-4d3ac3f107d3 /tmp btrfs subvol=/@/tmp 0 0
UUID=5165f1fd-82bd-48be-9132-4d3ac3f107d3 /srv btrfs subvol=/@/srv 0 0
UUID=5165f1fd-82bd-48be-9132-4d3ac3f107d3 /root btrfs subvol=/@/root 0 0
UUID=5165f1fd-82bd-48be-9132-4d3ac3f107d3 /opt btrfs subvol=/@/opt 0 0
UUID=5165f1fd-82bd-48be-9132-4d3ac3f107d3 /home btrfs subvol=/@/home 0 0
UUID=5165f1fd-82bd-48be-9132-4d3ac3f107d3 /boot/grub2/x86_64-efi btrfs subvol=/@/boot/grub2/x86_64-efi 0 0
UUID=5165f1fd-82bd-48be-9132-4d3ac3f107d3 /boot/grub2/i386-pc btrfs subvol=/@/boot/grub2/i386-pc 0 0
UUID=5165f1fd-82bd-48be-9132-4d3ac3f107d3 /.snapshots btrfs subvol=/@/.snapshots 0 0
UUID=e0ffda71-551b-45a8-bdc9-96d33f9ff33b swap swap defaults 0 0

#2

According to the man page, “debugfs” is for “ext2”, “ext3” and “ext4” file systems. So maybe it isn’t supposed to work for “btrfs”.

I’m not sure what happened in your case. However, notice that “/opt” is a subvolume, and needs to be mounted.

That suggest two possibilities:
(1) You put things in “/opt” when the subvolume was mounted, and then looked when it was not mounted;

(2) You put things in “/opt” when the subvolume was not mount, but when it was later mounted, that hid what was under the mount point.

I don’t normally use “btrfs”. But I do have a test system, where I will try out what happens when “/opt” is or is not mounted.

#3

What you said was interesting - I dont know how I could have copied / created my app folders in /opt if not mounted - but I did something as a “test”.

I got my pc up where I could see my files in /opt and everything was working. I then copied the entire subdirectory structures to a temp location and rebooted.
On the reboot I was back to where I could see /opt, but had NO subdirectories again and things were broke.
I copied the structure from my temp location back to this and everything started working again so I thought I would reboot.

I rebooted and it is still working. I have tried it a few times and things seem to be working now - although I still dont know why / how this could have happened??? Or how it works one time and not another.

I guess I really havent fixed anything - I’m just testing - because now I probably have 2 copies of the same apps and when I boot one time I am running off one and reboot again and running off another.

#4

This indicates that nrickert was right. And also, that you know have the data/apps in two physical places: One set in the folder /opt on the / fs, where the /opt subvolume was not mounted, one in the mounted subvolume which has that folder as it’s mount point. To verify this, you need to mount/unmount the subvolume and inspect it’s contents in both cases. But that raises the question why this subvolume was not mounted everytime. Info should be in the journal.

#5

Before you next decide to reboot:

Look in “/opt”. Maybe create a new file there with the “touch” command.
Check whether “/opt” is mounted (look at the output of “df” or of “mount”).

If it is mounted, then unmount it:


cd /  ### so you are not in "/opt"
umount /opt

Again, look in “/opt” to see what is there. See if the new file that you created with “touch” is there.

And then complete your reboot.

#6

Yet another case of https://forums.opensuse.org/showthread.php/539125-TW-after-update-to-20200211-My-user-hard-drive-partitions-do-not-get-mounted-at-boot-any-longer?

#7

Now that you mention it – that does seem a likely explanation.

#8

So, is this a systemD or BTRFS issue? I could redo theem without BTRFS if that would work.

#9

It is a systemd bug, but it only seems to happen when using “btrfs”.

#10

So I have this fixed now by following something from the linked thread above.

# systemctl disable --now btrfsmaintenance-refresh.service

This fixed my issue.

After doing this I had to reinstall my apps in /opt, but I can reboot , shutdown or anything and it mounts every time now with no issues.

Thanks everyone for your help on this. I just started using OpenSuse a few months ago and did not want to give it up - I love it so far.

Thanks again!!!

#11

We are glad to hear that. And thanks for reporting back.

#12

Yes, it fixes one issue, but it comes with side effects. For proper action see: Beware of btrfsmaintenance-refresh-cron.sh showing up during boot. You may experience serious trouble.](https://forums.opensuse.org/showthread.php/539503-boot-efi-is-not-mounted?p=2930559#post2930559)

#13

I am going to mess with the .service file to see if I can figure out why it is messing up. Im thinking it has something to do with the local-fs.target becasue in the journal i see them mount / unmount really quickly - maybe this is interfering with the mounting. In the meantime I can manually balance , defrag, scrub, and trim from here:

hris@T440:/usr/share/btrfsmaintenance> ls -ltotal 24
-rwxr-xr-x 1 root root 1903 Jun 24  2019 btrfs-balance.sh
-rwxr-xr-x 1 root root  927 Jun 24  2019 btrfs-defrag.sh
-rw-r--r-- 1 root root 2080 Jun 24  2019 btrfsmaintenance-functions
-rwxr-xr-x 1 root root 2776 Jun 24  2019 btrfsmaintenance-refresh-cron.sh
-rwxr-xr-x 1 root root 1227 Jun 24  2019 btrfs-scrub.sh
-rwxr-xr-x 1 root root  941 Jun 24  2019 btrfs-trim.sh

I have only run them here once and they seem to work just fine but have not dug into this too deep yet. Is there any reason I shouldnt run them manually from here that you know of?

#14

Sorry, I meant the thread but linked to the last post only. The fix is easy:

erlangen:~ # systemctl cat btrfsmaintenance-refresh.path 
# /usr/lib/systemd/system/btrfsmaintenance-refresh.path
[Unit]
Description=Watch /etc/sysconfig/btrfsmaintenance

[Path]
PathChanged=/etc/sysconfig/btrfsmaintenance

[Install]
WantedBy=multi-user.target

**# /etc/systemd/system/btrfsmaintenance-refresh.path.d/override.conf
[Unit]
After=local-fs.target
**
erlangen:~ #

For an explanation start here: https://forums.opensuse.org/showthread.php/539503-boot-efi-is-not-mounted?p=2930559#post2930559

#15

Thanks again for the info.

I now have it configured like you showed and seems to be working good.

I did have to enable the btrfs-defrag.timer for some reason - it was not enabled on mine. All 3 units are now enabled and scheduled.

#16

You’re welcome and thanks for the feedback.

#17

Hello nrickert,

the thread is 2 months old, but I have exactly the same issue and the symptoms appear as you told, which is knew for me and so a light of hope seems to shine again : my app in /opt is visible only when it’s unmounted (in terminal). I had posted last year for a similar case : https://forums.opensuse.org/showthread.php/537083-srv-empty-after-booting-recovering-folders-after-reboot
I can’t decide what I should do from this thread, because I haven’t understood clearly if it is OK to use one of the solutions given with btrfs-maintenance(…). Could I change the mount point of /opt, so that it takes my app into account ?
Anyway I guess I’d better get rid of btrfs and move to ext4.
Thanks for letting me know what you think about it.

Erik

#18

I have not heard any recent reports of the “btrfs” mounting bug. So maybe that is fixed. I’m not using “btrfs” except in a VM for testing Leap 15.2, and I have not seen the mounting bug show up there.

In your case, you could consider:

While “/opt” is unmounted, backup “/opt” to external media.

Later, with “/opt” mounted, restore from the external media.

#19

I don’t figure out clearly the mechanism implied in btrfs subvolums. The fact that the same point of a fs can be either a subdirectory of root fs, or an independent file system when mounted, is something I just discovered and of which of course I don’t see all the implications. So this issue seems to me like a three-card trick!
I’ll apply your suggestions tonight at shut down.
Thanks for your help.

#20

Show the subvolumes by running:

erlangen:~ # btrfs subvolume list /
ID 256 gen 209719 top level 5 path @
ID 258 gen 235369 top level 256 path @/var
ID 259 gen 235103 top level 256 path @/usr/local
ID 260 gen 235357 top level 256 path @/tmp
ID 261 gen 234521 top level 256 path @/srv
ID 262 gen 235320 top level 256 path @/root
ID 263 gen 228953 top level 256 path @/opt
ID 264 gen 234079 top level 256 path @/boot/grub2/x86_64-efi
ID 265 gen 209735 top level 256 path @/boot/grub2/i386-pc
ID 266 gen 235338 top level 256 path @/.snapshots
ID 853 gen 235369 top level 266 path @/.snapshots/494/snapshot
ID 907 gen 232395 top level 266 path @/.snapshots/544/snapshot
ID 908 gen 232399 top level 266 path @/.snapshots/545/snapshot
ID 915 gen 232914 top level 266 path @/.snapshots/552/snapshot
ID 916 gen 232915 top level 266 path @/.snapshots/553/snapshot
ID 919 gen 235042 top level 266 path @/.snapshots/556/snapshot
ID 920 gen 235293 top level 266 path @/.snapshots/557/snapshot
erlangen:~ #

Mount the top subvolume for comfortably inspecting all subvolumes:

erlangen:~ # mount -o subvolid=5 /dev/sdb8 /mnt/
erlangen:~ # ls /mnt/@/.snapshots/5/snapshot/
.snapshots  bin  boot  dev  etc  home  lib  lib64  mnt  opt  proc  root  run  sbin  selinux  srv  sys  tmp  usr  var
erlangen:~ #