Note: I would have submitted a bug report about this, but bugzilla.opensuse.org is not letting me log in. When I try, I only get:
“The IDPs response was missing the SAMLResponse parameter. This can be cause by inappropriate redirection, MITM behaviour, a failed login attepmt on the IDP, invalid response by the IDP, etc.
It is not possible to continue to login, either retry the IDP or use your local credentials to login.”
(Large and in red, in 2 different browsers, and yes, this is a verbatim quote.):
Ever since I updated to openSUSE Leap 16.0 (from 15.6) I’ve been wondering why my system gets so terribly sluggish so terribly often, to the point that even the desktop clock applet stops moving sometimes and opening a Dolphin window takes well > 30 s. I found out that there’s a full btrfs qgroup rescan going on once every hour, taking ~5 min every time. (Yes, my computer still has rotating disks.) This not only brings the PC to a near halt, but also prevents going to standby during 5 min in every hour, because the kworker thread doing the scan refuses to be frozen. (If someone closed their laptops during such a period, expecting it to sleep in its bag, it would just stay on, use up the battery and run hot…)
After some research I found that this hourly qgroup rescan is triggered by a “snapper-cleanup.timer” that, contrary to its description (“daily cleanup”), does in fact run hourly on openSUSE 16.0. Why has the frequency increased 24x in openSUSE 16.0, even when no hourly snapshots are and have ever been created on the system?
Would it be possible to create two cleanup timers, one that’s triggered hourly and one that’s triggered daily, so that the daily timer only does something when hourly snapshots are off, and the hourly timer only activates when hourly snapshots are on?
There are alreda several bugreports regarding this issue. Due to the fact that qgroup causes massive issues on sveral systems, the devs decided to disable this feature in general for the future. So you can disable qgroup and the issue is gone.
OK, then I must have missed these reports. I actually did search for related bugs in bugzilla but found none using the keywords “snapper” or “quota” or “qgroup”.
I already found a workaround for me: Copy “snapper-cleanup.timer” to /etc/systemd/system and restore the previous behaviour to only run 1x/day.
But my point is: I don’t think the present default for snapper cleanups is a sensible default on desktop systems (much less on laptops), due to the consequences outlined above, but simply disabling btrfs quota completely is not a sensible workaround:
Either cleanups due to snapshots exceeding a certain space limit is a desired feature, then quotas are needed, or this is not an expected feature, then snapper could refrain from triggering qgroup rescans in the first place, be quotas enabled or not.
A sensible default would be (imho) to only run hourly cleanups when hourly snapshots are enabled, otherwise a daily cleanup is sufficient.
Ah, I see… I should not have restricted my bug searching to Leap…
But “just disable quota” is just an insane suggestion: What other reason would exist to run an hourly snapshot cleanup if not to prevent snapshots from taking too much space? (Which you’ll not detect with quota off…)
No it is not. The bugreports explain that the upstream devs of BTRFS do no longer recommend the usage of qgroup in combination with snapshots and subvolumes .
Ha! Thanks for this post. I noticed a huge slowdown on my TW system yesterday after restarting and saw the btrfs qgroup process churning away. I didn’t think too much about it at the time because it seemed to sort itself out, but it’s nice to know the cause and fix.
I don’t know the latest status around qgroup, I know quota is just a no-go for years already for me. I also use HDD’s still for backup and bulk data and I learned my lessons.
Besides the normal ‘quota’ there is also ‘simple quota’ implementation, but never used is. I think it is already decade ago that I disabled quota. On Debian systems one can still see some error w.r.t. to that when snapper is run. Maybe because I never looked at it how and why that is there.
Anyway, I you want to limit space usage, a method is to simply limit the partition or blockstorage size (the main one for the filesystem). In case to ad-hoc space problems: Btrfs is multi-device, so you can just add an USB-stick for example (just temporary). But just some spare/extra partition on the same storage device is more realistic as that is also no issue for bootloaders (if those include BTRFS support).
So suppose you have an ARM SBC with small SD-card partition and you want more room for a zypper dup action while keeping all (old) snapshots for the time being, you can plug-in an USB-sick that would then appear as /dev/sda while the SD-card is /dev/mmcblk0, it is then: sudo btrfs device add /dev/sda /
Then when less space is needed: sudo btrfs device remove /dev/sda /
Thanks for the suggestion, I guess that would certainly function.
But I don’t think that this should be necessary with sane installation defaults. Remember: Linux in general and openSUSE specifically is just being promoted as a drop-in replacement for Windows 10 - see the first row on www.opensuse.org *. People coming from Windows do not expect that they have to limit their system partition’s size to prevent it from running full because of restore points. They expect the OS to keep track of the size shadow storage uses and to have an eye on that, but without slowing down the system once every hour to do so. They also do not expect that they might have to add (removable) storage to be able to install or update software.
Without quota on, no one has an idea of the space snapshots take up. But with the present default, having quota enabled unnecessarily slows the whole device 24x a day, when 1x / d would be enough.
Imho, having btrfs and snapshots configured right OOTB is one (of the few remaining cough YAST /cough) features that make openSUSE stand out from other distros, now that e.g. Fedora has promoted KDE Plasma to “edition” status with premier tier support. But btrfs and snapshots are only fun if installation defaults are so carefully configured that everything “just works” without having to be configured and without becoming noticeable during everyday work. Defaults were working nicely imho in openSUSE 15.6 (and 15.5, 15.4, …), so I don’t see the reasons for such a change in openSUSE 16.
*: Even this suggestion seems a bit strange since a PC that can run openSUSE Leap (x86_64 v2) will also run Windows 11 (SSE 4.2), and a PC that’s hard blocked from running Windows 11 also won’t boot openSUSE Leap 16.x.
I don’t know enough about the current situation with regard to installation defaults and snapshot quotas to comment on it, but I do know that x86_64 v2 includes processors as old as Nehalem, released in 2008. Because of the TPM requirement, Windows 11 doesn’t support any Intel processors older than Coffee Lake※, released in 2017!
Sure, there are ways around this, like using Rufus to bypass the requirement checks, but openSUSE supports older hardware out-of-the-box. Windows 11 does not.
I do not really know what defaults are for various openSUSE versions/variants. Long ago when snapper was new and doing automated snapshots, its clean-up config was set to 10 for every timeframe granularity. So 10 hours, 10 days, 10 weeks, 10 months, 10 years. Also hourly timeline snapshotting, not only when doing upgrades. Also in 2013 or so when I started using Btrfs, it was plagued by error -28 (out of space) which could not be fixed by the system itself. So ‘non-boot’ situation, which gave Btrfs the name ‘the filesystem that will eat your data’ as most people were not able to boot with a liveCD or so and use ‘surgery’ with the btrfs tool to fix it. Removing a snapshot would also result in -28 as that needed first more space metadata (as CoW filesystem). So that is also how I learned to add some device with new/extra space first, then no -28 error/crash and data still accessible.
Later in 2022 I did my first new Tumbleweed install on real computer (tablet thing with eMMC, standard Windows10) and saw that at least the hourly timeline snapshotting was not done anymore. I also did several own preferences (just 1 subvolume for whole root). I don’t remember what was put in /etc/snapper/configs/root by default, but I anyway tune that file heavily several times, depending on type of computer and use-case.
I think nowadays the assumption is SSD, either SATA or even much faster NVME. Also then you will clearly notice that snapper in (openSUSE) Tumbleweed is slower than on the same computer when Debian is booted. But also that has to do with defaults w.r.t. snapper which I also disabled mostly (like auto generating diffs). Those are great for developers, but a former Windows user would have no clue what it is and what can be done with it. It burns CPU cycles and drains batteries and noisy fans, that is what people note.
So the installer should probably do quite radical simplification if storage detection ==‘rotational’. Else I don’t know, I am too long a Btrfs user and done many tweaks, such that it also runs on RaspberryPi1 (single core ARMv6 unsupported by most distros). But that runs from SD-card, which does not have the potentially huge latency you will get if HDD. Before Btrfs, I used XFS and ReiserFS for SuSE Linux. Especially XFS is originally great for HDD’s.
