BTRFS qgroup reserved space underflow

English Install/Boot/Login
1

I’ve updated a bare metal 42.2 machine to 42.3 via this guide https://en.opensuse.org/SDB:System_upgrade . It went pretty smooth, except occasionally now I experience the computer freezing for a few seconds. I checked the journal and found this error http://susepaste.org/92951715 . I let the machine run overnight and this error is repeated approximately every 15 minutes. I looked back in the journal just before the upgrade and earlier for this error, but it is not present. - So it is something to do with the upgrade.

My filesystem is not full.

firechief422:/etc/snapper/configs # btrfs fi df /Data, single: total=33.01GiB, used=28.13GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=2.50GiB, used=1.03GiB
GlobalReserve, single: total=89.86MiB, used=0.00B

firechief422:/etc/snapper/configs # btrfs fi show /
Label: none  uuid: b981f08e-efed-4b65-9ca7-dd1017e5fa6f
        Total devices 1 FS bytes used 29.16GiB
        devid    1 size 60.00GiB used 38.07GiB path /dev/sda1

Here is what the quota groups are http://susepaste.org/8794379 .

How do I fix this?

Thanks

Mark

2

I’m also seeing this on an updated machine.
It’s always the same qgroup, for what that’s worth, and that qgroup appears to be attached to /tmp.
I’ve done a quota rescan and an attempt to rebalance on /tmp
That seems to have helped the slowdowns (I’ve not experienced the lockup for a while) but the warning/error still keeps coming at regular and short intervals)

3

some simple things: have you run a scrub? perhaps also delete old snapshots and re-balance (i.e. reduce the problem to a minimum).
note there is a kernel patch from feb which addresses an issue similar to the title.

4

I did perform a quota rescan, but that did not fix the issue.

I disabled the quotas with:

# btrfs quota disable /

The errors in the journal are gone, but I don’t know what the ramifications of disabling quotas are.

I wonder whether quotas were enabled in 42.2 or did something change in the quota settings? This installation of 42.2 was a direct install from DVD & not upgraded from 42.1 or 13.2.

Mark

5

Quotas are used to implement snapshot deletion policy based on free space percentage. This aims to alleviate out of space issues often seen in the past. As long as you are careful, monitor you root filesystem for free space and know how to handle out of space condition, quota can probably remain disabled.

6

if you keep quotas disabled i ‘think’ its required to change the snapshot ‘range’ limits in snapper config to fixed limits (check). maybe try renabling quotas since your ‘bad references’ may now be gone? P.S. scrub != quota rescan. (its a safe online fsck type procedure for btrfs)

7

I had the same issue on Leap 42.2 driving me insane.
I had the following lines in /etc/tmpfiles.d/tmp.conf


# D create and empty on boot, 7d expire
D /tmp 1777 root root 7d
# d create on boot, 7d expire
d /var/tmp 1777 root root 7d

Changing them into:


q /tmp 1777 root root 7d
q /var/tmp 1777 root root 7d

fixed the reserved space underflow of the /tmp subvolume for me.
(Be it by loosing the empty on boot of /tmp provided by D).

8

@wezer Thanks, I don’t have any files in /etc/tmpfiles.d/

I see that someone created a bug report https://bugzilla.opensuse.org/show_bug.cgi?id=1051052 for this same issue.

I see that there was a patch submitted in May to the kernel for this problem. https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg63896.html , I looked at the changelog for kernel-default and don’t see where this patch has been applied.

Mark

9

We also have the same issue after upgrading one of our servers from 42.2 to 42.3 but instead of lacks the system becomes completely unresponsive, even the keyboard LEDs don’t give a signal any more. The underlying disk consists of two SSDs in a hardware-Raid1.
There are no other related errors in the system log except the “BTRFS warning (device sda1): qgroup 265 reserved space underflow, have: 471040, to free: 475136” messages.

Is there anyone who can tell me more about the consequences of turning quotas off for snapshots and how exactly I can avoid to fill the disk?

Thanks in advance.

10

Provide sufficient space for BTRFS snapshots. Min is really 40 gig. More is better.

To mod snapshot behavior

https://en.opensuse.org/Portal:Snapper