btrfs frequently running out of space with lots unallocated

English Install/Boot/Login
#1

Every time I do a system update, I run out of space on my root partition.

btrfs is hellbent on using only 74GB out of 323GB total available on the partition so I have to then start deleting snapshots to try and complete the update.
There must be something I’m missing so I’m hoping someone has any suggestions.

Here’s some output which might help diagnose this:


# btrfs device usage /
/dev/mapper/cr_root, ID: 1
   Device size:           323.00GiB
   Device slack:              0.00B
   Data,single:            74.00GiB
   Metadata,single:         1.00GiB
   System,single:          32.00MiB
   Unallocated:           247.97GiB



# btrfs fi show /
Label: none  uuid: fe9cc6a2-a504-4af7-8728-785068ad6dda
        Total devices 1 FS bytes used 73.44GiB
        devid    1 size 323.00GiB used 75.03GiB path /dev/mapper/cr_root



# blockdev --report
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  1024   512  4096          0    512110190592   /dev/nvme0n1
rw  1024   512   512       2048       272629760   /dev/nvme0n1p1
rw  1024   512  4096     534528        16777216   /dev/nvme0n1p2
rw  1024   512  4096     567296    161061273600   /dev/nvme0n1p3
rw  1024   512  4096  998166528      1048576000   /dev/nvme0n1p4
rw  1024   512  4096  315140096    346818609152   /dev/nvme0n1p5
rw  1024   512  4096  993972224      2147483648   /dev/nvme0n1p6
rw  1024   512   512  992520192       743440384   /dev/nvme0n1p7
rw  1024   512  4096          0    346816512000   /dev/dm-0

I tried rebalancing to see what happens but the process ran out of space


# btrfs filesystem balance /              
ERROR: error during balancing '/': No space left on device                                                                    
There may be more info in syslog - try dmesg | tail
#2

I have seen some reports that there is a kernel bug (for “btrfs”) involved in this.

#3

It sounds similar to this: https://bugzilla.kernel.org/show_bug.cgi?id=74101
But there’s no help there and it appears to span multiple kernel versions (from 3 to, in my case, 5). Might be totally different issues with the same symptom.

#4

https://old.reddit.com/r/openSUSE/comments/ezbv5r/strange_issue_with_btrfs_partition/fgon5bt/

#5

You may provide additional information:

erlangen:~ # btrfs filesystem usage -T /
Overall:
    Device size:                  59.45GiB
    Device allocated:             15.03GiB
    Device unallocated:           44.42GiB
    Device missing:                  0.00B
    Used:                         13.08GiB
    Free (estimated):             44.83GiB      (min: 44.83GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:               37.11MiB      (used: 0.00B)

             Data     Metadata  System              
Id Path      single   single    single   Unallocated
-- --------- -------- --------- -------- -----------
 1 /dev/sdb5 13.00GiB   2.00GiB 32.00MiB    44.42GiB
-- --------- -------- --------- -------- -----------
   Total     13.00GiB   2.00GiB 32.00MiB    44.42GiB
   Used      12.59GiB 502.31MiB 16.00KiB            
erlangen:~ #

Running “btrfs filesystem balance /” never worked on my machine. I always use “btrfs balance start -musage=50 /” and “btrfs balance start -musage=50 /”. See also: https://ohthehugemanatee.org/blog/2019/02/11/btrfs-out-of-space-emergency-response/

#6

this is fixed since upstream kernel 5.5.2 and Tumbleweed kernel 5.5.1.

#7

After running balance with the suggested musage (and I presume dusage):


# btrfs filesystem usage -T /
Overall:
    Device size:                 323.00GiB
    Device allocated:             75.03GiB
    Device unallocated:          247.97GiB
    Device missing:                  0.00B
    Used:                         73.94GiB
    Free (estimated):            248.94GiB      (min: 248.94GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              126.08MiB      (used: 0.00B)

                       Data     Metadata  System              
Id Path                single   single    single   Unallocated
-- ------------------- -------- --------- -------- -----------
 1 /dev/mapper/cr_root 74.00GiB   1.00GiB 32.00MiB   247.97GiB
-- ------------------- -------- --------- -------- -----------
   Total               74.00GiB   1.00GiB 32.00MiB   247.97GiB
   Used                73.03GiB 929.42MiB 16.00KiB

In fact, after running balance, I have 0 available space once again.

@arvidjaar, you have a link to the issue? Looks like the latest snapshot installs kernel 5.5.2 so I’ll give that a try. Once I somehow clear enough space that is.
Otherwise it’s a wipe and move to ext4 but I’m not looking forward to the time spent doing that.

#8

You may try the following command:

systemctl start btrfs-balance.service

After completion display:


erlangen:~ # journalctl -u btrfs-balance.service 
-- Logs begin at Fri 2020-02-14 07:59:41 CET, end at Sun 2020-02-16 17:42:02 CET. --
Feb 15 09:34:45 erlangen systemd[1]: Started Balance block groups on a btrfs filesystem.
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Before balance of /
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Data, single: total=13.00GiB, used=12.45GiB
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: System, single: total=32.00MiB, used=16.00KiB
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Metadata, single: total=1.00GiB, used=501.11MiB
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: GlobalReserve, single: total=35.42MiB, used=0.00B
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: /dev/sdb5        64G     14G   50G   23% /
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Done, had to relocate 0 out of 15 chunks
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Dumping filters: flags 0x1, state 0x0, force is off
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]:   DATA (flags 0x2): balancing, usage=5
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Done, had to relocate 0 out of 15 chunks
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Dumping filters: flags 0x1, state 0x0, force is off
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]:   DATA (flags 0x2): balancing, usage=10
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Done, had to relocate 0 out of 15 chunks
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Done, had to relocate 0 out of 15 chunks
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Dumping filters: flags 0x6, state 0x0, force is off
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]:   METADATA (flags 0x2): balancing, usage=3
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]:   SYSTEM (flags 0x2): balancing, usage=3
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Done, had to relocate 1 out of 15 chunks
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: After balance of /
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Data, single: total=13.00GiB, used=12.45GiB
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: System, single: total=32.00MiB, used=16.00KiB
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Metadata, single: total=1.00GiB, used=501.14MiB
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: GlobalReserve, single: total=35.44MiB, used=0.00B
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
Feb 15 09:34:45 erlangen btrfs-balance.sh[20518]: /dev/sdb5        64G     14G   50G   23% /
Feb 15 09:34:45 erlangen systemd[1]: btrfs-balance.service: Succeeded.
erlangen:~ #
#9

Nevermind. I see the links are in the reddit thread mentioned earlier.

Thanks guys. Fingers crossed that’s the issue I’m experiencing as well.

#10

In case you’re curious, it’s still a mess:


Feb 16 12:04:30 thinkpad systemd[1]: Started Balance block groups on a btrfs filesystem.
Feb 16 12:04:30 thinkpad btrfs-balance.sh[25953]: Before balance of /
Feb 16 12:04:30 thinkpad btrfs-balance.sh[25953]: Data, single: total=73.00GiB, used=58.86GiB
Feb 16 12:04:30 thinkpad btrfs-balance.sh[25953]: System, single: total=32.00MiB, used=16.00KiB
Feb 16 12:04:30 thinkpad btrfs-balance.sh[25953]: Metadata, single: total=1.00GiB, used=910.11MiB
Feb 16 12:04:30 thinkpad btrfs-balance.sh[25953]: GlobalReserve, single: total=110.73MiB, used=0.00B
Feb 16 12:04:30 thinkpad btrfs-balance.sh[25953]: Filesystem           Size  Used Avail Use% Mounted on
Feb 16 12:04:30 thinkpad btrfs-balance.sh[25953]: /dev/mapper/cr_root  347G   65G     0 100% /
Feb 16 12:04:30 thinkpad btrfs-balance.sh[25953]: Done, had to relocate 0 out of 75 chunks
Feb 16 12:04:33 thinkpad btrfs-balance.sh[25953]: Dumping filters: flags 0x1, state 0x0, force is off
Feb 16 12:04:33 thinkpad btrfs-balance.sh[25953]:   DATA (flags 0x2): balancing, usage=5
Feb 16 12:04:33 thinkpad btrfs-balance.sh[25953]: Done, had to relocate 9 out of 75 chunks
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]: Dumping filters: flags 0x1, state 0x0, force is off
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]:   DATA (flags 0x2): balancing, usage=10
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]: Done, had to relocate 1 out of 66 chunks
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]: Done, had to relocate 0 out of 65 chunks
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]: Dumping filters: flags 0x6, state 0x0, force is off
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]:   METADATA (flags 0x2): balancing, usage=3
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]:   SYSTEM (flags 0x2): balancing, usage=3
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]: Done, had to relocate 1 out of 65 chunks
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]: After balance of /
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]: Data, single: total=63.00GiB, used=58.86GiB
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]: System, single: total=32.00MiB, used=16.00KiB
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]: Metadata, single: total=1.00GiB, used=910.09MiB
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]: GlobalReserve, single: total=110.70MiB, used=0.00B
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]: Filesystem           Size  Used Avail Use% Mounted on
Feb 16 12:04:34 thinkpad btrfs-balance.sh[25953]: /dev/mapper/cr_root  347G   65G     0 100% /
Feb 16 12:04:34 thinkpad systemd[1]: btrfs-balance.service: Succeeded.

I deleted some personal files just to make up enough space for the update. The result was it just runs out of space sooner (at 60 gigs instead of 74).

After some more snapshot deletions and collecting a few megabytes along the way, the update finally managed to finish. Now running 5.5.2 and hoping this goes away.

#11

Hm! Did you tinker with LVM?

#12

Looks like the newest kernel fixed this. I copied over some large files and those were done without running out of space again. The unallocated space was successfully shrunk as the filesystem worked to accommodate the files.

Thanks everyone.

#13

Fixing Btrfs Filesystem Full Problems

#14

This is absolutely unrelated to the issue discussed in this thread. If anything, balancing metadata increased probability to hit this issue.

#15

The original poster says: “For the record, a « btrfs balance start -m » apparently fixed it.”

Kernel 5.4 - BTRFS FS shows full with about 600 GB Free ?

#16

Hi all,

I am sorry, but I have to reopen this thread.
I am lost with BTRFS as I am experiencing the same issue as described above.

I have a very simple setup, just one hard drive with / on BTRFS
/home on BTRFS

Opensuse is Tumbleweed with latest updates installed:Operating System: openSUSE Tumbleweed 20210104
KDE Plasma Version: 5.20.4
KDE Frameworks Version: 5.77.0
Qt Version: 5.15.2
Kernel Version: 5.10.4-1-default
OS Type: 64-bit
Processors: 8 × AMD Ryzen 7 4700U with Radeon Graphics
Memory: 7.1 GiB of RAM
Graphics Processor: AMD RENOIR

Let’s focus on / only.


sudo btrfs filesystem usage /
Overall:
    Device size:                 167.00GiB
    Device allocated:             17.03GiB
    Device unallocated:          149.97GiB
    Device missing:                  0.00B
    Used:                         13.41GiB
    Free (estimated):            152.01GiB      (min: 152.01GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:               40.11MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:15.00GiB, Used:12.96GiB (86.40%)
   /dev/nvme0n1p2         15.00GiB

Metadata,single: Size:2.00GiB, Used:465.47MiB (22.73%)
   /dev/nvme0n1p2          2.00GiB

System,single: Size:32.00MiB, Used:16.00KiB (0.05%)
   /dev/nvme0n1p2         32.00MiB

Unallocated:
   /dev/nvme0n1p2        149.97GiB

As you can see there is plenty of space but unallocated and I am running out of space for Data,single (here: already at 86.4%).
Balance did not help, it was even worse after executing.
I was thinking that BTRFS is managing the allocated / unallocated space itself, but I might be wrong.
In different forums I got that this might be a bug. However, maybe it is a feature, and I am simply unable to find the right command to get rid of the unassigned range.

Since I am running the newest Kernel the bug (if so) should be fixed. But why I do have this issue???

Is there anyone which can help me?

#17

Which issue? You did not show any evidence of any problem. Show error message and explain when you see this error message.

#18

I was thinking that BTRFS is managing the allocated / unallocated space itself, but I might be wrong.

You are not wrong :wink:

You can manage it yourself, but I don’t see any reason to. See “Resize Filesystem” in

https://blog.programster.org/btrfs-cheatsheet

#19

I am sorry that I was silent the last days, but I was investigating this further.

It seems that I was mistaken (or an update solved the issue).
However, the allocated space is handled now in the right way, and I am not running any longer into memory issues.
I checked the behavior for different scenarios:

  • Checked the memory allocation while copying a large file to a location under /
  • Checked the memory allocation before and after large updates via zypper dup

So, I guess we can set this thread on hold, and I hope there is no need to reopen it in the future. :wink:
Many thanks for your help!