The size of / with btrfs filesystem

135246 · February 8, 2018, 11:28am

UPDATE: Oh, sorry, I just realized this is a wrong group. Could anybody move the thread to the right category please?

I ran out of space when doing “zypper dup” recently, and hence I started investigating what the cause is. I deleted btrfs snapshots, /tmp, /var/log, rpm cache,…, ran btrfs balance and restarted the system. This is what I get now. Note that / is mounted on /dev/sda6.

Running the df:


sudo df /dev/sda6 -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda6        41G   32G  8.0G  80% /

Running the btrfs df:


sudo btrfs filesystem df -h /
Data, single: total=30.00GiB, used=28.97GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.50GiB, used=803.84MiB
GlobalReserve, single: total=512.00MiB, used=0.00B

Running the du:


sudo du -sh -x / 
16G    /

Running the btrfs du:

sudo btrfs filesystem du -s --human-readable /
     Total   Exclusive  Set shared  Filename
  32.24GiB    25.38GiB     3.43GiB  /

I called the above commands within the time interval of 30 seconds. In addition, running the last command a couple of times it always gives a different output. I would like to emphasize that the balance is finished, the system was restarted and I am not doing anything else than running these commands and writing on this forum:


sudo btrfs filesystem du -s --human-readable /
     Total   Exclusive  Set shared  Filename
  32.24GiB    14.30GiB    14.51GiB  /
pavel@linux-97ls:~> sudo btrfs filesystem du -s --human-readable /
     Total   Exclusive  Set shared  Filename
  32.24GiB    25.33GiB     3.48GiB  /
pavel@linux-97ls:~> sudo btrfs filesystem du -s --human-readable /
     Total   Exclusive  Set shared  Filename
  32.24GiB    21.57GiB     7.24GiB  /
pavel@linux-97ls:~> sudo btrfs balance status /
No balance found on '/'
pavel@linux-97ls:~> sudo btrfs filesystem du -s --human-readable /
     Total   Exclusive  Set shared  Filename
  32.24GiB    25.38GiB     3.43GiB  /
pavel@linux-97ls:~> sudo btrfs filesystem du -s --human-readable /
     Total   Exclusive  Set shared  Filename
  32.24GiB    14.28GiB    14.53GiB  /

For completeness the current list of btrfs snapshots is

sudo snapper list
Type   | # | Pre # | Date                             | User | Cleanup | Description           | Userdata
-------+---+-------+----------------------------------+------+---------+-----------------------+---------
single | 0 |       |                                  | root |         | current               |         
single | 1 |       | Sat 16 Apr 2016 07:49:27 AM CEST | root |         | first root filesystem |

and I used the following command to balance


sudo btrfs filesystem balance start -dusage=100 /

Could anybody explain to me why the outputs of these programs differ?
Could anybody suggest how to increase the free space so that I do not run into problems when doing zypper dup or making new snapshots?
To sum up, according to the df command I only have 8GB of free space. However, just 12GB are actually used according to du. The / partition has size 41GB. It seems like that there is 21GB of some “filesystem metadata” which I can not control.

Thank you very much for your comments!

ndc33 · February 8, 2018, 1:41pm

if your check space soon after clearing snapshots, it takes a while to clean up.
check if you have any ‘rouge’ snapshots not listed in snapper using-


sudo btrfs subvolume list /

135246 · February 8, 2018, 2:51pm

The command ndc33 suggested gives the following (Although I do not know have any idea what this data mean… I have to study btrfs more):

sudo btrfs subvolume list /
ID 257 gen 484545 top level 5 path .snapshots
ID 258 gen 484882 top level 257 path .snapshots/1/snapshot
ID 259 gen 484712 top level 5 path boot/grub2/i386-pc
ID 260 gen 484550 top level 5 path boot/grub2/x86_64-efi
ID 261 gen 484831 top level 5 path opt
ID 262 gen 484551 top level 5 path srv
ID 263 gen 484880 top level 5 path tmp
ID 264 gen 484839 top level 5 path usr/local
ID 265 gen 484554 top level 5 path var/crash
ID 266 gen 484554 top level 5 path var/lib/libvirt/images
ID 267 gen 484554 top level 5 path var/lib/mailman
ID 268 gen 484554 top level 5 path var/lib/mariadb
ID 269 gen 484554 top level 5 path var/lib/mysql
ID 270 gen 484554 top level 5 path var/lib/named
ID 271 gen 484554 top level 5 path var/lib/pgsql
ID 272 gen 484880 top level 5 path var/log
ID 273 gen 484554 top level 5 path var/opt
ID 274 gen 484882 top level 5 path var/spool
ID 275 gen 484878 top level 5 path var/tmp
ID 1030 gen 484375 top level 5 path var/lib/machines
ID 1912 gen 484712 top level 257 path .snapshots/3/snapshot

I also tried the following:


sudo du -hs /.snapshots/
31G    /.snapshots/

The commands from my previous post still (ca 6 hours after cleaning the snapshots with snapper and doing balance) return the same values (except for btrfs du where Exclusive and Set Shared change).

ndc33 · February 8, 2018, 6:48pm

trying to get the size of snapshots using standard tooling is impossible and misleading.
best command (if you have quotas enabled) is

sudo btrfs qgroup show /

before you removed snapshots approximately what number where you up to? Have you ever done a full rollback? what is result of

sudo btrfs subvolume get-default /

i dont understand why you dont have @ for root, when did you install?

The snapshot number 3 looks suspicious (when i compare to output from my system). but id wait for more knowledgeable person to confirm! [for info: ‘sudo btrfs sub delete’ to remove snapshots not indexed by snapper]

arvidjaar · February 8, 2018, 7:31pm

This snapshot is not listed by snapper. What so content of /.snapshots/3? Is there info.xml file? And what is “btrfs sub get-default /”?

ndc33 · February 8, 2018, 7:45pm

> This snapshot is not listed by snapper.
which is why it looks susicoius

> What so content of /.snapshots/3?
im guessing a rouge snapshot

you didnt answer the question regards last snapshot number. (only old snapshot have large size)

> Is there info.xml file?
?

And what is “btrfs sub get-default /”?
shows your default, try and see.

135246 · February 8, 2018, 8:55pm

Before I deleted the last snapshot I think I was around ~275
I installed Thumbleweed almost two years ago. Since then I created a couple of snapshots myself and I think I rolled back at least once. All using snapper only.
There is no info.xml in /.snapshots/3/.
There is info.xml in /.snapshots/1/. Its content is

<?xml version="1.0"?>
<snapshot>
  <type>single</type>
  <num>1</num>
  <date>2016-04-16 05:49:27</date>
  <description>first root filesystem</description>
</snapshot>

The default snapshot is


sudo btrfs sub get-default /ID 258 gen 485039 top level 257 path .snapshots/1/snapshot

The sizes of snapshots are

sudo btrfs qgroup show /
qgroupid         rfer         excl 
--------         ----         ---- 
0/5          16.00KiB     16.00KiB 
0/257        16.00KiB     16.00KiB 
0/258        14.94GiB     11.54GiB 
0/259         2.38MiB      2.38MiB 
0/260        16.00KiB     16.00KiB 
0/261         2.38GiB      2.38GiB 
0/262         1.19MiB      1.19MiB 
0/263       773.79MiB    773.79MiB 
0/264         1.31MiB      1.31MiB 
0/265        16.00KiB     16.00KiB 
0/266        16.00KiB     16.00KiB 
0/267        16.00KiB     16.00KiB 
0/268        16.00KiB     16.00KiB 
0/269        16.00KiB     16.00KiB 
0/270        16.00KiB     16.00KiB 
0/271        16.00KiB     16.00KiB 
0/272        82.38MiB     82.38MiB 
0/273        16.00KiB     16.00KiB 
0/274        48.00KiB     48.00KiB 
0/275        21.73MiB     21.73MiB 
0/1030       16.00KiB     16.00KiB 
0/1912       14.93GiB     11.53GiB 
1/0          14.93GiB     11.53GiB 
255/1030     16.00KiB     16.00KiB

What snapshots are safe to delete using the command btrfs sub delete? I would like to delete as many as possible (except for the current one of course :-)), then make balance + defragment, and eventually I would like to do zypper dup.

ndc33 · February 8, 2018, 9:53pm

you say you have rolled back but you still on default 1? are you sure it was full rollback? (i thought the default number changed after rollback?)

so the reason i suspect a rouge snapshot can be seen in this thread https://forums.opensuse.org/showthread.php/529132-btrfs-a-mysterious-snapshot-consuming-my-space

for the quota size you can look up qgroup id and compare to the subvolume list to see the size of each snapshot. number 3 is:

0/1912 14.93GiB 11.53GiB

old snapshots get huge (all space consumed will become exclusive - i.e. have no relation to you present files).

the only snapshot thats safe to delete using btrfs subvolume (i assume) would be ones not in snappers index (unless you want issues with snapper).

No 3 looks and sounds (from the number being very low 3) that this is a rouge snapshot.

i would create another snapshot… list with snapper and btrfs subvolumes, inspect the space using qgroups then compare ; try and judge why you have a number 3?, and if you think its rouge - nuke it (but dont blame me if you break you system )

ravas · February 8, 2018, 10:26pm

findmnt
will also show what snapshot you’re starting from (at the top).

Checking for space I’ve been using:
btrfs fi usage /

Having snapshot 1 after more than a year of tumbleweed was eating 15-20gb for me.

I had to roll back because of something unrelated, which allowed me to: snapper rm 1

I wonder if you can just create a snapshot and rollback to it, thus allowing you to remove #1.

For phantom snapshots, see the last two posts here:
https://forums.opensuse.org/showthread.php/529502-btrfs-balance-and-its-friends-won-t-stop-eating-my-CPU-for-20-hours-straight

135246 · February 8, 2018, 10:47pm

EDIT: I wrote the text below before I read your newest posts. I will study your posts now.

I am reading a little bit about btrfs… Is my understanding correct?:

The btrfs filesystem on /dev/sda6 consists of a tree of subvolumes. The root subvolume is denoted by 5 by a convention.There is a parameter “default subvolume” of the filesystem which is the
subvolume (currently 258) which gets mounted on Folder when /dev/sda6 is mounted on Folder (in our case Folder=/). Every subvolume has as parameters its “parent subvolume” and the “relative path” with respect to the parent subvolume (e.g. subvolume 263 has relative path /tmp with respect to its parent subvolume 5, which as a convention points to Folder). When the parent subvolume is mounted, the subvolumes contained in it are accessible in their relative paths as normal directories. In fstab there is a line mounting /dev/sda6 to /, and then various lines mounting subvolumes like 263,etc to /tmp, etc.

As far as I understand, after a clean installation the subvolume 5 is set as default, and hence contains the data. At some point I did a rollback to snapshot 258 (which is a subvolume with parent subvolume 257 and relative path /1/. Here 257 is a subvolume whose parent subvolume is 5 and the relative path is .snapshots) which effectively set the default subvolume to be 258 and (maybe?) created a new snapshot e.g. 259, with the content of 5 in 257. Then I must have deleted 259. After this subvolume 5 still remains as the root subvolume by convention but now empty (?). Then I created some other snapshot 1912 (parent 257, relative path /3/) which somehow got lost from snapper. I conclude that it is safe to remove 1912 but I have to keep all other. OK?

How could 1912 got lost from snapper?

135246 · February 8, 2018, 11:32pm

I think I got it!

I did


sudo snapper rollback
sudo snapper modify -d "Snapshot on 8.2.2018" 5

which created a read only snapshot 4 (copy of 1) and a new rw snapshot 5, set it as the default subvolume and set its description.
I restarted the system and after that I could delete all previous snapshots using snapper:


sudo snapper delete 1-4

This resulted in in the clean state:

Type   | # | Pre # | Date                            | User | Cleanup | Description          | Userdata
-------+---+-------+---------------------------------+------+---------+----------------------+---------
single | 0 |       |                                 | root |         | current              |         
single | 5 |       | Thu 08 Feb 2018 11:03:21 PM CET | root |         | Snapshot on 8.2.2018 |

Then I removed the rogue snapshot not detected by snapper using btrfs subvolume:


sudo btrfs subvolume delete /.snapshots/3/snapshot -c

This resulted in the normal state:

sudo btrfs subvolume list /
ID 257 gen 485213 top level 5 path .snapshots
ID 259 gen 484712 top level 5 path boot/grub2/i386-pc
ID 260 gen 484550 top level 5 path boot/grub2/x86_64-efi
ID 261 gen 485196 top level 5 path opt
ID 262 gen 484551 top level 5 path srv
ID 263 gen 485216 top level 5 path tmp
ID 264 gen 485197 top level 5 path usr/local
ID 265 gen 484554 top level 5 path var/crash
ID 266 gen 484554 top level 5 path var/lib/libvirt/images
ID 267 gen 484554 top level 5 path var/lib/mailman
ID 268 gen 484554 top level 5 path var/lib/mariadb
ID 269 gen 484554 top level 5 path var/lib/mysql
ID 270 gen 484554 top level 5 path var/lib/named
ID 271 gen 484554 top level 5 path var/lib/pgsql
ID 272 gen 485218 top level 5 path var/log
ID 273 gen 484554 top level 5 path var/opt
ID 274 gen 485217 top level 5 path var/spool
ID 275 gen 485218 top level 5 path var/tmp
ID 1030 gen 484375 top level 5 path var/lib/machines
ID 2060 gen 485218 top level 257 path .snapshots/5/snapshot

I am running btrfs balance now and will post the filesystem sizes after it finishes. But I hope it will be OK.

Thank you very much for your help! I learned a lot!

135246 · February 9, 2018, 12:23am

Now the btrfs df size looks much better (Recall that sudo du -shx / gives 16G)

sudo btrfs filesystem df /
Data, single: total=19.00GiB, used=17.72GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.50GiB, used=443.06MiB
GlobalReserve, single: total=512.00MiB, used=0.00B

ravas · February 9, 2018, 12:42am

I’m curious about:
btrfs fi usage /

135246 · February 9, 2018, 12:56am

Does it look OK? There is still ~2.5GB discrepancy which I imagine is caused by the metadata.

sudo btrfs fi usage /
Overall:
    Device size:          40.00GiB
    Device allocated:          22.06GiB
    Device unallocated:          17.94GiB
    Device missing:             0.00B
    Used:              18.58GiB
    Free (estimated):          19.22GiB    (min: 10.25GiB)
    Data ratio:                  1.00
    Metadata ratio:              2.00
    Global reserve:         512.00MiB    (used: 0.00B)


Data,single: Size:19.00GiB, Used:17.72GiB
   /dev/sda6      19.00GiB


Metadata,DUP: Size:1.50GiB, Used:443.09MiB
   /dev/sda6       3.00GiB


System,DUP: Size:32.00MiB, Used:16.00KiB
   /dev/sda6      64.00MiB


Unallocated:
   /dev/sda6      17.94GiB

ravas · February 9, 2018, 1:20am

Free (estimated): 19.22GiB (min: 10.25GiB)

I’d be interested to know why the ‘min’ is so much lower.
Either way, that will likely be enough for your next zypper dup.

If you need more space you could try removing old kernels:
systemctl start purge-kernels.service

You can check the size of the systemd journal:
journalctl --disk-usage

If it’s very large you could try limiting it in /etc/systemd/journald.conf
The manjaro wiki suggests creating user specific files – although, I haven’t tried that.

https://doc.opensuse.org/documentation/leap/reference/html/book.opensuse.reference/cha.journalctl.html

arvidjaar · February 9, 2018, 5:14am

Free space depends on chunk profile (single/dup) and it is something user is free to change at any time (and it is per chunk, so you can have both single and dup chunks at the same time). Minimum is the worst case with all chunks having dup profile where all data is duplicated.

ndc33 · February 9, 2018, 6:35am

> Having snapshot 1 after more than a year of tumbleweed was eating 15-20gb for me.
this is a red herring. snapshot 1 (without rollback) is your current file system. There was no need to do a rollback and clear it (the way it is labelled in snapper is misleading). The 10gb or so you recovered came from clearing snapshot 3. Good to see you recovered 10gb or so of space. Not sure about the discrepancy with min, I assume your on spinning rust where the default is duplicated metadata? (im on ssd and so not familiar). sudo btrfs fi usage / is the better way for checking space used (as can be seen).

ravas · February 9, 2018, 7:14am

@arvidjaar

Thanks; where can I read more about “it is something user is free to change at any time”?

ravas · February 9, 2018, 7:18am

@ndc33

Thanks.
I’m confused.

“Snapshot #0 always refers to the current system.”

-> http://snapper.io/tutorial.html

“snapshot 1 (without rollback) is your current file system.”

-> you

What’s the difference?

ndc33 · February 9, 2018, 9:20am

from non expert recollection i believe snap 0 is actually just a sim link (i.e. not files, not snapshot).
[getting less sure…] 1 (without rollback) is current state of the system (holds changes made since the the read-only snapshots) and is what snap 0 link actually points to.