Can't boot, drive full

XFCE is my GUI.

Suddenly I can’t boot. The drive is 100% full. It was at 29% yesterday,

localhost:/home/ion #mount /dev/sdc3  /mnt/in 
localhost:/home/ion #

localhost:/home/ion # df /mnt/in/home/ion
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sdc3       66510264 63903280         0 100% /mnt/in
localhost:/home/ion # 

Normally I would use du to find where it all went but there are thousands of files. Is there a way to determine the largest consumers of disk space? Thanks in advance.

What I usually do from a command-line is use du -h --max=1 | sort -h to look at the current directory and one level down. That provides a good overview as to where the space is, then start drilling into the part of the drive that’s used the most, and iterate down that until I find where the space has gone.

The -h produces human-readable values, and using that on the sort command sorts those intelligently so the smaller directories are at the top, and the larger ones are at the bottom. If you use just a regular sort, because human-readable values are values like “80G”, it sorts using a standard alphanumeric sort, and larger stuff can appear higher in the list because ‘1GB’ appears alphanumerically above ‘1KB’ (for example).

So, you must have done something to fill the partition.

Did you “dd” files from another drive?

Did you copy files from your Nokia to your home-dir as a backup ??

Ran some software that produces tremendous amounts of temp data??

I had the same symptom a few years back, maybe not the same cause, I don’t know. I had Tumbleweed installed and a few weeks later the root disk was filled 100% which prevented the computer from running and booting. This was the first, and only, time I used btrfs. Since then I use ext4 again and have no problems with it.
I can have done something wrong because I don’t understand btrfs but it was a normal install as you would do a TW install. Maybe the root disk needed to be larger with btrfs than when it is formatted as ext4, could well be, but I didn’t, and still don’t, know that.

For (1), that’s because, as a default, when you choose BTRFS, snapshots are created, as time goes on.
Space is required for each and every snapshot. And so, that partition will consume more space over time.

1 Like

Snapper does have some default settings for cleaning up old snapshots based on time and number of snapshot parameters.

That said, I do usually end up cleaning them up by hand anyways.

@ionmich - what filesystem is on this partition, and where is it supposed to be mounted normally?

True … this happened to me back in (201? something) What I can’t quite agree with is

Normally I would use du to find where it all went but there are thousands of files

snapper doesn’t make “thousands of files” sounds like maybe a bad former config that triggered the btrfs COW … question is why now?

That was what I thought as well, but it happened in a matter of weeks. This would then mean I did not make the partition large enough.

I just see @dart364 writing about a bad COW config. Well, as I wrote, I just installed TW and that’s it. I configured nothing related to btrfs and that might be where I went wrong. No idea.

The most important question was brought up by hendersj: which filesystem is used. Because with btrfs, the native df command is the wrong tool to use.
df on btrfs always shows 100% usage. So this might be a red herring, and there are other reasons why the system no longer boots.

(If you have determined your filesystem and used the correct tools for usage calculation:)
It is not unusal that a process or app can run haywire and spams the journal in a short time and fills up all space. That means, also check the file sizes under /var/log/

Sorry … I didn’t see anywhere where you said you “just installed it” … ok then I would guess your root partition is too small try lsblk

windeath:/home/dart/Downloads # lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0 931.5G  0 disk 
├─sda1   8:1    0     8M  0 part 
└─sda2   8:2    0 931.5G  0 part /var
                                 /usr/local
                                 /root
                                 /srv
                                 /home
                                 /opt
                                 /boot/grub2/i386-pc
                                 /boot/grub2/x86_64-efi
                                 /.snapshots
                                 /
sr0     11:0    1  1024M  0 rom  
windeath:/home/dart/Downloads # 

Don‘t get confused. DeMus is not the TO.

Yes, my apologies for breaking in, just wanted to mention I once had a full disk on btrfs as well.
I will be quiet now.

No problem that was actually my mistake … I had the same thing happen back in (201?) and reinstalled … I just don’t use snapper anymore and try not to do something dumb :laughing:

1 Like

It doesn’t show 100% on my system (root is btrfs):

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p4  826G  415G  396G  52% /

But btrfs can make it difficult to determine what the space usage actually is because of how it works. du would also probably give erroneous results.

The correct way to check free space on btrfs is with btrfs filesystem df:

$ btrfs filesystem df /
Data, single: total=439.01GiB, used=393.53GiB
System, DUP: total=32.00MiB, used=80.00KiB
Metadata, DUP: total=18.00GiB, used=10.15GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

Similarly, btrfs filesystem du will give more accurate disk usage statistics, but it doesn’t accept the same options as the regular du command (for example, --max=1).

So, no GUI required. Navigation works like MC.

How about this oneliner I just composed during my morning coffee:

knurpht@Lenovo-P16:~> for i in `ls -d /home/*/`; do sudo btrfs fi du -s $i | tail -1; done
 295.66MiB   275.76MiB     9.90MiB  /home/beheerder/
 410.79GiB   410.36GiB   219.90MiB  /home/knurpht/
knurpht@Lenovo-P16:~> for i in `ls -d /home/knurpht/*/`; do sudo btrfs fi du -s $i | tail -1; done
   3.00GiB     3.00GiB       0.00B  /home/knurpht/Administratie/
  11.60GiB    11.60GiB       0.00B  /home/knurpht/Afbeeldingen/
 981.32MiB   981.32MiB       0.00B  /home/knurpht/bin/
     0.00B       0.00B       0.00B  /home/knurpht/Desktop/
  10.16GiB    10.16GiB       0.00B  /home/knurpht/Documents/
  71.91GiB    71.91GiB     1.96MiB  /home/knurpht/Downloads/
     0.00B       0.00B       0.00B  /home/knurpht/Element/
   8.79GiB     8.79GiB       0.00B  /home/knurpht/Install/
  36.00KiB    36.00KiB       0.00B  /home/knurpht/ModsTeam/
   4.61GiB     4.61GiB       0.00B  /home/knurpht/Movies/
 212.77GiB   212.70GiB    74.73MiB  /home/knurpht/Muziek/
  66.26GiB    66.26GiB       0.00B  /home/knurpht/Nextcloud/
     0.00B       0.00B       0.00B  /home/knurpht/OpenNovations/
   7.70MiB     7.70MiB       0.00B  /home/knurpht/openvino_cpp_samples_build/
 293.14MiB   293.14MiB       0.00B  /home/knurpht/projects/
     0.00B       0.00B       0.00B  /home/knurpht/Rpi3/
   1.47GiB     1.47GiB       0.00B  /home/knurpht/rpmbuild/
  70.10MiB    70.10MiB       0.00B  /home/knurpht/Saturn/
   1.48MiB     1.48MiB       0.00B  /home/knurpht/Schermopnamen/
 656.00KiB   220.00KiB   436.00KiB  /home/knurpht/Temp/
 137.67MiB    62.95MiB    74.73MiB  /home/knurpht/test/
   5.96MiB     5.96MiB       0.00B  /home/knurpht/Webcam/
 348.02MiB   348.02MiB       0.00B  /home/knurpht/Xiltrix/
 819.84MiB   819.84MiB       0.00B  /home/knurpht/XiltrixSaturn/
  44.00KiB    44.00KiB       0.00B  /home/knurpht/XiltrixSaturnDevel/
knurpht@Lenovo-P16:~> 

Edit: of course this could be optimized (show header on line 1 of the output), use awk to have a TOTALS line at the end etc.

No, it is not. filesystem df shows the allocated space. Which itself may be empty or may be full.

The best is btrfs filesystem usage -T which shows both allocated space and how much of it is in use.

Your command simply doesn’t work:
btrfs filesystem usage: not enough arguments: 0 but at least 1 expected

You must indicate subvolume path. For example:

sudo btrfs fi usage /                                                                                                                     1
Overall:
    Device size:                  80.54GiB
    Device allocated:             45.07GiB
    Device unallocated:           35.47GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                         32.04GiB
    Free (estimated):             46.88GiB      (min: 29.14GiB)
    Free (statfs, df):            46.88GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              108.61MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:41.01GiB, Used:29.60GiB (72.18%)
   /dev/nvme0n1p5         41.01GiB

Metadata,DUP: Size:2.00GiB, Used:1.22GiB (61.12%)
   /dev/nvme0n1p5          4.00GiB

System,DUP: Size:32.00MiB, Used:16.00KiB (0.05%)
   /dev/nvme0n1p5         64.00MiB

Unallocated:
   /dev/nvme0n1p5         35.47GiB

Besides, the first thing I would have done is “sudo zypper clean” and then, after deleting snapshots and especially if you have an SSD, enable compression!!!
Before enabling it, run sudo compsize -x / to see the "before" and "after" of compression.. To enable it, edit your “/etc/fstab” file and add “compress=zstd:15” property like in this line:

UUID=5fd2f68b / btrfs compress=zstd:15,noatime,discard 0 0

I have maximum compression enabled for the files I have on “/” which, in my case, is the system partition.

After adding “compress=zstd:15” and rebooting, you’ll have automatic compression enabled for every new file write. To compress existing files, you’ll use “sudo btrfs filesystem defragment -rvf -czstd /

and when it finishes, run “sudo compsize -x /” again and you’ll see the space saved by compression.

Fair enough. Either way, using the traditional du and df utilities is generally not going to give good answers for btrfs filesystems.

And seeing the unallocated space is definitely useful - I hadn’t remembered that usage was a command.

All these Replies provide a lot of wisdom with possible next steps.

I find it odd that the OP, @ionmich, posted this two days ago, with NO follow-up. Are replies too technical ??