SDA2 Full - Is it?

Hi,

My OpenSuse 15.1 server running LAMP configuration and samba spat me an error this morning. On closer inspection

df -h

it seems SDA2 is full.

CliffServ:/ # df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.9G  8.0K  7.9G   1% /dev
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           7.9G   50M  7.8G   1% /run
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda2        41G   39G     0 100% /
/dev/sda2        41G   39G     0 100% /srv
/dev/sda2        41G   39G     0 100% /tmp
/dev/sda2        41G   39G     0 100% /opt
/dev/sda2        41G   39G     0 100% /root
/dev/sda2        41G   39G     0 100% /boot/grub2/x86_64-efi
/dev/sda2        41G   39G     0 100% /usr/local
/dev/sda2        41G   39G     0 100% /var
/dev/sda2        41G   39G     0 100% /boot/grub2/i386-pc
/dev/sdb1       1.4T  3.7G  1.4T   1% /home
/dev/sda3       1.8T  108G  1.7T   6% /big
/dev/sdc1       466G  108G  358G  24% /exthdd
tmpfs           1.6G     0  1.6G   0% /run/user/0

For the life of me I cannot find any major GB consuming files on SDA2. /exthdd is an external drive which rsync backs up to every night. /home and /big are on separate partitions. I’ve checked for ‘snapshots’, logs, /tmp, /var/tmp without success.

mount

gives me:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=8182920k,nr_inodes=2045730,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
/dev/sda2 on / type btrfs (rw,relatime,space_cache,subvolid=256,subvol=/@)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=26,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16984)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/sda2 on /srv type btrfs (rw,relatime,space_cache,subvolid=263,subvol=/@/srv)
/dev/sda2 on /tmp type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/@/tmp)
/dev/sda2 on /opt type btrfs (rw,relatime,space_cache,subvolid=265,subvol=/@/opt)
/dev/sda2 on /root type btrfs (rw,relatime,space_cache,subvolid=264,subvol=/@/root)
/dev/sda2 on /boot/grub2/x86_64-efi type btrfs (rw,relatime,space_cache,subvolid=259,subvol=/@/boot/grub2/x86_64-efi)
/dev/sda2 on /usr/local type btrfs (rw,relatime,space_cache,subvolid=261,subvol=/@/usr/local)
/dev/sda2 on /var type btrfs (rw,relatime,space_cache,subvolid=260,subvol=/@/var)
/dev/sda2 on /boot/grub2/i386-pc type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/@/boot/grub2/i386-pc)
/dev/sdb1 on /home type xfs (rw,relatime,attr2,inode64,noquota)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/sda3 on /big type xfs (rw,relatime,attr2,inode64,noquota)
/dev/sdc1 on /exthdd type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=1640148k,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)

du -sh *

gives me:

CliffServ:/ # du -sh *
106G    big
2.1M    bin
86M     boot
8.0K    dev
23M     etc
108G    exthdd
2.3G    home
5.7M    info
712M    lib
11M     lib64
0       mnt
0       opt
du: cannot access 'proc/13119/task/13119/fd/4': No such file or directory
du: cannot access 'proc/13119/task/13119/fdinfo/4': No such file or directory
du: cannot access 'proc/13119/fd/4': No such file or directory
du: cannot access 'proc/13119/fdinfo/4': No such file or directory
0       proc
192K    root
50M     run
10M     sbin
0       selinux
37M     srv
0       sys
4.0K    tmp
4.8G    usr
1.1G    var

Can anyone shed any light on this for me?

Many thanks,
Jim

/dev/sda2 seems to have a btrfs file system and for that use:

$ sudo btrfs filesystem show
Label: none  uuid: 6086e423-5708-4b18-935f-9716a8f3900a
        Total devices 1 FS bytes used 17.09GiB
        devid    1 size 22.32GiB used 22.32GiB path /dev/sdc2

For more details use:

$ btrfs filesystem usage /
Overall:
    Device size:                  22.32GiB
    Device allocated:             22.32GiB
    Device unallocated:            1.00MiB
    Device missing:                  0.00B
    Used:                         16.69GiB
    Free (estimated):              5.26GiB      (min: 5.26GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:               54.64MiB      (used: 0.00B)

Data,single: Size:21.26GiB, Used:16.00GiB
   /dev/sdc2      21.26GiB

Metadata,single: Size:1.03GiB, Used:701.47MiB
   /dev/sdc2       1.03GiB

System,single: Size:32.00MiB, Used:16.00KiB
   /dev/sdc2      32.00MiB

Unallocated:
   /dev/sdc2       1.00MiB

This is the readout from

btrfs filesystem usage /

:

CliffServ:/ # btrfs filesystem usage /Overall:
    Device size:                  40.00GiB
    Device allocated:             40.00GiB
    Device unallocated:            1.00MiB
    Device missing:                  0.00B
    Used:                         38.45GiB
    Free (estimated):                0.00B      (min: 0.00B)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               61.25MiB      (used: 0.00B)


Data,single: Size:37.72GiB, Used:37.72GiB
   /dev/sda2      37.72GiB


Metadata,DUP: Size:1.13GiB, Used:373.58MiB
   /dev/sda2       2.26GiB


System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sda2      16.00MiB


Unallocated:
   /dev/sda2       1.00MiB

You said you checked for snapshots, but you did not show that (we almost never believe what is said, we want to see where the saying is based on).

(BTW, this is about sda2, not SDA2).

I’ve got no /.snapshots directory, or no configuration for snapshots in /etc/snapper/configs/

Bit of newbie scenario, but I can’t figure out where the disk space has gone for /dev/sda2

CliffServ:/ # du -sh *
106G    big
2.1M    bin
86M     boot
8.0K    dev
23M     etc
108G    exthdd
2.3G    home
5.7M    info
712M    lib
11M     lib64
0       mnt
0       opt
du: cannot access 'proc/2608/task/2608/fd/4': No such file or directory
du: cannot access 'proc/2608/task/2608/fdinfo/4': No such file or directory
du: cannot access 'proc/2608/fd/4': No such file or directory
du: cannot access 'proc/2608/fdinfo/4': No such file or directory
0       proc
196K    root
50M     run
10M     sbin
0       selinux
37M     srv
0       sys
4.0K    tmp
4.8G    usr
1.1G    var

So yes, /dev/sda2 is almost full.

You “du” lists also other stuff, can you run “sudo du -shx /”

Can you check what is on SDB:Cleanup_system and SDB:Disk_space.

CliffServ:/ # du -shx /
5.7G    /



Regarding the cleanup:

CliffServ:/ # journalctl --vacuum-time=1dVacuuming done, freed 0B of archived journals from /run/log/journal/90e5e2d316f349a4879a1dabad67a3fb.
Vacuuming done, freed 0B of archived journals from /run/log/journal/ba41582d5675444eb021d35b5fcfb774.
CliffServ:/ # zypper clean
All repositories have been cleaned up.
CliffServ:/etc/zypp # zypper purge-kernelsUnknown command 'purge-kernels'
Type 'zypper help' to get a list of global options and commands.


In case 'purge-kernels' is not a typo it's probably not a built-in command, but provided as a subcommand or plug-in (see 'zypper help subcommand').
In this case a specific package providing the subcommand needs to be installed first. Those packages are often named 'zypper-purge-kernels' or 'zypper-purge-kernels-plugin'.
CliffServ:/etc/zypp # snapper cleanup 1The config 'root' does not exist. Likely snapper is not configured.
See 'man snapper' for further instructions.

Regarding diskspace:

CliffServ:/etc/zypp # btrfs fi usage /Overall:
    Device size:                  40.00GiB
    Device allocated:             40.00GiB
    Device unallocated:            1.00MiB
    Device missing:                  0.00B
    Used:                         38.45GiB
    Free (estimated):              1.91MiB      (min: 1.91MiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               61.25MiB      (used: 0.00B)


Data,single: Size:37.72GiB, Used:37.72GiB
   /dev/sda2      37.72GiB


Metadata,DUP: Size:1.13GiB, Used:373.58MiB
   /dev/sda2       2.26GiB


System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sda2      16.00MiB


Unallocated:
   /dev/sda2       1.00MiB
CliffServ:/etc/zypp # snapper listThe config 'root' does not exist. Likely snapper is not configured.
See 'man snapper' for further instructions.
CliffServ:/ # btrfs subvolume list /ID 256 gen 4475567 top level 5 path @
ID 258 gen 4366996 top level 256 path boot/grub2/i386-pc
ID 259 gen 4366996 top level 256 path boot/grub2/x86_64-efi
ID 260 gen 4475572 top level 256 path var
ID 261 gen 4400039 top level 256 path usr/local
ID 262 gen 4475572 top level 256 path tmp
ID 263 gen 4401077 top level 256 path srv
ID 264 gen 4475568 top level 256 path root
ID 265 gen 4366996 top level 256 path opt
ID 270 gen 4398017 top level 260 path var/lib/machines
CliffServ:/ # journalctl --disk-usage
Archived and active journals take up 48.0M in the file system.
CliffServ:/ # /usr/bin/systemd-tmpfiles --clean[/usr/lib/tmpfiles.d/ddclient.conf:1] Line references path below legacy directory /var/run/, updating /var/run/ddclient → /run/ddclient; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/munge.conf:1] Line references path below legacy directory /var/run/, updating /var/run/munge → /run/munge; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/net-snmp.conf:1] Line references path below legacy directory /var/run/, updating /var/run/net-snmp → /run/net-snmp; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/radvd.conf:1] Line references path below legacy directory /var/run/, updating /var/run/radvd → /run/radvd; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/salt.conf:2] Line references path below legacy directory /var/run/, updating /var/run/salt → /run/salt; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/salt.conf:3] Line references path below legacy directory /var/run/, updating /var/run/salt/master → /run/salt/master; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/salt.conf:4] Line references path below legacy directory /var/run/, updating /var/run/salt/minion → /run/salt/minion; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/samba.conf:1] Line references path below legacy directory /var/run/, updating /var/run/samba → /run/samba; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/slurm.conf:2] Line references path below legacy directory /var/run/, updating /var/run/slurm → /run/slurm; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/svnserve.conf:1] Line references path below legacy directory /var/run/, updating /var/run/svnserve → /run/svnserve; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/tmp.conf:13] Duplicate line for path "/var/tmp", ignoring.
[/usr/lib/tmpfiles.d/var.conf:21] Duplicate line for path "/var/lib", ignoring.
[/usr/lib/tmpfiles.d/var.conf:23] Duplicate line for path "/var/spool", ignoring.

Did you run “du -shx /” as root? For me there is a mismatch between “df -h” and “sudo du -shx /”, 25 Gb versus 23 Gb but that is XFS.

I am not a BTRFS expert, but you might want to have a look at Fixing Btrfs Filesystem Full Problems.

Also quite some good information on Ubuntu thinks Btrfs disk is full but it’s not

Will have a read thanks.

Is this looking to be a drive issue then? I’m not massively experienced in Linux, and so cant understand how btrfs fi usage / says I’m all out of space, yet du -sh * tells me there are no majorly big directories.

Might just backup and reinstall tbh.

As far as I know that is a basic panic action used on MS Windows. One then still does not know what and why and will probably run into the same problem in due time.

True, I agree with your comment. But unfortunately I don’t know what else to do.

I can’t locate any files on /dev/sda2 which are consuming disk space. According to the checks I’ve done I shouldn’t be using more than 20gb. So where has the rest of it gone?

My inexperience here let’s me down. I’m a mechanical engineer and welder with basic Linux experience.

My only option is to reinstall and reconfigure, because tomorrow morning when my staff come in I need to have the server running in order for the business to function smoothly.

#underpressure

I understand. Good luck.

Just in case that happens again:
**
BTRFS and free space - emergency response**

“I run BTRFS on my root filesystem (on Linux), mostly for the quick snapshot and restore functionality. Yesterday I ran into a common problem: my drive was suddenly full.”

Thanks, I’ll have a read

Hi Folks,

I managed to free up 200mb last night which allowed me to get LAMP, samba share and webmin up and running for today. This extra space also allowed me to run a btrfs balance command:

CliffServ:/ # btrfs balance start -dusage=88 /
Done, had to relocate 1 out of 42 chunks

However, this has only freed up a couple of extra MB. So, as a short term fix, I’ve added a 16gb USB stick to the btrfs filesystem.

CliffServ:/ # btrfs fi usage /Overall:
    Device size:                  54.46GiB
    Device allocated:             40.72GiB
    Device unallocated:           13.74GiB
    Device missing:                  0.00B
    Used:                         38.39GiB
    Free (estimated):             14.73GiB      (min: 7.86GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               60.59MiB      (used: 0.00B)


Data,single: Size:38.65GiB, Used:37.66GiB
   /dev/sda2      37.65GiB
   /dev/sdd1       1.00GiB


Metadata,DUP: Size:1.00GiB, Used:371.72MiB
   /dev/sda2       2.00GiB


System,DUP: Size:32.00MiB, Used:16.00KiB
   /dev/sda2      64.00MiB


Unallocated:
   /dev/sda2     293.00MiB
   /dev/sdd1      13.45GiB
CliffServ:/ # btrfs fi df /Data, single: total=38.65GiB, used=37.66GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=371.73MiB
GlobalReserve, single: total=60.61MiB, used=0.00B

Moving forward, I would like to try and resolve this issue. But I’m confused as to what is filling sda2.

  1. I have little or no /tmp or /var/tmp files
  2. Snapshots disabled
  3. Balance has freed negligible space
  4. According to ‘du -shx /’ I’ve only used 5.7gb on sda2 (root partition)
  5. I can’t find any directories / files on sda2 using major space

Running the balance should have converted unused allocation into more storage for data, but this has not happened.

Any other thoughts?

Thanks in advance

It is a bit beside your present problem, but as soon as you have a rather stable situation, I would suggest you upgrade to 51.2 asap and then prepare the up grade to 15.3 within some months.

Yes, why sda2 is full is weird.

Can you run as root:

du -hs /
du -hs /srv
du -hs /tmp
du -hs /opt
du -hs /root
du -hs /boot/grub2/x86_64-efi
du -hs /usr/local
du -hs /var
du -hs /boot/grub2/i386-pc

Searching for “du df mismatch” I found:

  • Run fsck
  • Is a btrfs subvolume not mounted?
  • Boot from USB, mount /sda2 and then do “du”
  • Check “df -i”

Regarding “Is a btrfs subvolume not mounted?”, please post the output of “sudo btrfs subvolume list -p /”