no space left on device(28)

Hello,
using opensuse 42.3 on a 100gb drive with brtfs i get the login message

no space left on device(28)

the drive has 16gb free.
if i login on root i can see that the system is rewritten, but
its says it cant write if i try to do.

I looked for answers already but i am stuck.
probably its a lock ?
it did happen after a total blocking of firefox, night 24 dec and the live stream from npo blocked 100% and then youtube aswell.
i might have used the SWITCH after all which is not harmfull at old systems, yet this is a 2016 system.

if i use leap15 to look at the other partition from 42.3 it does only see the /home, but /home/user is émpty
if i login on 42.3 and look at /home/user its FULL but … like a glass box. MC can look in side (f9 , quick view)
BUT the systems says the is no space to write to
perhaps the partition is mis-linked or locked

btw
when i look FROM leap15 TO leap42.3 i NEVER was able to look into the 42.3 home directory… which is or strange or overprotected and very annoying if you want to use your own asm on a different system. but i CAN copy the ROOT of from 42.3 to 15 using leap15 !!!, thats REVERSED to my opinion…

now i first have to login in on 15 again since i am on another type of 15 (lubuntu)


info from leap 15 side of view

cri@linux-3le6:~> sudo btrfs fi show
Label: 'osuse15'  uuid: 56a4d76c-d10b-4425-b51f-2d35df94fd65
    Total devices 1 FS bytes used 49.97GiB
    devid    1 size 200.00GiB used 52.07GiB path /dev/sda4

Label: none  uuid: a51159ea-b961-4d30-bbab-9089dfa42bcb
    Total devices 1 FS bytes used 83.41GiB
    devid    1 size 100.00GiB used 100.00GiB path /dev/sda1


i linked the partition in fstab:

cri@linux-3le6:~> sudo btrfs fi df /home/ch
Data, single: total=95.93GiB, used=81.59GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=2.00GiB, used=1.82GiB
GlobalReserve, single: total=183.41MiB, used=0.00B


cri@linux-3le6:/home> ls -l /home/ch
total 0
drwxr-xr-x 1 root root 1760 Mar 14  2019 bin
drwxr-xr-x 1 root root 1088 Mar 14  2019 boot
drwxr-xr-x 1 root root    0 Jul  7  2017 dev
drwxr-xr-x 1 root root 5908 Dec 29 00:28 etc
drwxr-xr-x 1 root root    0 Jul  7  2017 home
drwxr-xr-x 1 root root 3280 Aug 17 17:48 lib
drwxr-xr-x 1 root root 5232 Mar 14  2019 lib64
drwxr-xr-x 1 root root    0 May 10  2017 mnt
drwxr-xr-x 1 root root    0 Jul  7  2017 opt
drwxr-xr-x 1 root root    0 Jul  7  2017 proc
drwx------ 1 root root  454 Dec 29 17:14 root
drwxr-xr-x 1 root root    0 Jul  7  2017 run
drwxr-xr-x 1 root root 4602 Mar 14  2019 sbin
drwxr-xr-x 1 root root    0 May 10  2017 selinux
drwxr-xr-x 1 root root    0 Jul  7  2017 srv
drwxr-xr-x 1 root root    0 Jul  7  2017 sys
drwxrwxrwt 1 root root   94 Nov 18  2017 tmp
drwxr-xr-x 1 root root  130 Mar 14  2019 usr
drwxr-xr-x 1 root root  124 Mar 14  2019 var




from LEAP 42.3 side of view
manualy typed


lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda            0:0        0      931.5 G  0  disk
|-sda1         0:1        0      100g     0 part   /var/tmp
|-sda2         0:2        0          8 G   0  part swap
|-sda3         0:3        0          1k    0   part
|-sda4         0:4        0      200 G   0   part
sr0             ...          0       1024M  0 rom

is the mount set to /var/tmp ???
it that part of a lock ??

You have now clearly explained your problem.

I normally use “ext4”, so I don’t have much experience with “btrfs”. Hopefully someone else can step in and offer help.

As to why you see “/var/tmp” in that last output – it is probably misleading. Maybe that was the subvolume most recently mounted.

I unsure how to interpret those “btrfs” commands.

First, and FYI
I collected what I consider the best sources for information about BTRFS on the Web and posted links to them here

https://en.opensuse.org/User:Tsu2/systemd-1#BTRFS

You’ll notice that there are articles there for when the file system is full. Unfortunately while the Marc Merlins article is considered the best (even the official BTRFS Wiki references it), it’s highly technical and hard to understand. Still if you can get whatever you can out of that and combine with the other articles I reference, you’ll probably be adequately armed with the info you need to display information about your BTRFS volumes, subvolumes, usage and then how to fix.

It’s a good thing you’re using the BTRFS commands you posted, it’s a common mistake to use “normal” commands used on other file systems, but those would return inaccurate info. You’ve got a good start troubleshooting but you haven’t run commands which would be most useful like “btrfs du” and possibly “btrfs df”
Also, I don’t know that running commands from another OS would be that useful (but who knows, everyone probably can have a different opinion). As long as the system is bootable even if RO, the procedures described in the articles in my Wiki should work.

As for /var/tmp, IIRC nowadays it’s deployed in ram as tmpfs, so shouldn’t be related to a disk space issues (again, df or btrfs df should verify and inform).

Also, you should know that ordinarily for any openSUSE up to and including 15.0 and with a primary disk at least approx 50GB, only the root partition is BTRFS and /home is likely formatted XFS… Only if the disk is smaller, then the next most likely probable disk layout is that /home is deployed as a directory and not in its own partition. For these reasons, it’s highly unlikely that any time you spend looking at /home will have any effect on disk usage and running out of space… You should likely be looking at your root (/) partition only.

If you have problems working your way through the references in my Wiki, go ahead and post details about your problem…

TSU

Some recommendations on prevention and a Q&D possible fix…

I see your overall disk is 100GB, would be interesting to know how your partitions are portioned. Although I deal almost exclusively with virtual machines nowadays, my general rule of thumb should probably apply to physical machines as well… Although the openSUSE installer modifies the disk layout to install everything in a single partition and not separate partitions at 50GB, I’ve found in the past that the BTRFS volume (LEAP 15.0 and earlier is the root partition only) should not be less than 100GB, and then if /home is on a separate partition can be anything beyond that. I’m going to guess that when you do a proper readout of your disk layout and usage, your BTRFS root partition will be even smaller than 50GB which I consider much too small to support snapshots.

Years ago, a quick and dirty way I sometimes did to free up space is to simply delete approx 20GB of files… Which could be in the various tmp directories (I don’t remember of LEAP 42.x was still writing tmp directories to disk or to tmpfs), and doing a disk usage analysis for largest files and either deleting or temporarily moving them off the partition. In fact, I even started loading or creating enormous files just for them to be safely removable in the event I needed to free up disk space. Once sufficient free space was created and the system could boot normally, then I would clean out unnecessary snapshots which is almost always the cause of the problem in the first place… Then I would move those enormous files back into place to give me elbow room the next time a problem happened.

TSU

Hi stu2, thanks for your links

i found some single page info like
https://computingforgeeks.com/working-with-btrfs-filesystem-in-linux/
https://doc.opensuse.org/documentation/leap/archive/42.3/reference/html/book.opensuse.reference/cha.advdisk.html

I did try things like ’ du -sh /’ 'df -h
but my little problem is that i can read it on screen but NOT save it to a file.
Further i need to be on the 42.3 system to see what is active and make list of that.
i did make some picture with my handhold but that only works half, sinde a few lines is already nasty to get correct onn txt. like the ‘lsblk’ in typed out manualy.

I have 3 gb of memory.
would it be possible to create a virtual disc, but whithout writing ANYthing to disc , not even a mentioning or tmp file.
then MC could maybe write it to a USB-floppy drive ??
but thats to far away as solution.

i will re-try
lsof / |ggrep deleted
—>> systemctl restart ‘deleted service’

check inodes
df -i /

du -sm /* | sort -n
du sm /* | grep -v /home |sort -n

df -m
df -kPH

i noticed you explicitly mention btrfs in a different way
“btrfs du”
“btrfs df”
should i try all above with btrfs as start command ?
i found that fsck has many underlaying programs which you indirectly mention.

time to read-a-lot now
Cheers

btw
the error code is
no space left on device(28)
where can i find code 28 ??

perhaps:
Label: none uuid: a51159ea-b961-4d30-bbab-9089dfa42bcb
Total devices 1 FS bytes used 82.45GiB
devid 1 size 100.00GiB used 100.00GiB path /dev/sda1

the devid is 100% used

mark:

Look at filesystem show output: legolas:~# btrfs fi show
Label: btrfs_pool1 uuid: 4850ee22-bf32-4131-a841-02abdb4a5ba6
Total devices 1 FS bytes used 441.69GiB
devid 1 size 865.01GiB used 751.04GiB path /dev/mapper/cryptroot Only about 50% of the space is used (441 out of 865GB), but the device is 88% full (751 out of 865MB). Unfortunately it’s not uncommon for a btrfs device to fill up due to the fact that it does not rebalance chunks (3.18+ has started freeing empty chunks, which is a step in the right direction).

When you are working on a BTRFS formatted partition, you should always use the commands which use "btrfs: because the usual commands without “btrfs” will often not display accurate information.

If you are unable to save a file of a command and its result, use a phone’s camera to take a picture of your screen, then from that phone you should be able to either transfer to another machine (like a working PC) or upload directly to a cloud service where you can post a URL to it in your Forum post… If you can’t figure out how to do this, post details about exactly what you are trying to do and someone should be able to either find a good online guide or describe steps for you.

The articles in my Wiki contain recommended procedures for fixing your problem, which you choose will depend on your situation, what you are able to currently do.

But, it all starts with obtaining more information about your situation running the commands described in the articles in my Wiki.

TSU

Run


sudo snapper list

This will produce a list of snapshots, numbered.
Now run


sudo snapper remove ###-###

where ###-### is the range of snapshots to be removed.

YES
YES
YES
!!
i am TROUGH
and i can reach my own software again

thanks ! Knurpht
" geh gau mee oe ! , Moi! "

Congrats.
Simplest and most direct approach.

TSU