Unable to boot due to no space in /

Here’s the historyof this issue: The system has been running only openSUSE since about2016 with upgrade 3 months +/- after each new rev came out. Alwaysdone by reinstalling while reusing the old /home. Both / and /homewere ext4 and remain so (“if it ain’t broke don’t fix it”).

When I got to 15.1all was fine for many months until one day the system would not boot. Shortly after thegrub selection the screen went black, keyboard became non-functioning, mouse cursor showed and responded to mouse movements. In the upper left corner of the otherwise black screen was a blinkinghyphen or underscore (maybe supposed to be a prompt (?)) that didnothing but blink.

I didn’t bother toinvestigate, as I was in a big hurry, so I just re-installed and wentback to work. All was fine for a while until the problem returned,exactly like before. Again I was in a hurry, so I tried a Mint 20live disk. All seemed to go well, so I installed Mint and went backto work for about a week, until the problem returned.

This time I finallygot time to really pay attention, so I installed 15.2 (still the same/home from the start), and I waited. After only two days it happenedagain.

OK, enough history. Reading through this forum I got the idea to boot to a thumb driveand check to see if the / partition was full. It was, despite being39 Gb, so I used a gparted disk and gave the partition another 18Gb.

When I rebooted allwent well until I left the room for an hour. When I came back thescreen was dark, no surprise likely power saving, however the PCwould not wake up. I powered down and restarted, but the old issuewas back, so I couldn’t boot up. Again I used a live disk to bootand checked the disk usage only to find that the newly expanded 57Gbpartition now had no free space!

Furtherinvestigation shows that /var/logscontains now 47.6Gb.

Atthis point I would greatly appreciate some advice.

Thesystem is a Lenovo P580 laptop. Intel i7 with integrated GPU, noother GPU, 16Gb RAM, 1Tb SSD, openSUSE Leap 15.2 only (no dual bootever). /home and / are stillext4. KDE desktop.

(SorryI can’t give more conventional diagnostics, but I can’t even bootup).

Something is generating a lot of log entries - like you I only use ext4 - I have since it was released in the 2008 - I used ext3 from 1999 - If it works stay with it and have some good backups.

Delete most of the /var/log that end in .log or .xz from the recovery boot up to free space and them boot OpenSUSE again.

After you boot up open 2 terminals and in one sudo dmesg and the other sudo journalctl -f and see what is happening

Since you are only using ext4 it would not hurt to delete off all the btrfs junk in /etc - sudo find /etc -name btrfs* -print | xargs -n 1 rm -rf ( I had to run 5 time to get rid of all the btrfs junk that runs ) I think it might be the source of you logs filling up.

Lots of error. read logs to find out the recurring error.


 > systemctl list-unit-files | grep -i 'btrfs'
btrfsmaintenance-refresh.path                                    masked         
btrfs-balance.service                                            static         
btrfs-defrag.service                                             static         
btrfs-scrub.service                                              static         
btrfs-trim.service                                               static         
btrfsmaintenance-refresh.service                                 masked         
btrfs-balance.timer                                              masked         
btrfs-defrag.timer                                               masked         
btrfs-scrub.timer                                                masked         
btrfs-trim.timer                                                 masked         
 > 

IMHO, it’s not a usable solution to go around deleting “junk” configuration files – they’re installed by packages with dependencies –


 > rpm --query --whatprovides /etc/grub.d/80_suse_btrfs_snapshot
grub2-snapper-plugin-2.04-lp152.7.18.2.noarch
 > 

Other files are links which mask off the systemd Btrfs services –


 > l -d /etc/systemd/system/*btrfs*
lrwxrwxrwx 1 root root    9 18. Aug 15:55 /etc/systemd/system/btrfs-balance.timer -> /dev/null
drwxr-xr-x 2 root root 4096 18. Aug 15:21 /etc/systemd/system/btrfs-balance.timer.d/
lrwxrwxrwx 1 root root    9 18. Aug 15:56 /etc/systemd/system/btrfs-defrag.timer -> /dev/null
lrwxrwxrwx 1 root root    9 18. Aug 15:55 /etc/systemd/system/btrfsmaintenance-refresh.path -> /dev/null
lrwxrwxrwx 1 root root    9 18. Aug 15:55 /etc/systemd/system/btrfsmaintenance-refresh.service -> /dev/null
lrwxrwxrwx 1 root root    9 18. Aug 15:55 /etc/systemd/system/btrfs-scrub.timer -> /dev/null
drwxr-xr-x 2 root root 4096 18. Aug 15:21 /etc/systemd/system/btrfs-scrub.timer.d/
lrwxrwxrwx 1 root root    9 18. Aug 15:56 /etc/systemd/system/btrfs-trim.timer -> /dev/null
 > 

On the other hand ‘/etc/sysconfig/btrfsmaintenance’ isn’t owned by any package but, I suspect that, it was created at installation time by a package’s installation script and, it’s not doing anything if systemd’s Btrfs services are masked …

Well - I see no need for btrfs snapshot management in grub2 when using ext4 and top showed 5 btrfs pids all the time til I killed the stupid btrfs stuff in /etc

I have been running without the btrfs files in etc since 42.0 - no issues - no lockups - no problems except having to install MATE desktop as the default.

I am old - I have been playing with computers since 1963 - I don’t like the complex desktops like gnome3 or kde. I loved gnome2 and MATE is gnome 2 with another name.

My 2 cents - YMMV

/etc/sysconfig/btrfsmaintenance is created from /usr/share/fillup-templates/sysconfig.btrfsmaintenance. Set periods to ‘none’ if desirable:

erlangen:~ # grep PERIOD /etc/sysconfig/btrfsmaintenance
BTRFS_DEFRAG_PERIOD="daily"
BTRFS_BALANCE_PERIOD="daily"
BTRFS_SCRUB_PERIOD="daily"
BTRFS_TRIM_PERIOD="daily"
erlangen:~ # 

BTW: btrfsmaintenance runs unobtrusively in the background.

Welcome to the wide wonderful world of package dependencies …

Could have been a DEC PDP-4 (18-bit) or PDP-5 (12-bit) or, something else …

  • When I began working for DEC Field Service in 1979, I wasn’t playing – I was repairing PDP-11 machines …

IBM 1401 (I was the printer output gather - I had learn octal and hex math in school [due to President Kennedy’s University of Maryland Science and Math Study Group program] so I helped with the core dumps 5th grade) and a GE 225 on a Teletype playing football.
PDP8 w/2 sykes tapes and a TTY - PDP11/30 w/decwriter (2) RK05 and DECtape running Unix and a GT40 running Lunar Lander in 1972/73
VaxClusters in the 1980’s for Banks and Credit Unions.

Sorry to have been unresponsive… medical appointments all day, one of the benefits of growing old. I will follow your suggestions shortly and report back.

Well that took me down a rabbit hole full of surprises. I booted to a thumb drive and found very few files in /var/log the ended in .log or .xz, not enough to make any difference, so I started looking at the other files there to see what might be at fault.

Here’s a partial du of the files I found:

/var/log # du -s -h messages messages-20210112 warn-20210118 warn warn-20210112
600K    messages
16G     messages-20210112
6.5G    warn-20210118
16K     warn
16G     warn-20210112

Any idea what wrote these and/or what they mean? My impulse is to get rid of them, but I seek advice first.

Even if I do get rid of them I want to know where they came from and if the problem is apt to simply recur.

Hi
You need to see what errors are being produced to see what is what, tail the last 50 lines or so…

I did try earlier, but the PC locked up. Not sure what that was about, but I’ll reboot and try again.

Hi
Run, or just look at what’s being broduced with journalctl;


tail -n 50 /var/log/<your log name>

or

journalctl -f

https://www.dropbox.com/s/ir9r22m76fyocwx/tail%20-50.txt?dl=0

Hi
Isn’t baloo some sort of indexer… it’s a KDE/Plasma thing… not my cup of tea, wait for others more in the know :wink:

Hi
Might help?
https://bugs.kde.org/show_bug.cgi?id=406868
https://forums.opensuse.org/showthread.php/534745-KDE-and-Baloo-File-Indexing

these logs get created every boot in /var/log

messages
warn

I would cat /var/log/warn to see what it contains

it is ok to sudo rm messages* warn* in the recovery mode and then boot OpenSUSE

also open a terminal and tail -f /var/log/warn

and open a terminal and tail -f /var/log/messages

Any thoughts as to why one file might be 16G +/- ?

I’m skeptical about the first as it was marked fixed in October, but…

The second link offers some interesting data and a possible fix.

I will try clearing out the files as larryr suggests and watch what happens next.

@caprus and others …

Yes, Baloo had some problems – 2 years ago …

But, those issues have been fixed …
[HR][/HR]How to work around issues such as those described here:

  1. Check the status of the Baloo indexer: “balooctl status”
  2. Check if any files could not be indexed: “balooctl failed”
  3. Stop the Baloo indexer: “balooctl disable”
  4. Logout and remove everything in ‘~/.local/share/baloo/’ from a VT session – tty1 … tty6 …
  5. Log back in again and check the Baloo status – restart it if need be.
  6. Periodically check the Baloo status paying particular attention to the list of files which can not be indexed …

 > LANG=C balooctl status
Baloo File Indexer is running
Indexer state: Idle
Total files indexed: 23,325
Files waiting for content indexing: 0
Files failed to index: 0
Current size of index is 16.78 MiB
 > 

“balooctl -h” for further Baloo control commands …