Greetings, everyone. I’ve ran into a big issue that’s essentially killed my system. My computer has 2 drives and therefore 2 partitions: a root partition for my system in a 240gb SSD NVME and a home partition for my files in a 3TB Hard Drive. But yesterday, a strange incident happened: I was playing a game when I started getting notifications about specific folders being full (Folders like root, boot, var, usr). I knew those folders were system folders, but It didn’t make sense to me, because I don’t install programs or similar things in my root partition…I’ve basically left that partition alone since I installed the distro, and put all my files in the home partition, which still has 1 TB of space left. I assumed it was getting full with temp files that’d be deleted after I rebooted the computer, but It wouldn’t boot again after that. It repeatedly gets stuck in a black screen.
After that, we booted it with an Ubuntu Pendrive and it turns out that, indeed, the root hard drive is full! But we can’t FIND what’s bloating the SSD because ubuntu won’t let us view every file. In fact, we can only know that the partition is full by checking the disc’s partition, because just checking the properties of the displayed folders tells us there’s only 9 gbs being used, not 200. The var folder, where supposed Opensuse stores log files, is completely empty, and since the system won’t boot, we can’t use YAST at all. If it’s important, the root partition is in 3 pieces : a 629 mb part for booting, a 34 gb part for swap memory and the rest is all for the system in BTRFS, which is 99.8% full.
Why can’t I access all the root files on ubuntu? Is this an Ubuntu thing or is it because the system partition is in BTRFS? Where is the log file stored, since the var folder is empty? Why can’t I find what files are bloating the system? (And WHAT files would bloat 200 gb in a Linux distro even?) I can’t use YAST, so what CAN I use? Is there any way to solve this situation at all?
Could you check the hidden folder /.snapshots?
This folder is used by snapper by default to save snapshots. In can grow quite big, depending on how snapper is configured.
If you didn’t encrypt your ssd it should be possible to access it from a live iso.
If you do find that it is this kernel problem, and your log files (once you can view them) are huge, then you need to empty the log files. I’m no expert, and internet search revealed a few options. The best seemed to be (for the two log files that were huge for me):
That COULD be the issue (Although the implications that LOG FILES are what somehow bloated 200 gb terrifies me), but then how am i going to check this if I can’t even boot the system, and Ubuntu can’t check the complete root files?
It worked. Your truncate commands worked and the computer booted up normally again! I don’t have enough words to properly thank you for this, but I’m honestly mortified at the absurdity of the situation. 200+ Gigabytes of TEXT/LOG FILES.
Do you know what caused this issue? You said something about a kernel situation, but then how can we fix this so it doesn’t fill with log files again?
Another poster in this forum ended up helping us find part of the problem. Apparently the distro was creating Log files at an excessive rate. And it booted normally after we forced these log files to clear.
But to answer your question, we were just looking at the supposed folders using an ubuntu flashdrive. The var files and snapshot folders are ONLY accessible through OpenSUSE. Ubuntu just finds empty folders.
You need to remove the current kernel (5.14.21-150400.24.84.1) as your logs will be filling up fast again right now! I used YaST Software, following the directions here:
I had to scroll to the bottom of the list of versions to find the one to remove.
Ok, now YAST is saying my kernel is 5.14.21-150400.24.84.1, and so is the terminal command you put. I don’t know what to believe anymore, so I’ll assume that the kernel name’s correct now.
So It seems that the kernel we have is the bugged one. But i checked just now, and the system folder isn’t filling up. Seems that it’s actually emptying somewhat. It was 25 gb when we booted up, then jumped to 28.8 gb. Now it’s 27.8 gb.
But other than that, seems we have the broken kernel version
Yeah, I just tried it. You would have to chroot into your openSUSE- install from the live environment to be able to check/manipulate /.snapshots and /var/log/
Also I realized that the return regarding the size of /.snapshots (du -sh /.snapshots) would have been wrong anyways. One would have to check it with snapper ls.
But anyways: Good that @cmt29uk and you got it resolved.
In addition to that you might want to look into logrotate and how to configure it to limit the size of your logs in the long run.
In such case of excessive log spamming logrotate would need to run really often though… not the best solution
