A puzzler for you!

English Install/Boot/Login
1

Right. This is the fourth or fifth time I’ve come to the OpenSUSE forums for help, and in all of my previous visits, you folks have helped me out greatly! I hope I don’t stump you this time, but I have broken rather a lot of things. Long story very short, my root partition is 100% full (zero bytes remaining) and ClamAV did it. Or rather, I did it when I ran ClamAV with the wrong flags.
Lengthening the story, I believe that ClamAV put a bunch of temp files (to mark its progress for future reference i guess) when i did something like

sudo clamscan -ri /

It threw a bunch of errors, “unable to create temp file (file it was trying to make).” I let it continue for a few minutes, thinking it might just be having permission issues, and that the temp files may simply be for good form, logs or such.

I can clean out the root partition a bit from the terminal that is provided by booting in this state. After that, I can restart and get into X and i3wm. However, after sitting there for a few minutes, looking through my file system in more comfort and less panic, one of the df’s I run tells me that, lo and behold, my root partition once again has zero bytes remaining. It could be that Snapper is regenerating the snapshots I’m destroying for space. (I already tried reverting, the space being filled makes that an impossibility.) I’m really not sure what it could be other than that. The virus was just a browser hijacker, reinstalling my browser would have solved this problem quite quickly and efficiently. But instead I misused clamscan.

So the symptoms of this long story in short: I can clean up my root partition from CLI enough to reboot and get into X, but it quickly breaks itself again, making a lot of things stop working, and sending me back to CLI when I next reboot.

Thanks so much to anyone who can help me!
–Henry Wilson

2

You must provide more information before someone can help you. For example, which Desktop, and probably most important for this problem, what filesystems? Are you using btrfs, for example?

3

I did not read your story, but as a mod I also try to help people to get the best help for their problems.

Your title is one of the worst I have seen. It does not have any technical keywords to the subject of your problem. So the chance that many, with only limited spare time to help people here, will simply not open your thread and skip to one with a title that appeals to their field of knowledge.

4

Ack! (grimaces) I’m sorry, hcvv. I’ll be more specific in the future. I’m relatively new to forums, this is literally the first one I’ve been remotely active on.

 And Fraser Bell, first off, thanks for responding. the desktop I use is i3, but I also have KDE Plasma and icewm installed. My root filesystem is btrfs, and my home filesystem is XFS.

 I'll add as much extra information as I can here. My boot drive is a 256GB Samsung 950 pro, an NVME M.2 ssd. It is partitioned into root and home partitions, 40 GiB btrfs at /, the rest (~196GiB) XFS at /home/. Additionally, I have a 1 TB mechanical hard disk mounted to /media/storagedisk/ for data, projects I'm working on. 
 df returns a table with one normal row for /home/, one normal row for /media/storagedisk/, and a separate row for each distinct sudirectory (/tmp/, /bin/, etc) of my root partition as full. Each of them is listed as a 40GiB space. This is the format that df usually returns.  
 About half the time I log on, the terminal near-continually prints errors that it can't write logs. My computer cannot connect to the local network. Whenever I try to sudo something, it throws an error something like "sudo: can't write (log file), no space left on device." So I have log in as root.

After I get home from class today (or if I get lucky and this goes quickly, before I leave) I am going to try to see what exactly it is that fills my root partition back up after i make space.

Thank you!

5

Okay. My computer is now in emergency mode. It welcomed me very nicely, too.
I’m really not sure of this, but I have two ideas:

df lists, in addition to the filesystems that I know about and actively use (root filesystem, home filesystem, and external drive filesystem,) four temporary filesystems:
devtempfs, 8GB, 0% full, on /dev;
tempfs, 8GB, 0% full, on /dev/shm;
tempfs, 8GB, 2% full, on /run;
tempfs, 8GB, 0% full, on /sys/fs/cgroup.
Now, it lists my root fs (twelve times as different subdirectories) as a 42G that’s filled up with 30.4G and, get this, 100% full. Now, this makes me think that perhaps these (one listed four times?) temporary fs are expanding to be as big as they would ideally like to be as I clear out my root partition, eating up the space I make after a restart or something. Or, perhaps, I’ve made a mistake that I didn’t notice somewhere along the way and some unused space in my root partition is locked off.

A second Idea I have is that perhaps either I failed to properly destroy snapper’s snapshots when I was trying to make space, or it’s since rebuilt more. Now, the moment this thought came across me, I tried to go through snapper directly, with

snapper rm 557

to remove the biggest snapshot that’s on my machine. It gave my an error, if I remember correctly saying that it got no response from systemd, which I assume is caused by my root partition being full. So I tried doing it directly.

chmod 777 /.snapshots/557

so that I can then delete it unhindered. But, it tells me that it can’t change permissions due to the filesystem being full.

With the number of things this problem is stopping me from doing trying to fix it, I’m beginning to think that perhaps I should go in with a rescure CD (in my case USB, no optical drive) and that way find and get rid of things taking up space without worrying about permissions or crashes.

Also, to try to figure out why the root directory was filling back up, I ran (and I’m not going to get this character-for-character right)

find -name 'home' -prune -o -type f -newer *(file with timestamp 2017-04-18)* > *(text file)*

and read that text file. The first four things it mentioned were config files in the top directories of each snapshot. Then it listed tens of thousands of inconspicuous things. The fact that the config files were changed in the last day makes me sure that snapper knows about my semi-successful attacks on its snapshots, and in my mind lends credence to the idea that it’s rebuilding them whenever I break them. (it is possible for it to do this quite quickly due to my boot drive being a PCIe SSD. In theory it could write about a gigabyte per second.)

Also, I feel obligated to mention, this is not an emergency. I have all my data backed up, so recovering from this is, at worst, a day of reconfiguring software back to the way I like it. I’d just really like to learn how to recover from things like this, because I’m sure this won’t be the last time I deal with full system crashes.

I’m really not sure if there’s more information I should add here. If I think of anything, I’ll post it. If you need any information, let me know, and I’ll dig it out.

Thanks, folks!

6

I suspected btrfs, but needed the additional information.

Off the top of my head, I would say that the snapshots are filling your root partition, a common and often-mentioned “problem” here on the forums. You should read up a bit on Snapper, clean out old snapshots, and set Snapper more appropriately. Plenty of info to be found on the Forums about that.

See http://activedoc.opensuse.org/book/opensuse-reference/chapter-4-snapshotsrollback-with-snapper#sec.snapper.disable

Also, read the rest of that chapter, learn how to manage snapshots and use Snapper.

7

And here is some advice from my friend, Malcolm Lewis:

Start off configure /etc/snapper/configs/root to use say:

# limit for number cleanup
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="2-3"
NUMBER_LIMIT_IMPORTANT="3-4"

Use the following commands to see what is present and disk usage

snapper list
btrfs fi usage /

Then manually run the cronjobs, if the system gets turned off sometimes they don’t run:

/etc/cron.daily/suse.de-snapper
/etc/cron.weekly/btrfs-balance.sh
8

Thank you, Fraser Bell. A month or so after I installed openSuSE (leap 42.1 at the time) I noticed that Snapper was using a lot of space, because installing libraries in with Yast always warned me of low disk space. After a day or two of research, I figured Snapper out pretty well and did almost exactly what you and Malcolm (who has helped me several times before!) suggest. That fixed my problem. I believe that this is a different, likely related problem. For one thing, I can’t currently do much with Snapper, as the problem is stopping it from talking to systemd, which renders it nearly useless. There are a couple of discrepancies in how various tools are reporting my root filesystem that I believe are closer to the problem.

I have gotten a rescue USB going, and I have mounted the old root filesystem of my machine. I then proceeded to run *df *to see what it said. It reported it as a 40GB volume with 30GB of space taken up, but also as 100% full, which is the first discrepancy I’ve seen. This also does not involve the tempfs I mentioned earlier, as I am now booted off of a different device entirely. Then, when I went looking for the snapshots (they do take up a lot of space, and having some free space would make my tools work again and help me fix this) they were not in the .snapshots folder of my normal boot drive. It reported as empty to the root user of the rescue usb. Additionally, when I ran

du -hd1

(my standard first step for disk space analysis) from the root of the old boot drive, it listed a total of only 13GB of used space, which conflicts with df’s 30gb. The size of the snapshots is about that difference. So I’m not sure where the 10GB of completely unaccounted-for space are, and I’m not sure where the 16-17GB of missing snapshots are. I believe that the 10 GB of unaccounted-for space could be the problem that I am having currently, but I’m not sure and I need help

Again, thank you for your advice, Fraser. I really do wish the issue were only that.:wink:

9

Use the btrfs utilities if you want to get accurate btrfs information…

https://wiki.archlinux.org/index.php/Btrfs#Displaying_used.2Ffree_space
https://en.opensuse.org/SDB:BTRFS

10

If you removed snapper files directly ie without using the snapper command then your file system if probably broken. I’m not sure you can fix it. Much of snapper is done at a very low level and removing the files directly can destroy the relationships and meta data needed to maintain a healthy file system.

By any chance do you keep /home on root or do you heavily use a database that keeps it’s files on root? Because you say df shows 30gig used and that is pretty high for just system files. But then df is not reliable on BTRFS. Extra data change on a snapper based system can quickly use up space unless you plan for it.

11

Hm. Deano, that was an incredibly useful piece of information. I now know what the filesystem thinks it’s doing. Unfortunately, it says there are 11 free gigabytes, which its performance does not reflect. Still, this tool has many helpful utilities that will be useful in my SuSE days to come! Gogalthorp, you have also helped me in the past, with much success. You don’t sound hopeful on this one though. I did indeed, quite foolishly, try to manually destroy snapper snapshots. That may explain why my machine has sunken fully into emergency mode from its original problem of having a full disk. df and btrfs both report 30G used, but I do have rather a lot of things installed through Zypper. I think I have four web browsers installed, to give you a sense of the superfluity I am prone to. I am currently putting the Leap 42.2 installer on my rescue drive, and will be going through all the (mostly safe) recovery steps in the page that Deano linked on BTRFS.

Thank you both much, time for me to go continue on this weird downward-crashing-learning-spiral-thing of terror.

12

Gone through btrfs-pain repeatedly in the past: Best advice is to edit the recommended file system to EXT4 on every new install. Don’t get why this btrfs is preferred over stable EXT4 by default (!), even for beginners. Must be some ego-problem of some developer…

13

I can see is it for Tumbleweed where you have major updates that may break the system makes it easier to drop back. Also in some development situations and maybe for enterprise. And it works fine for most if they give enough space for the usage pattern to. That said I use EXT4 :stuck_out_tongue:

14

I used BTRFS on 3-4 machines for some months with 40 GB root iirc. During this short period of time I had 4 to 5 times panic because BTRFS filled up HDDs completely and significant pain to get the systems back running. In more than 1 year of TW (or is it already 2 years?) on several machines TW updates fu*ed up networking once on 1-2 machines, grub 3-4 times on 1-3 machines and sometimes applications. I think NEVER would have been BTRFS of any help to resolve these issues. With a dead grub you need to boot from install DVD, with messed up applications you can boot to old kernel from grub menu. Maybe BTRFS is magic to some freaks, but it’s a pain to normal users and far from ready to be DEFAULT for all installs of opensuse. My opinion. However, I have to a certain degree adapted to the strange mental state of people working in computer industry. If the rest of the world would try to do their work the same way, no car would drive more than a few miles, no people would get healthy again when visiting a doctor and nobody would have food and clean water for more than a couple of hours per year. But e.g. the design of the water pipes would be changed every few months…

15

…excellent example of what I mean: https://www.bloomberg.com/news/features/2017-04-19/silicon-valley-s-400-juicer-may-be-feeling-the-squeeze

16

Alright folks, gogalthorp let me know gently that I’d probably destroyed it. I went ahead and tried a few more things just in case, and sure enough, my root filesystem was toast. I have reinstalled SuSE, and was shocked to see, my computer was almost as it was just before the crash. My home partition had not been touched by the reinstall or by the original crash! As such, it took me about an hour and a half to set everything back up. Zypper is VERY powerful. As a result, this whole fiasco has actually reinforced my enthusiasm for Linux. My steam games library didn’t even need to be set back up. The symlink I set up to make Vivado (FPGA hardware development tool) synthesize turned out to still be valid. Heck, Xilinx didn’t even bat an eye with the licensing (at least not yet.) Thank you for your help, all. I’m still not sure how the problem got started in the first place, but it had something to do with ClamAV scanning my root partition. As a matter of fact, it may have tripped up on brtfs and snapper themselves. Who knows? Anyway, I also switched to EXT4 on reinstall, and everything’s golden right now. I’d like to see windows get reinstalled and have half its software still run, the other half need a single command issued to get it back. Though, to be perfectly fair, I’ve also never seen a malware scanner mess up windows’ system hive (is that what they call it?)

At any rate, thank you all very much. I now just have to get my data drive back into my fstab and install discord, then I’m back to normal.
Take care, all! :slight_smile:

17

Good to hear it is running. And, yes, reading the last several posts, you did what I would have suggested next by re-installing. Yes, it is amazing how easy a re-install is and the way openSUSE handles it, returning everything else usually in its last-seen state – as long as you remember to NOT format the /home partition.

BTW: Since it was mentioned in this post already, I might as well add that I, too, do not use btrfs, still sticking with the (IMHO) much more reliable ext4 filesystem.