I need urgent help with a major issue. I just preformed a fresh install of openSUSE Tumbleweed. I am now able to log in, however my machine is so slow I’m barely able to use it and even write this post.
I managed to log in from root and catch a glimpse of what’s happening with “top”: There are two processes called btrfs-cleaner and btrfs-transacti. The two are using 100% of my CPU, to the point where the rest of the system has no resources left to operate. The horrifying part is that it’s completely impossible to kill them, something I’ve never experienced with any other process before: The command “kill -9 PID” does nothing to them, they are essentially being forced onto my system.
How do I get rid of them please? I managed to search for a bit of info: They’re apparently related to some snapshot backup mechanism that’s part of btrfs. I’ve never heard of it before so this is completely new to me, my previous install was on an ext4 partition so I’ve never ran into those things. What am I supposed to do to prevent this issue please?
A quick update on the situation: It appears that after several minutes, the two processes have disappeared on their own. I’m no longer seeing them in “top” and my machine is no longer freezing up and needing to be restarted. I’m currently afraid to reboot as they might come back once the computer starts up again. I’ll wait for someone to enlighten me on what exactly they’re doing and what they are, potentially how to disable this snapshot system if it’s not necessary.
I wouldn’t disable snapshotting. My educated guess is that this was a a one time issue; I’ve seen it once on my laptop, immediately after install and first zypper dup, never again.
I literally wanted to nuke it from my system at first, the experience was quite scary since I didn’t know what was happening. After my last post it only did this a few more times briefly, and so far it seems to be slowly getting rarer and rarer. Indeed I imported a lot of system packages from my old install then did a “zypper dup” too, many things happened at once. I was not aware of this mechanism at all till the freezes so at first I thought my new SSD had hardware access issues.
From what I could deduct, this is a Linux equivalent to the System Restore feature in Windows? It seems to use btrfs subvolumes to keep copies of files modified by YaST or system updates, which grub2 then lets you account for when booting.
For safety however, I’d really like to know if there’s a way to kill those processes when they’re about to freeze the system. “kill -9 PID” doesn’t affect them, which I find pretty scary since IIRC the system shouldn’t allow an immortal process.
Otherwise I’d still suggest the admins to please check the default process priority (niceness) for those snapshot processors. I wouldn’t have minded if it just made the system slower a bit, but they rendered it completely unusable (not even NumLock / CapsLock leds could be toggled for 5 minutes). Blender 3D uses 100% of all CPU cores when it’s rendering, and even then the system runs perfectly fine… I’m surprised at how this managed to top even its resource usage.
Now that I understand the system somewhat, I still had one more question about those snapshots. Figured it’s not worth opening another thread since I just made this one, plus it’s related to the performance issues too.
Is there a limit to how many of them will get generated before old ones start being deleted? Either in GB (size used by the .snapshots directory), in number of total snapshots, or based on date (prune snapshots older than X months). I don’t want this system to pointlessly take up space on my drive, especially since it’s an SSD and I don’t seek to reduce its lifetime by doing pointless reads / writes on it. If there’s no limit it may eventually get to a point where you can’t install new packages due to drive space, all to maintain some ancient changes which would be impossible to try reverting after so much time has passed and the packages and settings have radically changed.
Snapshots are not actually copies. They are just deleted or changed blocks that are preserved until released plus a bit of meta data to index it. So really not much writing is done.
You can set how many and often snapshots are taken and how long they are perserved
I have the same problem on my laptop (i7-8550U/256GB SSD/1 TB HDD/16GB RAM) with Leap 15.0. Btrfs is used on root partition (on SSD), as suggested by default install. From time to time, snapshots are created and it lasts from a couple of minutes to ten-twenty minutes. During that time, laptop can’t be used at all (especially when btrfs-transacti uses 100% of CPU). Last night right after midnight, btrfs-transacti/btrfs/btrfs-balance occupied CPU fully (100%), and I simply can’t do anything since then. The only time when I can interact with the laptop is during switching between the three btrfs tasks. After reboot, btrfs tasks just continued to work and occupy the CPU. I left the laptop at home (to complete the snapshot if that is the reason). Could the monthly snapshot be the reason for this and is there any suggestion how to avoid similar situations?
I’m sad to say I have this problem too with my laptop (Leap 15) - though not on my desktop with the exact same setup partition-wise.
Yesterday, the laptop couldn’t boot until the whole btrfs-xxx process was done. It lasted Two. Full. Hours >:(
Yes… that matches exactly what I observed back then. And remember I experienced this on my desktop computer, with a powerful CPU and a fast SSD drive: I can’t imagine how a laptop could even handle it.
This is a VERY dangerous default setting. I still believe the installer should immediately stop defaulting to BTRFS with snapshots and instead recommend EXT4. Sadly the maintainers believe there are good reasons to keep that, which I honestly cannot understand yet.
Hi
I have similar issue, after wakeup from suspend, laptop freezes for few minutes, checked htop and seems like btrfs services cause cpu spike.
Any solution or config tweak to solve this? maybe some snapshot shedule tuning?
I decided to spend a few minutes looking into this… again(You never know when something new is posted).
The first thing I considered was whether it’s possible to modify the “niceness” (process priority), possibly making any BTRFS maintenance run in the background so other things aren’t affected. I couldn’t find a ready solution for this and I doubt that modifying the Snapper service would do anything, you’d likely have to manage what Snapper does, not Snapper itself.
You can modify what is snapshotted and when. Although I believe the default configuration is pretty good at this, if your system is set up in some unusual way… like a third party virtualization app like VMware or want to save your virtual machines to an unusual location, or if you’re running a RDBMS or other major app with its own idea where to store files, you’ll want to apply approprate tweaks https://wiki.archlinux.org/index.php/Snapper#Preventing_slowdowns
If you’re wondering how Snapper will manage snapshots on your system, you can start by looking at the default configuration at
/etc/snapper/configs/root
The settings are described in the MAN page
man snapper-configs
You can either copy and enable your copy or edit the default snapper config file to modify various settings like
When snapshots are taken, by hours, days, etc. – There is even a special setting for hourly
Snapshot retention, cleaning out old snapshots
How much space (percentage of your total file system space) can be used for snapshot data
How much free space do you want to reserve so that when your snapshots are full you can still login and things will work (By default .2 of the FS is reserved)
Do you want regular Users to run Snapper? You can specify individual Users, or better a Group to have access to Snapper.
Getting back to the idea of configuring BTRFS operations to execute with background priority, so far I’ve seen nothing that can be done using systemd objects and commands… So, I suppose it can and probably should be done using “old school” methods to configure a process’ niceness (You define the command and set the niceness value, this is far easier than changing the nice value after a process is already running which requires discovering the PID which randomly changes whenever the process starts when it’s not set).
Hi
Thanks for detailed info, I will play with config and hopefully make this issue less annoying (on other case I have few “Spare” minutes to go and grab coffee ).
Hi.
I had the same problem and after researching in the openSUSE documentation I found out that there are three types of snapshots: timeline, installation and administration.
I disabled the timeline and administration, leaving the installation active, which I think are essential. In this way the number of snapshots was significantly reduced and I have not had problems again.
Very interesting. Thanks for this useful information! Can this be disabled from the YaST2 GUI?
I’m going to install a new motherboard next month and might have to reinstall the whole OS in the end. I’m ever so slightly considering giving BTRFS another chance, as long as I can disable snapper the moment I experience something like this again.
I don’t think I will though, as I think BTRFS itself has issues: When I tried this it wasn’t just the machine freezing, but the computer stopped booting after a forced restart. It was frozen on a mount job that would never finish. I took a photo of the screen to keep for reference, before I had to reinstall openSUSE at that stage… it’s blurry but experienced people can hopefully read it and explain what happened.
I can’t help you with your problem, but I have Tumbleweed and Leap 15.1 installed on my laptop and, in my opinion, the problem is not BTRFS but Tumbleweed: it’s the price to pay for rolling releases. Leap is much more stable and this does not happen.
).