For the last few years, my Tumbleweed PC gets into a state where it’s in a constant ‘disk reading’ which robs the bandwidth to the disk for other applications. Upon till recently, it’s occurring after an update (zypper dup), and my workaround has been - do the update last thing at night, leaving it running overnight, and by morning it has settled down… Shutting down, restarting doesn’t help.
I have read many threads about baloo and akonadi, but I’m yet to find a thread that says ‘solved’.
Unfortunately, it’s been happening more often. The last 2 times, I have shutdown all the apps, and let it run overnight - it settles down by morning but I have a Mozilla Crash Report without any detail… When it’s occurring I’ve used ‘iotop’ and can see that there are several instances of akonadi-imap-resources hogging the drive. Using akonadictl give me
#sudo akonadictl
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to ‘/tmp/runtime-root’
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to ‘/tmp/runtime-root’
D-Bus session bus is not available!
24 – exe=/usr/bin/akonadictl
19 – appname=akonadictl
17 – apppath=/usr/bin
9 – signal=6
9 – pid=3205
KCrash: Application ‘akonadictl’ crashing…
KCrash: Attempting to start /usr/libexec/drkonqi
qt.qpa.plugin: Could not load the Qt platform plugin “xcb” in “” even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
A quick question … you’re using Wayland (vs X11)??
If using Wayland, next time, login using X11 KDE and see if anything is different.
(sidenote: yea, I remember those days in the past when baloo was hogging resources … it’s been a while, but I seem to remember simply killing the process, until the fix finally came along … have not experienced the baloo issue in quite a while, and have not experienced issues with akonadi).
I have been trying several things since - I’ve just found that if I kill baloo_file it stops, until I reboot…
I don’t know why baloo_file aggravates akonadi, but I feel they are connected by drkonqi which doesn’t seem to be working.
I has tried zypper install -f drkongi5, but that didn’t help.
The message:
qt.qpa.plugin: Could not load the Qt platform plugin “xcb” in “” even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
Can anyone help me with this - what application is referring to? qt, xcb, drkongi…?
The last 2 times, I have shutdown all the apps, and let it run overnight - it settles down by morning but I have a Mozilla Crash Report without any detail… When it’s occurring I’ve used ‘iotop’ and can see that there are several instances of akonadi-imap-resources hogging the drive.
That doesn’t tell much. The output shows only a minute of user cpu time. There is no hogging.
When pasting commands and output use triple backticks as I do below.
systemd-cgtop --cpu=time displays total time since boot. Be patient and wait for some cpu time to accumulate. Then zoom into the active processes by specifying e.g.:
Thank you again @karlmistelberger for your reply. However, what you must realise is, when this bug is active, this PC is very, very, very difficult to use. I don’t have confidence that your strategy is going to lead to a solution - can you increase my confidence?
You seem to have identified that it’s an i/o issue rather than a CPU issue. Looking at CPU performance information for the process is, I agree, unlikely to shed any light on the issue.
I would be interested in seeing the output of cat /proc/loadavg as well as a snapshot of iotop while the issue is occurring. The output from nproc could also be useful (just to know how many concurrent processes your system can handle - io bottlenecks tend to be more problematic if the number of concurrent processes (not the total number of processes - that’s easy to mix up) exceeds the number of CPUs/CPU cores available (been a while since I looked at this, but that’s what I recall).
I’m not a KDE user, but as I understand it, Akonadi is an indexing engine for the KDE PIM tools (KMail, etc). A few things come to mind as possible things to look at:
Is the Akonadi database corrupted somehow? You reinstalled the package, but that is unlikely to replace configuration files and the existing database.
I would look at the SMART diagnostics for the drive (probably when the runaway process is not running and the loadavg is back to normal levels), as well as looking at the system journal to see if there are any hardware issues with the drive - hardware issues can cause io_wait states to hang around while the drive tries to resolve the error itself if there’s a read/write error or some other problem with the electronics.
If you don’t use the KDE PIM applications, you don’t need Akonadi, as I understand it. If you do, then resetting the database is likely to be useful until such time as it is corrupted again. As I understand it, you can select the database engine to use - there may be performance reasons to consider switching. Note that if you opt to disable Akonadi, there are also Plasma widgets that use it (including, apparently, the clock widget), so not using it may not be an option depending on how you use your system.
ETA: I would also find it useful to see the output of iostat on the system while the issue is ongoing. That would confirm (as I noted to Karl) that the issue is caused by a disk-bound process hanging up while waiting for the I/O channel to catch up to processing.
That’s because the bottleneck isn’t the CPU, it’s the disk I/O channel. The process is likely in an io_wait state waiting for the backlogged disk to finish an I/O operation before the process can continue. If the process is blocking other queued processes, that’s what causes this to happen, even when the CPU utilization looks perfectly fine.
There are lots of different resources that can create a bottleneck in system performance - CPU is only one of many. DIsk I/O, network I/O, memory available, if memory is being swapped (which can be both a memory bottleneck and a disk I/O bottleneck all in one). As the OP has been looking at iotop, that tells me that it’s not a CPU-bound process resource issue.
This shows a fairly normal iotop output, sorted by I/O. On a system that’s having issues, you’d see a process that was fairly heavily accessing the disk subsystem.
Actually, another command that would be useful is iostat - that gives an output like this:
I had problems, quite a while back, with the akonadi* tools and baloo stuff. I killed baloo after booting up … eventually a fix arrived (via zypper dup), and no probs since.
As far as the akonadi* tools, I don’t use any KDE PIM tools, so removed ALL of them, plus all extraneous akonadi* tools, and obviously no probs with it anymore.
This PC was Leap before I transitioned to Tumbleweed. I remember those issues in Leap.
From the early days in Tumbleweed, a few minutes after boot, Dr Konqi would put a sad face on my status bar about baloo crashing. I did try a few things like balooctl stop, start, reset… but none of that helped. It wasn’t causing an issue at that time so, I didn’t worry about it.
My memory may not be accurate, but I think this issue started when Dr Konqi stopped putting the status up. Now I can see that Dr Konqi is not running, I think part of the solution is to get it running again.
If I try to run drkonqi independently, I get the qt error message:
Yeah, generally “it works for me” is not helpful as a troubleshooting step unless it’s followed with an comparative analysis of the working system and the non-working system to see what is different that is material to the problem.
The OP identified (correctly) that the issue is a disk channel bottleneck.
@pedroSMS, were you able to pull the /proc/loadavg info (either by catting that endpoint or by pulling the info from iostat, as well as providing a snapshot of the iotop output. I would probably make sure that the options show the accumulated data over time (option a), and the processes (option p - the default shows threads, and that might be useful, further down the line if we don’t uncover anything).
A screengrab would be fine after a few minutes of seeing the issue, but of course iotop is also going to be sluggish while it’s gathering data because of the load on the system (as you are painfully aware).
Alternatively, what I would also probably look to do is shut down the service entirely, back up the databases, nuke them, and then let it rebuild and see if that resolves the issue. The issue is, I think, unlikely to be with a corrupted installation (though you can also test this using the RPM command - rpm -Vv will give you an output that shows you if anything has changed in the package’s files).
I also would recommend using smartctl and viewing the system logs to see if there is a hardware issue. If it is a failing hard drive, then none of the other steps are going to ultimately be helpful.
I don’t think I have communicated how difficult this PC is to use when this bug is active - it can 30-60 minutes to boot. It can take 2-5 minutes to respond to keyboard stokes, 5-10 minutes to respond mouse clicks, over 30 minutes for firefox to load…
Screenshots!!! Forget it!!!
The bug is not active at the moment. Although, I do see some updates available…
I need to work with this PC - I’m NOT going spend my day entertaining someones curiosity.
My experience since '77 may seem to be a bit fuzzy, but we all should know much faster Fuzzy is. The solution I’m looking for starts with help getting Dr Konqi working…
Believe me, I do understand how difficult the system is to use when it does this. I’ve seen and experienced it firsthand. I completely understand your frustration.
I have direct experience in what causes this specific thing to happen. I spent time working with kernel developers (years ago) who were working on a proprietary set of tools to handle resource allocation at the hardware resource level to guarantee different users in a multi-tenant web hosting solution (an extremely large hosting provider) access to system resources (memory, CPU, disk channel I/O, network bandwidth, etc). I was responsible for writing in-depth, detailed technical documentation, so I learned about how this is done hands-on (combined with in-depth technical discussions with the developers working on the tools), at times in my research intentionally causing the very issue that you are seeing so that I could understand it well enough to write about it.
This is not theoretical for me; I have actual experience with this type of issue, and I know exactly what I’m asking and how difficult it is to get it. I wouldn’t ask if I didn’t think it would be useful to narrowing down something specific about your particular circumstances.
If you want to just bash at it with a hammer, then the option of killing the processes, nuking the akonadi database files, and then restarting it and letting it reindex everything is a good place to start. The GNOME ‘tracker’ tool had similar issues once upon a time (it may still have them and I haven’t encountered them), and other tools that used a database in similar fashion that was easily corrupted causing all sorts of performance problems - both of which required deleting the databases in order to correct the issue. That is a reasonable approach to try that may provide a quick solution.
I would, again, also look at the hardware info that I suggested before to make sure the drive isn’t failing. Kill the offending processes and look at info from smartctl and the system logs. If the drive is failing, then no amount of reinstalling software (which, again, is not likely to be corrupted, but you can check that easily instead of just brute-force reinstalling it, which doesn’t remove the config files or the data files that the RPM uses). I have personally seen how a failing hard drive creates performance issues, and this is not a potential avenue to disregard, because no amount of tinkering with the software or deleting and recreating the database files is going to fix a problem with a failing piece of hardware.
@pedroSMS that sounds like some more serious issues going on with those boot times…
At grub if you press the e key to edit and at the end of the linux/linuxefi line, add a 3 and press F10 to boot (that is multi-user target), so not desktop environment… how long does it take to boot that way (systemd-analyze)?
