Applications Slow-Down after a Few Days from Bootup

I’m looking for tips on how to troubleshoot this in LEAP 15.3. After being online for around 5 or 6 days, Opensuse becomes incredibly sluggish, but nothing shows up out of the ordinary for taking up system resources or having excessive load. Any program, whether it be from the command line, system processes, or even loading webpages hosted by various different applications, they all slow right down and take ages to load/respond. Some even start erroring out, complaining that other applications are taking too long to respond and they time out. I have no idea what’s causing it, or how this came about. But rebooting when I experience it is the only thing that fixes it. The reboot process can take a long time (30 min to an hour for shutdown) when this occurs, but then the boot up sequence is nice and quick and everything is great for another 5 or 6 days.

Any suggestions on where to look or monitor? I’ve seen eminent hard drive failures cause system slowdowns… but not clear up when a reboot happens. Smartctl and mdstat doesn’t seem to show anything out of the ordinary, neither does /var/log/messages nor top.

Thanks.

Does simply logging out and back in also take 30+ minutes?

Sometimes when I’m bothered by slowdowns I simply close “all” window, then reopen the apps I actually need. When I notice slowdowns are too serious, I shut down all apps, log out, then log right back in, and all is good. This way reboots are usually only for new kernels. I suspect memory leakage and fragmentation caused by too many open tabs and browsers, but am not bothered enough by having to log out every 7-10 days to cause me to try to do anything about it.

Logging out doesn’t seem to take too long. It goes to the shutdown processes screen fairly quick and then crawls through them. What I haven’t tried is just logging out and in on the UI… and will try this the next time it experiences issues.

The system is used primarily as a file and media server. So when running, there aren’t really any graphical programs… mostly command line processes, scripts, etc. Even those are slow when I SSH into the box (SSH login start off slow, but then commands and typing is quick, execution remains slow of other commands). It seems that once a program gets going, it’s okay… but it takes a long time for it to get there, and if it needs to execute something external or interface with another program, everything falls apart from there.

I’d start by checking memory usage, and perhaps test it with a memcheck utility.

Every once in a while, when I’m using a native proprietary app plus libreoffice plus another proprietary app in a windows 7 VM I suddenly get all memory used (this with 32 GB RAM!) and also about half the swap space (which is normally never used). Reboot takes ages, to the point that sometimes I have to force one when I can’t wait.

Good thinking, thank you. I haven’t noticed any high memory or swap usage when the slowdown occurs, but running a memtest seems like an easy and prudent step.

We you write “/var/log/messages”, I assume you also checked the journal (journalctl --boot, journalctrl --boot -1, or similar).

The symptoms you describe are typical of low memory situations, but I would have expected the kernel out-of-memory killer to kick in, if it did, there would be journal entries with text such as:

MESSAGE: Out of memory: Killed process 9241 (stress-ng) total-vm:24939164kB, anon-rss:13997556kB, file-rss:0kB, shmem-rss:4kB, UID:500 pgtables:48804kB oom_score_adj:1000
PRIORITY: 3
SYSLOG_IDENTIFIER: kernel

In the past I have seen media hardware/driver errors cause responsiveness issues. The journal should log those too.

Thanks for your suggestion! I haven’t tried journalctl just yet during a slowdown, and will on the next occurrence. Currently, there is nothing in the log since the last boot to indicate any issues… be interesting if I see something similar to your note here.

There may be no need to wait. If the /etc/systemd/journald.conf Storage option is set to auto or persistent, journald logs from past boots should still be available. In which case, journalctl --boot may be used to look back any number of boots (-1, -2, -3, …).

Unfortunately, it wasn’t set. So I’ve made it auto now… Just a matter of catching it in the next couple days as the issue occurs again.

IME it may be necessary to manually create /var/log/journal for persistent journal to take effect regardless of the journald.conf setting.

An update for this thread: The issue occurred again just over 8 days of uptime. I checked the journal entries and absolutely nothing is noted. I even searched the whole journal for entries that contain “memory”, and very little came back other than standard config log info at startup. Memory usage is only 4.39 GB of 15.6GB, and swap is at 320 MB of 16 GB. The load averages are fine at 0.29, 0.26, and 0.35. Nothing stands out for any processes monopolizing the system. As such, I’m proceeding with a memory scan.

I’d install htop and run it as root & set it up so that res or virt column are the selected columns ( just click on the column name to select it) and see what is eating the memory.

Also when shutting down hit the escape key to see what is shutting down and what it is hung on.

Well it could also be a kernel subsystem that might not necessarily show up as purely memory use. I’m not a btrfs user, but I’ve read this:

https://wiki.gentoo.org/wiki/Btrfs:***Btrfs hogging memory (disk cache)**
When utilizing some of Btrfs’ special abilities (like making many --reflink copies or creating high amounts of snapshots), a lot of memory can be consumed and not freed fast enough by the kernel’s inode cache. This issue can go undiscovered since memory dedicated to the disk cache might not be clearly visible in traditional system monitoring utilities. The slabtop utility (available as part of the sys-process/procps package) was specifically created to determine how much memory kernel objects are consuming…*

I’m not implying btrfs is necessarily to blame, just that it’s worth reading through the journal and looking for non-memory related unusual events that don’t appear in the journal when things are running OK.

You mentioned, that shutting down all apps and logging out fixes things. I wonder if it could be a X11/Wayland/video-driver issue. Or perhaps a thermal issue. Before logging out out or before logging back in you might check /home/your_username/.local/share/sddm/xorg-session.log in case there are any clues there. During an event you might also use a phone/tablet/another-box to remote ssh in and see if the ssh session is responsive, investigate if it’s a desktop-only issue.

I don’t know if it would help, but I recently wrote a desktop utility to warn we if a process was continuously consuming CPU or memory. I wrote it because chrome kept running away with the CPU. I often failed to notice for hours until I realised my system fans had become noisy. Closing tabs or returning to the home page don’t help. I suspect it is a GPU issue from using chrome with GPU-acceleration enabled. Anyway, I wrote an utility to raise DBUS desktop-notifications when this happens, plus it displays an at-a-glance overview of CPU and RSS activity:

It’s and offshoot of something I wrote to raise journal events as DUS desktop-notifications (but it grew to be a GUI journal browser), both are described in this thread:
https://forums.opensuse.org/showthread.php/561962-Jouno-a-different-way-of-tracking-system-activity

The memory scan conducted 3 passes, and had 100% pass ok. So that doesn’t seem to be the issue.

Yup, love htop, have used it for years. I check it constantly, but there’s no issues apparent when the slowdown occurs.

Thanks for the suggestions! I’ll look into Btrfs, it could be the issue as my system drives run this. I don’t think it’s CPU or Memory related at this point…. Load averages and such just don’t seem to indicate this, unless there’s something terminally wrong, but early indications don’t seem to say that.

Sorry, but logging out does NOT change anything. I tested this out once it was suggested, and there was no difference. Looking at the shutdown process list, there are no issues there…. But it just CRAWLS along. Each process takes a while to close out. But finally does so. So whatever is slowing the system down, is affecting everything right to turning the system off. Then, once back up again, all is well again for another few days. So I suspect whatever is causing the slowdown isn’t closing out until the reboot occurs. Just need to figure out what process that is. What I might try next is a guess and test method… start shutting down all programs and processes manually, until I find something that doesn’t close properly.

Here’s an update: I’ve narrowed it down to the offending program → Docker seems to be culprit.

When the issue occurs, I cannot cleanly shutdown the two docker containers I am running (pi-hole and unifi). Seems as though the docker daemon goes unresponsive, and uncleanly terminates. No log info that I can find as of yet as to why/how/when. But when I force kill all of the docker processes (containers and for the daemon process) the system comes back to life. I can then access commands/programs/etc normally, and I can also restart the Docker daemon and the two docker containers and all is well once again for another 4 to 8 days.

I’m open to suggestions on troubleshooting docker… I’ll be googling around, but looks as though I’ve localized the issue. Thanks to all those here that helped out.

1. Scan Your System for Virus or Malware

One of the main reasons for Windows 10 apps slow to open issue is that your PC might be infected with a virus that slows your Windows 10, 8, or 7 PC response time.
You can use Windows Defender or a third-party antivirus program to scan your system and remove virus or malware.
Step 1. Go to “Settings” > “Update & Security” > “Windows Security”.
Step 2. Click “Virus & threat protection”.
Step 3. In the “Threat history” section, click “Scan now” to scan for viruses on your computer.

2. Run System File Checker to Fix the Slow Issue

There are several reasons why you are experiencing Windows 10 is very slow in opening any application. This could be because of some corrupted file or data on the system that causes you some issues. Using Command Prompt running as Administrator, you can perform these commands that can fix issues on your system.
Step 1. Type **command prompt **in the Search box.Righ-clickCommand Prompt and choose Run as Administrator.
Step 2. Type the following command in the command prompt window and press “Enter” on the keyboard.
sfc /scannow
Step 3. Let the scan finish and fix any potential errors.
Step 4. Reboot your Windows computer and check if you have the same error message.

3. Repair or Reinstall Problematic Apps

If it’s a certain app that has the slow to open problem, try to reinstall it and check if it is still loading slowly. You can use the Control Panel to repair or reinstall the apps that are not running as smoothly as they should.
Step 1. Open Control Panel > select “Programs and Features”.
Step 2. Select the problematic app on the list. Right-click on it and select Repair or Uninstall.
Step 3. Restart your computer and reinstall the programs.