Kworker hogging CPU in Leap 15.3

English Install/Boot/Login
1

Apologies if I’m posting to the wrong forum. I’ve been a long time user of Open Suse distros. Had Suse 12.3 then Tumbleweed and moved over to Leap 15.1 then 15.2 which I ran for quite a while. Upgraded to 15.3 and started noticing some behavioral problems. At first 15.3 ran reasonably but after an recent update noticed some issues with response. Using “top” I found that “kworker” was hogging the 1st cpu almost continuously at 100%. Got so that the system became unresponsive, i.e. the mouse would move in large jumps and apps would essentially grind to a halt. I read up on “kworker” but couldn’t determine a specific issue. Thinking maybe it was the update I re-installed 15.3 from a DVD. Problem remained. Backed off and installed Leap 15.2. There I found that “kworker” was seen but not as much hogging of the cpu’s. Before this incident I don’t think I was really aware of “kworker” as it must have been pretty much a background process. BTW, for anyone not familiar, “kworker” is apparently the designation of the kernel threads being executed in user space? I’m open to correction on this description. Back to my problems. When “kworker” is hogging the cpu it apparently blocks processes executing on the other three cpu’s. Top shows “kworker” with %mem of 0 but substantial amounts of time.

Questions: 1.) has anyone else encountered a similar problem?
2.) if so was there a solution.
3.) any suggestions?

best regards
J.Jefferies

2

Ksysguard displays some 40 instances of kworker on host erlangen, but load as printed by top is low:

Tasks:** 361 **total,**   1 **running,** 360 **sleeping,**   0 **stopped,**   0 **zombie 
%Cpu0  :**  0.0 **us,**  0.0 **sy,**  0.0 **ni,**100.0 **id,**  0.0 **wa,**  0.0 **hi,**  0.0 **si,**  0.0 **st     %Cpu1  :**  0.0 **us,**  0.0 **sy,**  0.0 **ni,**100.0 **id,**  0.0 **wa,**  0.0 **hi,**  0.0 **si,**  0.0 **st 
%Cpu2  :**  0.0 **us,**  0.0 **sy,**  0.0 **ni,**100.0 **id,**  0.0 **wa,**  0.0 **hi,**  0.0 **si,**  0.0 **st     %Cpu3  :**  0.0 **us,**  0.0 **sy,**  0.0 **ni,**100.0 **id,**  0.0 **wa,**  0.0 **hi,**  0.0 **si,**  0.0 **st 
%Cpu4  :**  0.0 **us,**  0.0 **sy,**  0.0 **ni,**100.0 **id,**  0.0 **wa,**  0.0 **hi,**  0.0 **si,**  0.0 **st     %Cpu5  :**  0.0 **us,**  0.0 **sy,**  0.0 **ni,**100.0 **id,**  0.0 **wa,**  0.0 **hi,**  0.0 **si,**  0.0 **st 
%Cpu6  :**  5.6 **us,** 11.1 **sy,**  0.0 **ni,** 83.3 **id,**  0.0 **wa,**  0.0 **hi,**  0.0 **si,**  0.0 **st     %Cpu7  :**  0.0 **us,**  0.0 **sy,**  0.0 **ni,**100.0 **id,**  0.0 **wa,**  0.0 **hi,**  0.0 **si,**  0.0 **st 
MiB Mem :** 31923.87+**total,** 14702.44+**free,** 6319.051 **used,** 10902.37+**buff/cache 
MiB Swap:**    0.000 **total,**    0.000 **free,**    0.000 **used.** 23960.09+**avail Mem

Show the following:

erlangen:~ # inxi -zFm 
System:    Kernel: 5.14.14-1-default x86_64 bits: 64 Console: pty pts/1 Distro: openSUSE Tumbleweed 20211031 
Machine:   Type: Desktop Mobo: ASRock model: Z170 Pro4S serial: <filter> UEFI: American Megatrends v: P3.50 date: 06/23/2016 
Memory:    RAM:total: 31.18 GiB used: 7.76 GiB (24.9%) 
           Array-1:capacity: 64 GiB slots: 4 EC: None 
           Device-1: ChannelA-DIMM0 size: No Module Installed 
           Device-2: ChannelA-DIMM1 size: 16 GiB speed: 2133 MT/s 
           Device-3: ChannelB-DIMM0 size: No Module Installed 
           Device-4: ChannelB-DIMM1 size: 16 GiB speed: 2133 MT/s 
CPU:       Info: Quad Core model: Intel Core i7-6700K bits: 64 type: MT MCP cache:L2: 8 MiB 
           Speed: 848 MHz min/max: 800/4200 MHz Core speeds (MHz):1: 848 2: 995 3: 1927 4: 800 5: 800 6: 800 7: 800 8: 800 
Graphics:  Device-1: AMD Lexa PRO [Radeon 540/540X/550/550X / RX 540X/550/550X] driver: amdgpu v: kernel 
           Display:server: X.Org 1.20.13 driver:loaded: amdgpu,ati unloaded: fbdev,modesetting,vesa 
           resolution: 3840x2160~60Hz 
           OpenGL:renderer: Radeon RX550/550 Series (POLARIS12 DRM 3.42.0 5.14.14-1-default LLVM 13.0.0) v: 4.6 Mesa 21.2.5 
Audio:     Device-1: Intel 100 Series/C230 Series Family HD Audio driver: snd_hda_intel 
           Device-2: AMD Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] driver: snd_hda_intel 
           Sound Server-1: ALSA v: k5.14.14-1-default running: yes 
           Sound Server-2: PulseAudio v: 15.0 running: yes 
Network:   Device-1: Intel Ethernet I219-V driver: e1000e 
           IF: eth0 state: up speed: 1000 Mbps duplex: full mac: <filter> 
Drives:    Local Storage:total: 6.38 TiB used: 3.19 TiB (49.9%) 
           ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 950 PRO 512GB size: 476.94 GiB 
           ID-2: /dev/sda vendor: Crucial model: CT2000BX500SSD1 size: 1.82 TiB 
           ID-3: /dev/sdb vendor: Western Digital model: WD40EZRX-22SPEB0 size: 3.64 TiB 
           ID-4: /dev/sdc vendor: Samsung model: SSD 850 EVO 500GB size: 465.76 GiB 
Partition: ID-1: / size: 51.69 GiB used: 28.82 GiB (55.8%) fs: btrfs dev: /dev/nvme0n1p3 
           ID-2: /boot/efi size: 99.8 MiB used: 8.3 MiB (8.3%) fs: vfat dev: /dev/nvme0n1p1 
           ID-3: /home size: 406.34 GiB used: 294.54 GiB (72.5%) fs: ext4 dev: /dev/nvme0n1p4 
           ID-4: /opt size: 51.69 GiB used: 28.82 GiB (55.8%) fs: btrfs dev: /dev/nvme0n1p3 
           ID-5: /var size: 51.69 GiB used: 28.82 GiB (55.8%) fs: btrfs dev: /dev/nvme0n1p3 
Swap:      Alert: No swap data was found. 
Sensors:   System Temperatures:cpu: 33.0 C mobo: 33.0 C gpu: amdgpu temp: 51.0 C 
           Fan Speeds (RPM):fan-1: 0 fan-2: 507 fan-3: 0 fan-4: 0 fan-5: 0 fan-6: 0 gpu: amdgpu fan: 966 
Info:      Processes: 360 Uptime: 9h 23m Shell: Bash inxi: 3.3.07 
erlangen:~ #
3

Hi, I’m trying to follow your train of thought.

On my system here is top:
top - 23:57:59 up 5 days, 12:42, 14 users, load average: 1.90, 2.01, 2.10
Tasks:** 337 total, 1 running, 336 sleeping, 0 stopped, 0 zombie
top - 23:58:12 up 5 days, 12:43, 14 users, load average: 1.92, 2.01, 2.10
Tasks: 337 total, 2 running, 335 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.3 us, 23.4 sy, 0.0 ni, 61.0 id, 0.0 wa, 0.0 hi, 14.3 si, 0.0 st
KiB Mem : 8055184 total, 1647524 free, 3634908 used, 2772752 buff/cache
KiB Swap: 82109136 total, 81846736 free, 262400 used. 3415612 **avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
**30007 root 20 0 0 0 0 R 100.0 0.000 3:56.02 kworker/u8:7+events_unbound **
7855 root 20 0 1074476 200344 153468 S 6.250 2.487 41:00.95 X
**30780 jeff 20 0 39428 4248 3376 R 6.250 0.053 0:00.01 top **
1 root 20 0 220676 8620 6056 S 0.000 0.107 0:33.48 systemd
2 root 20 0 0 0 0 S 0.000 0.000 0:00.34 kthreadd
3 root 0 -20 0 0 0 I 0.000 0.000 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.000 0.000 0:00.00 rcu_par_gp
8 root 0 -20 0 0 0 I 0.000 0.000 0:00.00 mm_percpu_wq

Note that kworker is running at 100% of 1 cpu.

[FONT=monospace]inxi -zFm shows
System: Kernel: 5.3.18-lp152.95-default x86_64 bits: 64 Desktop: KDE Plasma 5.18.6 Distro: openSUSE Leap 15.2
Machine: Type: Desktop System: Dell product: Inspiron 620 v: 00 serial: <filter>
Mobo: Dell model: 0GDG8Y v: A00 serial: <filter> BIOS: Dell v: A10 date: 02/22/2013
**Memory: RAM:**total: 7.68 GiB used: 3.60 GiB (46.9%)
**Array-1:**capacity: 8 GiB slots: 2 EC: None
Device-1: DIMM0 size: 4 GiB speed: 1600 MT/s
Device-2: DIMM1 size: 4 GiB speed: 1600 MT/s
CPU: Topology: Quad Core model: Intel Core i5-2320 bits: 64 type: MCP L2 cache: 6144 KiB
Speed: 1868 MHz min/max: 1600/3300 MHz **Core speeds (MHz):**1: 1775 2: 1662 3: 1742 4: 2019
Graphics: Device-1: Intel 2nd Generation Core Processor Family Integrated Graphics driver: i915 v: kernel
Display: x11 server: X.Org 1.20.3 driver: modesetting unloaded: fbdev,vesa **resolution:**1: 1600x1200~60Hz
2: 1920x1080~60Hz
**OpenGL:**renderer: Mesa DRI Intel Sandybridge Desktop v: 3.3 Mesa 19.3.4
Audio: Device-1: Intel 6 Series/C200 Series Family High Definition Audio driver: snd_hda_intel
Sound Server: ALSA v: k5.3.18-lp152.95-default
Network: Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet driver: r8169
IF: p3p1 state: up speed: 1000 Mbps duplex: full mac: <filter>
**Drives: Local Storage:**total: 1.02 TiB used: 641.69 GiB (61.5%)
ID-1: /dev/sda vendor: PNY model: CS900 120GB SSD size: 111.79 GiB
ID-2: /dev/sdb vendor: Samsung model: HD103UJ size: 931.51 GiB
Partition: ID-1: / size: 36.43 GiB used: 8.30 GiB (22.8%) fs: ext4 dev: /dev/sda1
ID-2: /home size: 37.26 GiB used: 31.47 GiB (84.5%) fs: btrfs dev: /dev/sda3
Swap: ID-1: swap-1 type: partition size: 78.31 GiB used: 256.2 MiB (0.3%) dev: /dev/sdb3
**Sensors: System Temperatures:**cpu: 49.0 C mobo: N/A
**Fan Speeds (RPM):**cpu: 470 mobo: 873
Info: Processes: 343 Uptime: 23:56:21 up 5 days 12:41, 14 users, load average: 1.69, 2.05, 2.13 Shell: bash
inxi: 3.1.00

Of interest is that the fan speeds up when kworker hits 100% cpu utilization. So I can hear
the system slowing down as well as see it.But top shows kworker using 0 memory but 100% cpu.
Any thoughts on what might be causing the blockage?

thanks
J.

[/FONT]

4

Your system is swapping heavily. Turn off swap by running ‘swapoff --all’ as root. Many systems do fine with 8 GB RAM and 0 GB swap.

5

Hi karlmistelberger
Tried your suggestion turning swapoff. Top now shows
top - 23:57:59 up 5 days, 12:42, 14 users, load average: 1.90, 2.01, 2.10
Tasks:** 337 total, 1 running, 336 sleeping, 0 stopped, 0 zombie
top - 23:58:12 up 5 days, 12:43, 14 users, load average: 1.92, 2.01, 2.10
Tasks: 337 total, 2 running, 335 sleeping, 0 stopped, 0 zombie
top - 10:28:57 up 5 days, 23:13, 14 users, load average: 2.13, 1.99, 1.71
Tasks: 343 total, 3 running, 340 sleeping, 0 stopped, 0 zombie
top - 10:29:11 up 5 days, 23:14, 14 users, load average: 2.18, 2.00, 1.72
Tasks: 342 total, 2 running, 340 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.2 us, 24.4 sy, 0.0 ni, 72.6 id, 0.0 wa, 0.0 hi, 2.9 si, 0.0 st
KiB Mem : 8055184 total, 1445508 free, 3392972 used, 3216704 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 3379088 **avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
**14079 root 20 0 0 0 0 R 100.0 0.000 3:38.30 kworker/u8:4+events_unbound **
9183 root 20 0 0 0 0 I 0.332 0.000 0:00.35 kworker/1:1-events
10018 jeff 20 0 3413692 242204 108852 S 0.332 3.007 3:24.04 plasmashell
**14368 jeff 20 0 39428 4248 3372 R 0.332 0.053 0:00.01 top **
1 root 20 0 220676 8628 6064 S 0.000 0.107 0:34.28 systemd
2 root 20 0 0 0 0 S 0.000 0.000 0:00.36 kthreadd
3 root 0 -20 0 0 0 I 0.000 0.000 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.000 0.000 0:00.00 rcu_par_gp
8 root 0 -20 0 0 0 I 0.000 0.000 0:00.00 mm_percpu_wq
9 root 20 0 0 0 0 S 0.000 0.000 0:03.21 ksoftirqd/0

I’m not an expert in sussing out these sort of issues. But it appears to me that
kworker is continuing its set of threads. I rather don’t mind if 1 of 4 cpu’s
is dedicated to the OS but it does appear to slow the overall performance. Is there
anyway to determine what processes are involved with kworker? My thought is the
possibility of some hardware dependency approaching a failure mode and causing
unnecessary spawning of kernel processes.
Thanks for the interest.
J.

6

Take a look at: linux kernel - verifying where 'kworker/n:n' (in ps -aux) is invoked from - Stack Overflow

7

Uncomment swap in fstab, reboot.

8

Did you mean comment out? swap has been uncommented all along:

/etc # cat fstab
UUID=b7d7a9b5-4359-43fa-aa67-94da2aedc116 / ext4 defaults 0 1
UUID=fb3798b4-909c-44fe-bfd9-13056389ea6f /home btrfs defaults 0 0
UUID=88060e05-e0e8-46b1-b1a4-ab9a7b45b4f3 /mnt/sdb1 ext4 defaults 0 0
UUID=58d63884-e53b-48cb-8691-6d2249000134 /mnt/sdb2 ext4 defaults 0 0
UUID=7c7c087a-0298-4dd0-bf9a-5a42cec3c63f swap swap defaults 0 0

9

Yes, comment out. Keep the commented out line, just in case you want to restore.

10

To me, swapping would be a symptom, not a cause. Swapping would not normally be expected unless something is pressuring memory. Because kworker does work on behalf of other processes, you really need to find the upstream cause. Maybe you are running too many things? Or something really big like a big VirtualBox or a Android development environment?

Is it the same for all users? You can create a new user to test that. Is it the same if no one is logged in to X11 (use ssh to login remotely and run top/htop)? Or perhaps login to a different kind of X11 session such as LXDE.

For a cause I would be looking at what’s actively running and what resources are being consumed and what resources are running short: CPU, memory, disk, and network.

Does journalctl reveal anything unusual.

Perhaps a filesystem is running out of space and the system is busy looking for blocks. Btrfs can look like it has free space when it doesn’t.

I find I prefer htop for digging around, and I sort by CPU and Memory. There is also iotop for finding IO hogs.

Sorry that these sound so much like stabs in the dark, but these are the places I’d start looking.

11

BTW: With 3 days uptime kworkers are idle on host erlangen:

**erlangen:~ #** ps ax|grep kworker 
   23 ?        I<     0:00 **kworker**/1:0H-events_highpri] 
   29 ?        I<     0:00 **kworker**/2:0H-kblockd] 
   35 ?        I<     0:00 **kworker**/3:0H-events_highpri] 
   41 ?        I<     0:00 **kworker**/4:0H-kblockd] 
   53 ?        I<     0:00 **kworker**/6:0H-kblockd] 
   59 ?        I<     0:00 **kworker**/7:0H-events_highpri] 
  140 ?        I<     0:00 **kworker**/1:1H-kblockd] 
  163 ?        I<     0:00 **kworker**/3:1H-kblockd] 
  170 ?        I<     0:00 **kworker**/u17:0] 
  177 ?        I<     0:00 **kworker**/4:1H-kblockd] 
  179 ?        I<     0:00 **kworker**/6:1H-kblockd] 
  180 ?        I<     0:00 **kworker**/2:1H-kblockd] 
  384 ?        I<     0:00 **kworker**/5:1H-kblockd] 
  393 ?        I<     0:00 **kworker**/7:1H-kblockd] 
  939 ?        I      0:00 **kworker**/7:2-cgroup_destroy] 
 2230 ?        I      0:00 **kworker**/1:0-mm_percpu_wq] 
 3575 ?        I<     0:00 **kworker**/0:2H-kblockd] 
 4068 ?        I      0:00 **kworker**/2:0-events] 
 4351 ?        I      0:00 **kworker**/1:2-events] 
 4518 ?        I      0:00 **kworker**/6:0-events] 
 4636 ?        I      0:00 **kworker**/4:1-mm_percpu_wq] 
 4691 ?        I      0:00 **kworker**/3:2-rcu_gp] 
 4845 ?        I      0:00 **kworker**/5:1-mm_percpu_wq] 
 5290 ?        I      0:00 **kworker**/5:0-events] 
 5294 ?        I      0:00 **kworker**/6:1-events] 
 5306 ?        I      0:00 **kworker**/3:1-mm_percpu_wq] 
 5686 ?        I      0:00 **kworker**/2:1-mm_percpu_wq] 
 6250 ?        I      0:00 **kworker**/4:0-events] 
 6446 ?        I      0:00 **kworker**/0:2-mm_percpu_wq] 
 6515 ?        I      0:00 **kworker**/u16:1-events_unbound] 
 6519 ?        I      0:00 **kworker**/7:0-mm_percpu_wq] 
 6776 ?        I      0:00 **kworker**/u16:4-btrfs-endio-write] 
 7080 ?        I      0:00 **kworker**/u16:0-btrfs-endio-write] 
 7081 ?        I      0:00 **kworker**/u16:2-btrfs-endio-write] 
 7137 ?        I      0:00 **kworker**/u16:3-flush-259:0] 
 7138 ?        I      0:00 **kworker**/u16:5-events_unbound] 
 7139 ?        I      0:00 **kworker**/u16:6-btrfs-endio-write] 
 7140 ?        I      0:00 **kworker**/u16:7-btrfs-endio-write] 
 7141 ?        I      0:00 **kworker**/u16:8-btrfs-endio-write] 
 7142 ?        I      0:00 **kworker**/u16:9-btrfs-endio-write] 
 7143 ?        I      0:00 **kworker**/u16:10-btrfs-endio-write] 
 7196 ?        I      0:00 **kworker**/4:2-events] 
 7394 pts/1    S+     0:00 grep --color=auto **kworker**
13850 ?        I<     0:00 **kworker**/0:0H-kblockd] 
28451 ?        I<     0:00 **kworker**/5:2H-kblockd] 
31309 ?        I      0:01 **kworker**/0:0-events] 
**erlangen:~ #**