Kernel fault when starting ss

Hy!

While diagnosing some network spamming apparently caused by Pidgin

https://forums.opensuse.org/showthread.php/575344-New-(-)-MDNS-spam-How-to-stop

I stumbled upon a non-functional ss on my TW (I have other TW installs and Leap 15.4 where ss is working, so apparently related to hardware?).

When starting ss in a console I simply get the message “Killed” and no further output, starting ss again leaves the console unresponsive.

On first ss I get in dmesg

   63.215947] BUG: unable to handle page fault for address: ffffffffffffffc8
   63.215957] #PF: supervisor read access in kernel mode
   63.215960] #PF: error_code(0x0000) - not-present page
   63.215963] PGD 165615067 P4D 165615067 PUD 165617067 PMD 0 
   63.215970] Oops: 0000 #1] PREEMPT SMP PTI
   63.215974] CPU: 0 PID: 2540 Comm: ss Tainted: G           O      5.19.8-1-default #1 openSUSE Tumbleweed cfee37eadc5a540a2720ec5a9e55d8deb9565ab8
   63.215981] Hardware name: Dell Inc. Precision WorkStation T7500  /06FW8P, BIOS A18 10/15/2018
   63.215985] RIP: 0010:raw_diag_dump+0x104/0x200 [raw_diag]
   63.215991] Code: 89 c5 49 8b 5e 08 89 dd 83 e5 01 0f 85 ff 00 00 00 4c 89 74 24 08 45 89 e6 89 14 24 eb 0c 48 8b 1b f6 c3 01 0f 85 80 00 00 00 <4c> 3b 7b c8 75 ee 44 39 f5 7c 6a 41 0f b6 45 00 66 39 43 a8 75 5f
   63.215997] RSP: 0018:ffffb4f38368b9a0 EFLAGS: 00010246
   63.216001] RAX: ffff8fcb60245810 RBX: 0000000000000000 RCX: ffff8fccb0fc9348
   63.216004] RDX: 0000000000000000 RSI: ffffffffb0278ec0 RDI: ffffffffb0278ec0
   63.216008] RBP: 0000000000000000 R08: ffffffffaf667cc0 R09: 0000000000000006
   63.216011] R10: ffffdcfcc648f800 R11: 0000000000000090 R12: 0000000000000000
   63.216014] R13: ffff8fcb60245810 R14: 0000000000000000 R15: ffffffffb026fdc0
   63.216018] FS:  00007f522d85c800(0000) GS:ffff8fd03ee00000(0000) knlGS:0000000000000000
   63.216022] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
   63.216026] CR2: ffffffffffffffc8 CR3: 000000023096e002 CR4: 00000000000206f0
   63.216030] Call Trace:
   63.216033]  <TASK>
   63.216038]  __inet_diag_dump+0x3c/0xb0 [inet_diag 6e5a23768fff3f0f8fbb1d3bede71750b73bbc75]
   63.216045]  netlink_dump+0x123/0x310
   63.216052]  __netlink_dump_start+0x1b6/0x2f0
   63.216057]  inet_diag_handler_cmd+0xb7/0xe0 [inet_diag 6e5a23768fff3f0f8fbb1d3bede71750b73bbc75]
   63.216063]  ? inet_diag_dump_start_compat+0x10/0x10 [inet_diag 6e5a23768fff3f0f8fbb1d3bede71750b73bbc75]
   63.216069]  ? inet_diag_dump_compat+0xc0/0xc0 [inet_diag 6e5a23768fff3f0f8fbb1d3bede71750b73bbc75]
   63.216075]  ? inet_diag_unregister+0x40/0x40 [inet_diag 6e5a23768fff3f0f8fbb1d3bede71750b73bbc75]
   63.216082]  sock_diag_rcv_msg+0x114/0x140
   63.216089]  ? sock_diag_bind+0x50/0x50
   63.216093]  netlink_rcv_skb+0x51/0x100
   63.216098]  sock_diag_rcv+0x24/0x40
   63.216102]  netlink_unicast+0x239/0x380
   63.216106]  netlink_sendmsg+0x250/0x4c0
   63.216110]  sock_sendmsg+0x5f/0x70
   63.216117]  ____sys_sendmsg+0x22e/0x270
   63.216121]  ? import_iovec+0x17/0x20
   63.216126]  ? sendmsg_copy_msghdr+0x7b/0xa0
   63.216131]  ___sys_sendmsg+0x80/0xc0
   63.216136]  ? __rseq_handle_notify_resume+0x362/0x480
   63.216143]  ? __sys_recvmsg+0x54/0xa0
   63.216147]  __sys_sendmsg+0x57/0xa0
   63.216152]  do_syscall_64+0x5b/0x80
   63.216158]  ? do_syscall_64+0x67/0x80
   63.216162]  ? syscall_exit_to_user_mode+0x17/0x40
   63.216167]  ? do_syscall_64+0x67/0x80
   63.216171]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
   63.216177] RIP: 0033:0x7f522da29524
   63.216181] Code: ff eb b7 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 90 90 80 3d 7d c4 0d 00 00 74 13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 48 83 ec 28 89 54 24 1c 48 89
   63.216187] RSP: 002b:00007fffec05efc8 EFLAGS: 00000202 ORIG_RAX: 000000000000002e
   63.216191] RAX: ffffffffffffffda RBX: 000000000000000a RCX: 00007f522da29524
   63.216195] RDX: 0000000000000000 RSI: 00007fffec05f0b0 RDI: 0000000000000003
   63.216198] RBP: 00000000000000ff R08: 0000000000000004 R09: 0000000000000001
   63.216201] R10: 00007fffec05ef9c R11: 0000000000000202 R12: 0000000000000003
   63.216205] R13: 00007fffec05f030 R14: 0000000003010014 R15: 00007fffec05f170
   63.216210]  </TASK>
   63.216221] Modules linked in: raw_diag inet_diag unix_diag wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel af_packet nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_tables ebtable_nat ebtable_broute iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_mangle iptable_raw iptable_security rfkill ip_set nfnetlink ebtable_filter ebtables ip6_tables iptable_filter bpfilter vboxnetadp(O) vboxnetflt(O) qrtr vboxdrv(O) snd_seq_dummy snd_seq_oss snd_emu10k1_synth snd_emux_synth snd_seq_midi_emul snd_seq_virmidi snd_seq_midi snd_seq_midi_event snd_seq dmi_sysfs iTCO_wdt intel_pmc_bxt gpio_ich iTCO_vendor_support ppdev stv6110x intel_powerclamp coretemp lnbp21 dell_smm_hwmon kvm_intel kvm dell_wmi ledtrig_audio dell_smbios dcdbas
   63.216267]  irqbypass pcspkr sparse_keymap dell_wmi_descriptor wmi_bmof stv090x ddbridge dvb_core joydev mc i2c_i801 i2c_smbus snd_hda_codec_hdmi tg3 snd_hda_intel lpc_ich snd_intel_dspcfg snd_intel_sdw_acpi libphy snd_emu10k1 snd_hda_codec snd_util_mem snd_ac97_codec snd_hda_core ac97_bus snd_rawmidi snd_hwdep snd_seq_device i7core_edac snd_pcm snd_timer snd soundcore parport_pc parport tiny_power_button raid1 md_mod acpi_cpufreq mptctl fuse configfs ip_tables x_tables ext4 mbcache jbd2 hid_cherry hid_generic usbhid crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel nouveau aesni_intel crypto_simd cryptd serio_raw mptsas xhci_pci video xhci_pci_renesas sr_mod cdrom scsi_transport_sas drm_ttm_helper xhci_hcd uhci_hcd ehci_pci ttm ehci_hcd mxm_wmi mptscsih mptbase drm_display_helper usbcore firewire_ohci cec firewire_core rc_core crc_itu_t wmi button sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr
   63.216358] Unloaded tainted modules: pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1 pcc_cpufreq():1
   63.216438] CR2: ffffffffffffffc8
   63.216447] --- end trace 0000000000000000 ]---
   63.216455] RIP: 0010:raw_diag_dump+0x104/0x200 [raw_diag]
   63.216465] Code: 89 c5 49 8b 5e 08 89 dd 83 e5 01 0f 85 ff 00 00 00 4c 89 74 24 08 45 89 e6 89 14 24 eb 0c 48 8b 1b f6 c3 01 0f 85 80 00 00 00 <4c> 3b 7b c8 75 ee 44 39 f5 7c 6a 41 0f b6 45 00 66 39 43 a8 75 5f
   63.216476] RSP: 0018:ffffb4f38368b9a0 EFLAGS: 00010246
   63.216485] RAX: ffff8fcb60245810 RBX: 0000000000000000 RCX: ffff8fccb0fc9348
   63.216494] RDX: 0000000000000000 RSI: ffffffffb0278ec0 RDI: ffffffffb0278ec0
   63.216502] RBP: 0000000000000000 R08: ffffffffaf667cc0 R09: 0000000000000006
   63.216511] R10: ffffdcfcc648f800 R11: 0000000000000090 R12: 0000000000000000
   63.216519] R13: ffff8fcb60245810 R14: 0000000000000000 R15: ffffffffb026fdc0
   63.216527] FS:  00007f522d85c800(0000) GS:ffff8fd03ee00000(0000) knlGS:0000000000000000
   63.216537] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
   63.216545] CR2: ffffffffffffffc8 CR3: 000000023096e002 CR4: 00000000000206f0
   63.216554] note: ss[2540] exited with preempt_count 1

Not necessarily. Stack trace looks entirely software and other systems have different kernel or do not have the same conditions.

   63.215947] BUG: unable to handle page fault for address: ffffffffffffffc8
...
   63.215985] RIP: 0010:raw_diag_dump+0x104/0x200 [raw_diag]
...
   63.216038]  __inet_diag_dump+0x3c/0xb0 [inet_diag 6e5a23768fff3f0f8fbb1d3bede71750b73bbc75]
   63.216045]  netlink_dump+0x123/0x310
   63.216052]  __netlink_dump_start+0x1b6/0x2f0

It sounds like ss triggers kernel in sending information about raw sockets and this function misbheaves.

What is output of “cat /proc/net/raw”?
Do any of “ss -t” or “ss -u” or “ss -x” work?

Thanks for reply, I have:

cat /proc/net/raw
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops

Option for ss works on first try (-t -u and -x) and following tries! But after “ss only”: “Killed” and then unresponsive console.

…found a second TW install with the same problem, ss results in “Killed” and in dmesg

   64.531822] BUG: unable to handle page fault for address: ffffffffffffffc8
   64.531839] #PF: supervisor read access in kernel mode
   64.531844] #PF: error_code(0x0000) - not-present page
   64.531849] PGD 46e15067 P4D 46e15067 PUD 46e17067 PMD 0 
   64.531860] Oops: 0000 #1] PREEMPT SMP PTI
   64.531867] CPU: 3 PID: 1940 Comm: ss Tainted: G         C O      5.19.7-1-default #1 openSUSE Tumbleweed ba7fdbb06c6875547c8f60e0124d59f9a76e7a21
   64.531876] Hardware name: Gigabyte Technology Co., Ltd. Default string/N3150ND3V, BIOS F5a 01/19/2018
   64.531880] RIP: 0010:raw_diag_dump+0x104/0x200 [raw_diag]
   64.531890] Code: 89 c5 49 8b 5e 08 89 dd 83 e5 01 0f 85 ff 00 00 00 4c 89 74 24 08 45 89 e6 89 14 24 eb 0c 48 8b 1b f6 c3 01 0f 85 80 00 00 00 <4c> 3b 7b c8 75 ee 44 39 f5 7c 6a 41 0f b6 45 00 66 39 43 a8 75 5f
   64.531895] RSP: 0018:ffffb39e00d9f910 EFLAGS: 00010246
   64.531901] RAX: ffff8a6804760410 RBX: 0000000000000000 RCX: ffff8a680c488348
   64.531906] RDX: 0000000000000000 RSI: ffffffff8cc78ec0 RDI: ffffffff8cc78ec0
   64.531910] RBP: 0000000000000000 R08: ffffffff8c067cc0 R09: 0000000000000006
   64.531914] R10: ffffe4b884595e00 R11: 0000000000000090 R12: 0000000000000000
   64.531918] R13: ffff8a6804760410 R14: 0000000000000000 R15: ffffffff8cc6fdc0
   64.531923] FS:  00007f4f2d5ac800(0000) GS:ffff8a6977d80000(0000) knlGS:0000000000000000
   64.531928] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
   64.531932] CR2: ffffffffffffffc8 CR3: 000000013ca98000 CR4: 00000000001006e0
   64.531938] Call Trace:
   64.531944]  <TASK>
   64.531952]  __inet_diag_dump+0x3c/0xb0 [inet_diag 87ed22f243ab43ed08e3aae006527f59006217fc]
   64.531966]  netlink_dump+0x123/0x310
   64.531977]  __netlink_dump_start+0x1b6/0x2f0
   64.531985]  inet_diag_handler_cmd+0xb7/0xe0 [inet_diag 87ed22f243ab43ed08e3aae006527f59006217fc]
   64.531994]  ? inet_diag_dump_start_compat+0x10/0x10 [inet_diag 87ed22f243ab43ed08e3aae006527f59006217fc]
   64.532003]  ? inet_diag_dump_compat+0xc0/0xc0 [inet_diag 87ed22f243ab43ed08e3aae006527f59006217fc]
   64.532011]  ? inet_diag_unregister+0x40/0x40 [inet_diag 87ed22f243ab43ed08e3aae006527f59006217fc]
   64.532020]  sock_diag_rcv_msg+0x114/0x140
   64.532030]  ? sock_diag_bind+0x50/0x50
   64.532036]  netlink_rcv_skb+0x51/0x100
   64.532043]  sock_diag_rcv+0x24/0x40
   64.532050]  netlink_unicast+0x239/0x380
   64.532057]  netlink_sendmsg+0x250/0x4c0
   64.532064]  sock_sendmsg+0x5f/0x70
   64.532072]  ____sys_sendmsg+0x22e/0x270
   64.532079]  ? import_iovec+0x17/0x20
   64.532086]  ? sendmsg_copy_msghdr+0x7b/0xa0
   64.532093]  ___sys_sendmsg+0x80/0xc0
   64.532099]  ? ___sys_recvmsg+0x98/0x110
   64.532105]  ? ___sys_recvmsg+0x98/0x110
   64.532110]  ? __wake_up_common_lock+0x87/0xc0
   64.532119]  ? __wake_up_common_lock+0x87/0xc0
   64.532126]  __sys_sendmsg+0x57/0xa0
   64.532133]  do_syscall_64+0x5b/0x80
   64.532142]  ? syscall_exit_to_user_mode+0x17/0x40
   64.532148]  ? do_syscall_64+0x67/0x80
   64.532154]  ? syscall_exit_to_user_mode+0x17/0x40
   64.532160]  ? do_syscall_64+0x67/0x80
   64.532166]  ? __sys_setsockopt+0xe3/0x1e0
   64.532172]  ? syscall_exit_to_user_mode+0x17/0x40
   64.532177]  ? do_syscall_64+0x67/0x80
   64.532183]  ? do_syscall_64+0x67/0x80
   64.532189]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
   64.532197] RIP: 0033:0x7f4f2d7794f4
   64.532203] Code: ff eb b7 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 90 90 80 3d ad c4 0d 00 00 74 13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 48 83 ec 28 89 54 24 1c 48 89
   64.532208] RSP: 002b:00007ffe1883de18 EFLAGS: 00000202 ORIG_RAX: 000000000000002e
   64.532214] RAX: ffffffffffffffda RBX: 000000000000000a RCX: 00007f4f2d7794f4
   64.532218] RDX: 0000000000000000 RSI: 00007ffe1883df00 RDI: 0000000000000003
   64.532222] RBP: 00000000000000ff R08: 0000000000000004 R09: 0000000000000001
   64.532226] R10: 00007ffe1883ddec R11: 0000000000000202 R12: 0000000000000003
   64.532230] R13: 00007ffe1883de80 R14: 0000000003010014 R15: 00007ffe1883dfc0
   64.532237]  </TASK>
   64.532240] Modules linked in: raw_diag unix_diag tcp_diag inet_diag bnep af_packet nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_tables ebtable_nat ebtable_broute iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c iptable_mangle iptable_raw iptable_security ip_set nfnetlink ebtable_filter ebtables ip6_tables iptable_filter bpfilter joydev dmi_sysfs spi_nor mtd at24 intel_rapl_msr iTCO_wdt intel_pmc_bxt spi_intel_platform snd_hda_codec_hdmi spi_intel iTCO_vendor_support mei_hdcp mei_pxp ppdev snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio intel_rapl_common snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec intel_powerclamp coretemp snd_hda_core pcspkr snd_hwdep hci_uart snd_pcm i2c_i801 i2c_smbus lpc_ich btqca snd_timer r8169 btrtl realtek mdio_devres mei_txe intel_xhci_usb_role_switch snd btbcm libphy btintel soundcore roles mei bluetooth fan
   64.532350]  thermal atomisp_lm3554(C) atomisp_gmin_platform(C) parport_pc videodev parport rfkill_gpio ecdh_generic mc rfkill tiny_power_button intel_int0002_vgpio button fuse configfs ip_tables x_tables ext4 mbcache jbd2 hid_generic usbhid crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel i915 drm_buddy xhci_pci xhci_pci_renesas aesni_intel drm_display_helper xhci_hcd crypto_simd cryptd cec usbcore sdhci_pci rc_core cqhci sdhci ttm mmc_core video i2c_hid_acpi i2c_hid pwm_lpss_platform pwm_lpss sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua ledtrig_timer msr
   64.532428] Unloaded tainted modules: acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 fjes():1 fjes():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 bbswitch(O):2
   64.532466] CR2: ffffffffffffffc8
   64.532472] --- end trace 0000000000000000 ]---
   64.532475] RIP: 0010:raw_diag_dump+0x104/0x200 [raw_diag]
   64.532483] Code: 89 c5 49 8b 5e 08 89 dd 83 e5 01 0f 85 ff 00 00 00 4c 89 74 24 08 45 89 e6 89 14 24 eb 0c 48 8b 1b f6 c3 01 0f 85 80 00 00 00 <4c> 3b 7b c8 75 ee 44 39 f5 7c 6a 41 0f b6 45 00 66 39 43 a8 75 5f
   64.532488] RSP: 0018:ffffb39e00d9f910 EFLAGS: 00010246
   64.532493] RAX: ffff8a6804760410 RBX: 0000000000000000 RCX: ffff8a680c488348
   64.532497] RDX: 0000000000000000 RSI: ffffffff8cc78ec0 RDI: ffffffff8cc78ec0
   64.532501] RBP: 0000000000000000 R08: ffffffff8c067cc0 R09: 0000000000000006
   64.532505] R10: ffffe4b884595e00 R11: 0000000000000090 R12: 0000000000000000
   64.532509] R13: ffff8a6804760410 R14: 0000000000000000 R15: ffffffff8cc6fdc0
   64.532513] FS:  00007f4f2d5ac800(0000) GS:ffff8a6977d80000(0000) knlGS:0000000000000000
   64.532518] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
   64.532523] CR2: ffffffffffffffc8 CR3: 000000013ca98000 CR4: 00000000001006e0
   64.532527] note: ss[1940] exited with preempt_count 1


[QUOTE=suse_rasputin;3158332]

cat /proc/net/raw
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops

[/qoute]So it is more complicated :slight_smile:

Option for ss works on first try (-t -u and -x) and following tries!

You should open bug report on bugzilla.opensuse.org.

So it is more complicated :slight_smile:

Option for ss works on first try (-t -u and -x) and following tries!

You should open bug report on bugzilla.opensuse.org.

https://bugzilla.opensuse.org/show_bug.cgi?id=1203380

Hmm… these bug reports are kind of “write it to the wiki”? Nobody cares?!?

Fyi at least is seems to be working on my machine now.


~> grep CPE_NAME /etc/os-release  
**CPE_NAME**="cpe:/o:opensuse:tumbleweed:20220926" 
~> ss|wc 
    374    2983   43384
~>

I’m on the same version, still get “Killed” after typing ss + ENTER…

Well on my machine i do not see this effect even when i boot an old snapshot from 20220912. So just out of curiosity what CPU do you use ?


>inxi -C 
**CPU:**
  **Info:** 6-core **model:** AMD Ryzen 5 5600X **bits:** 64 **type:** MT MCP **cache:**
    **L2:** 3 MiB 
  **Speed (MHz):****avg:** 2195 **min/max:** 2200/3700 **cores:****1:** 2209 **2:** 2208 **3:** 2194 
    **4:** 2191 **5:** 2192 **6:** 2193 **7:** 2193 **8:** 2193 **9:** 2195 **10:** 2195 **11:** 2192 **12:** 2194


various. but only intel insider…

…with kernel 6.0 (or whatever fixed it…) ss apparently works without options here.