wifi broken after upgrade, Thinkpad X1 Carbon (7th)

olebole11 · August 26, 2019, 2:16pm

I have a Thinkpad X1 Carbon (7th), running OpenSUSE Tumbleweed. This morning I upgraded some new packages with
zypper dup --no-allow-vendor-change

Multiple packages where upgraded including the kernel and kernel-firmware - my current versions are:

kernel-default-5.2.9-1.2.x86_64
kernel-firmware-20190815-1.1.noarch

After a restart my wifi stopped working. dmesg continuously and repeatedly shows the following error messages:

 3371.434064] iwlwifi 0000:00:14.3: Microcode SW error detected. Restarting 0x0.
 3371.434160] iwlwifi 0000:00:14.3: Start IWL Error Log Dump:
 3371.434164] iwlwifi 0000:00:14.3: Status: 0x00000080, count: 6
 3371.434167] iwlwifi 0000:00:14.3: Loaded firmware version: 46.93e59cf4.0
 3371.434171] iwlwifi 0000:00:14.3: 0x00000071 | NMI_INTERRUPT_UMAC_FATAL
 3371.434173] iwlwifi 0000:00:14.3: 0x00A022F0 | trm_hw_status0
 3371.434176] iwlwifi 0000:00:14.3: 0x00000000 | trm_hw_status1
 3371.434178] iwlwifi 0000:00:14.3: 0x004881BE | branchlink2
 3371.434181] iwlwifi 0000:00:14.3: 0x0047932A | interruptlink1
 3371.434183] iwlwifi 0000:00:14.3: 0x00015436 | interruptlink2
 3371.434185] iwlwifi 0000:00:14.3: 0x0001A83E | data1
 3371.434187] iwlwifi 0000:00:14.3: 0x00001000 | data2
 3371.434190] iwlwifi 0000:00:14.3: 0xE0000008 | data3
 3371.434192] iwlwifi 0000:00:14.3: 0x00000000 | beacon time
 3371.434194] iwlwifi 0000:00:14.3: 0x0003B142 | tsf low
 3371.434197] iwlwifi 0000:00:14.3: 0x00000000 | tsf hi
 3371.434199] iwlwifi 0000:00:14.3: 0x00000000 | time gp1
 3371.434201] iwlwifi 0000:00:14.3: 0x0003B143 | time gp2
 3371.434203] iwlwifi 0000:00:14.3: 0x00000001 | uCode revision type
 3371.434206] iwlwifi 0000:00:14.3: 0x0000002E | uCode version major
 3371.434208] iwlwifi 0000:00:14.3: 0x93E59CF4 | uCode version minor
 3371.434210] iwlwifi 0000:00:14.3: 0x00000312 | hw version
 3371.434213] iwlwifi 0000:00:14.3: 0x18489008 | board version
 3371.434215] iwlwifi 0000:00:14.3: 0x00210405 | hcmd
 3371.434217] iwlwifi 0000:00:14.3: 0x00022000 | isr0
 3371.434220] iwlwifi 0000:00:14.3: 0x00000000 | isr1
 3371.434222] iwlwifi 0000:00:14.3: 0x20001802 | isr2
 3371.434224] iwlwifi 0000:00:14.3: 0x404000C0 | isr3
 3371.434226] iwlwifi 0000:00:14.3: 0x00000000 | isr4
 3371.434228] iwlwifi 0000:00:14.3: 0x00210405 | last cmd Id
 3371.434231] iwlwifi 0000:00:14.3: 0x0001A83E | wait_event
 3371.434233] iwlwifi 0000:00:14.3: 0x00000000 | l2p_control
 3371.434235] iwlwifi 0000:00:14.3: 0x00000020 | l2p_duration
 3371.434238] iwlwifi 0000:00:14.3: 0x00000000 | l2p_mhvalid
 3371.434240] iwlwifi 0000:00:14.3: 0x00000000 | l2p_addr_match
 3371.434242] iwlwifi 0000:00:14.3: 0x0000000D | lmpm_pmg_sel
 3371.434245] iwlwifi 0000:00:14.3: 0x19071250 | timestamp
 3371.434247] iwlwifi 0000:00:14.3: 0x00001840 | flow_handler
 3371.434333] iwlwifi 0000:00:14.3: Start IWL Error Log Dump:
 3371.434343] iwlwifi 0000:00:14.3: Status: 0x00000080, count: 7
 3371.434353] iwlwifi 0000:00:14.3: 0x20000034 | NMI_INTERRUPT_WDG
 3371.434362] iwlwifi 0000:00:14.3: 0x00000000 | umac branchlink1
 3371.434368] iwlwifi 0000:00:14.3: 0xC008885E | umac branchlink2
 3371.434378] iwlwifi 0000:00:14.3: 0xC0084C54 | umac interruptlink1
 3371.434385] iwlwifi 0000:00:14.3: 0xC0080708 | umac interruptlink2
 3371.434395] iwlwifi 0000:00:14.3: 0x00000400 | umac data1
 3371.434401] iwlwifi 0000:00:14.3: 0xC0080708 | umac data2
 3371.434406] iwlwifi 0000:00:14.3: 0xDEADBEEF | umac data3
 3371.434412] iwlwifi 0000:00:14.3: 0x0000002E | umac major
 3371.434418] iwlwifi 0000:00:14.3: 0x93E59CF4 | umac minor
 3371.434424] iwlwifi 0000:00:14.3: 0x0003B12B | frame pointer
 3371.434429] iwlwifi 0000:00:14.3: 0xC0886C44 | stack pointer
 3371.434435] iwlwifi 0000:00:14.3: 0x00210405 | last host cmd
 3371.434448] iwlwifi 0000:00:14.3: 0x00004008 | isr status reg
 3371.434469] iwlwifi 0000:00:14.3: Fseq Registers:
 3371.434478] iwlwifi 0000:00:14.3: 0x4F3F0710 | FSEQ_ERROR_CODE
 3371.434486] iwlwifi 0000:00:14.3: 0x00000000 | FSEQ_TOP_INIT_VERSION
 3371.434495] iwlwifi 0000:00:14.3: 0xBA74F3E6 | FSEQ_CNVIO_INIT_VERSION
 3371.434503] iwlwifi 0000:00:14.3: 0x0000A384 | FSEQ_OTP_VERSION
 3371.434512] iwlwifi 0000:00:14.3: 0xF8CC6AC6 | FSEQ_TOP_CONTENT_VERSION
 3371.434521] iwlwifi 0000:00:14.3: 0x51158822 | FSEQ_ALIVE_TOKEN
 3371.434530] iwlwifi 0000:00:14.3: 0xDAEB39F6 | FSEQ_CNVI_ID
 3371.434539] iwlwifi 0000:00:14.3: 0x8DE59BA3 | FSEQ_CNVR_ID
 3371.434548] iwlwifi 0000:00:14.3: 0x01000100 | CNVI_AUX_MISC_CHIP
 3371.434559] iwlwifi 0000:00:14.3: 0x01300202 | CNVR_AUX_MISC_CHIP
 3371.434570] iwlwifi 0000:00:14.3: 0x0000485B | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
 3371.434613] iwlwifi 0000:00:14.3: 0xA5A5A5A2 | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
 3371.434626] iwlwifi 0000:00:14.3: Collecting data: trigger 2 fired.
 3371.434636] ieee80211 phy1746: Hardware restart was requested
 3371.434685] iwlwifi 0000:00:14.3: FW error in SYNC CMD GEO_TX_POWER_LIMIT
 3371.434700] CPU: 7 PID: 2080 Comm: wpa_supplicant Not tainted 5.2.9-1-default #1 openSUSE Tumbleweed (unreleased)
 3371.434707] Hardware name: LENOVO 20QDCTO1WW/20QDCTO1WW, BIOS N2HET21W (1.04 ) 06/03/2019
 3371.434709] Call Trace:
 3371.434729]  dump_stack+0x85/0xc0
 3371.434764]  iwl_pcie_send_hcmd_sync+0x45b/0x480 [iwlwifi]
 3371.434778]  ? wait_woken+0x70/0x70
 3371.434807]  iwl_trans_send_cmd+0x59/0xb0 [iwlwifi]
 3371.434880]  iwl_mvm_send_cmd+0x2e/0x80 [iwlmvm]
 3371.434908]  iwl_mvm_send_cmd_pdu+0x43/0x50 [iwlmvm]
 3371.434930]  iwl_mvm_up+0x979/0xa10 [iwlmvm]
 3371.434955]  __iwl_mvm_mac_start+0x1c6/0x2d0 [iwlmvm]
 3371.434978]  iwl_mvm_mac_start+0x36/0xf0 [iwlmvm]
 3371.435040]  drv_start+0x43/0xf0 [mac80211]
 3371.435101]  ieee80211_do_open+0x21a/0x830 [mac80211]
 3371.435154]  ? ieee80211_check_concurrent_iface+0x14f/0x1c0 [mac80211]
 3371.435166]  __dev_open+0xb5/0x150
 3371.435174]  __dev_change_flags+0x15d/0x1b0
 3371.435181]  dev_change_flags+0x21/0x60
 3371.435188]  devinet_ioctl+0x5a9/0x6f0
 3371.435196]  inet_ioctl+0x190/0x1c0
 3371.435205]  ? filename_lookup+0xda/0x150
 3371.435210]  ? dev_get_by_name_rcu+0x6e/0x90
 3371.435214]  ? dev_get_by_name_rcu+0x6e/0x90
 3371.435222]  sock_do_ioctl+0x47/0x140
 3371.435231]  sock_ioctl+0x24f/0x3c0
 3371.435238]  do_vfs_ioctl+0x447/0x640
 3371.435245]  ? __do_sys_newlstat+0x35/0x40
 3371.435251]  ksys_ioctl+0x5e/0x90
 3371.435259]  ? trace_hardirqs_off_thunk+0x1a/0x33
 3371.435266]  __x64_sys_ioctl+0x16/0x20
 3371.435272]  do_syscall_64+0x6e/0x1e0
 3371.435281]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
 3371.435288] RIP: 0033:0x7f4a428087c7
 3371.435296] Code: 00 00 90 48 8b 05 d1 86 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a1 86 0c 00 f7 d8 64 89 01 48
 3371.435300] RSP: 002b:00007fffd81aa048 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
 3371.435308] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f4a428087c7
 3371.435311] RDX: 00007fffd81aa050 RSI: 0000000000008914 RDI: 0000000000000009
 3371.435314] RBP: 0000000000000009 R08: 00007f4a428d1aa0 R09: 0000000000000007
 3371.435317] R10: 000055a0e2d7b010 R11: 0000000000000246 R12: 0000000000000000
 3371.435319] R13: 000055a0e2d8cef8 R14: 00007fffd81aa050 R15: 00007fffd81aa3a0
 3371.435335] iwlwifi 0000:00:14.3: Firmware not running - cannot dump error
 3371.757832] iwlwifi 0000:00:14.3: Found debug destination: EXTERNAL_DRAM
 3371.757838] iwlwifi 0000:00:14.3: Found debug configuration: 0
 3371.758674] iwlwifi 0000:00:14.3: loaded firmware version 46.93e59cf4.0 op_mode iwlmvm
 3371.758718] iwlwifi 0000:00:14.3: Detected Intel(R) Wireless-AC 9560 160MHz, REV=0x318
 3371.767832] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
 3371.768593] iwlwifi 0000:00:14.3: Allocated 0x00400000 bytes for firmware monitor.
 3371.809946] iwlwifi 0000:00:14.3: base HW address: dc:fb:48:bc:74:79
 3371.881296] ieee80211 phy1747: Selected rate control algorithm 'iwl-mvm-rs'
 3371.881840] thermal thermal_zone5: failed to read out thermal zone (-61)
 3371.883027] iwlwifi 0000:00:14.3 wlp0s20f3: renamed from wlan0
 3371.951345] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
 3372.067268] iwlwifi 0000:00:14.3: Applying debug destination EXTERNAL_DRAM
 3372.131762] iwlwifi 0000:00:14.3: FW already configured (0) - re-configuring

Below is my network controller info:

# lspci -kvnn | sed -n '/Network/,/^$/ p'
00:14.3 Network controller [0280]: Intel Corporation Cannon Point-LP CNVi [Wireless-AC] [8086:9df0] (rev 11)
        Subsystem: Intel Corporation Device [8086:0030]
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at ea238000 (64-bit, non-prefetchable) [size=16]
        Capabilities: [c8] Power Management version 3
        Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [80] MSI-X: Enable+ Count=16 Masked-
        Capabilities: [100] Null
        Capabilities: [14c] Latency Tolerance Reporting
        Capabilities: [164] Vendor Specific Information: ID=0010 Rev=0 Len=014 <?>
        Kernel driver in use: iwlwifi
        Kernel modules: iwlwifi

I tried to boot an older kernel version (kernel-default-5.2.8-1.1) which didn’t solve anything.
I thought that maybe rolling back to an older version of kernel-firmware could help, but I haven’t figured out how to do that. I simple cannot seem to find an older version.

I hope someone can help me. Thanks.

Best regards,
olebole11[/size]

nrickert · August 26, 2019, 4:42pm

If you are using “btrfs”, then you could do a snapper rollback to an earlier snapshot where everything worked.

Or check with the history at http://download.opensuse.org/history/. Find a suitable snapshot date, then look there for the repo as of that date. You should be able to download the firmware rpm and install manually.

Or install “tumbleweed-cli” and then you have a fancy way of switching back to earlier versions of Tumbleweed.

knurpht · August 26, 2019, 6:13pm

Install kernel-firmware-iwlwifi and see if that makes a difference. Just read that the kernel-firmware packages on TW are now split into separate packages.

olebole11 · August 26, 2019, 9:07pm

Thank you very much. I didn’t know about tumbleweed-cli and switching between snapshots. Very nice. I reverted back to an earlier version which I remembered to work (20190815) and this solved my problems.

If I can ask you, how does future updates work now when I switched to the snapshot based approach? Do I just keep track of new snapshots myself and update when I feel like it or do I get some kind of status notifications (like the old dialog notifying about new package updates)?

I tried to install the mentioned package but as far as I can tell it required a newer kernel version (5.3.x) than were available to me and I didn’t dare to move to kernel:HEAD or something like that But thank you for taking your time to help me.

nrickert · August 26, 2019, 10:39pm

You can occasionally check the download page: http://download.opensuse.org/tumbleweed/iso/
That will tell you what’s the latest available.

Or you could just lock the firmware, and any other package that you think might be related to the problem. And then update to the current release (except for the locked packages). And consider filing a bug report about the broken package.

awice · August 27, 2019, 6:59am

Thanks for this post and the replies. I ran into the same problem on a Lenovo X1 Yoga (Gen 4) and was able to resolve with the following:

Download http://download.opensuse.org/history/20190815/tumbleweed/repo/oss/noarch/kernel-firmware-20190712-1.1.noarch.rpm

zypper install --oldpackage kernel-firmware-20190712-1.1.noarch.rpm

Reboot. After reboot wireless should work. Run the following command to force the package manager to keep that version.

zypper addlock kernel-firmware