Dealing with BTRFS errors on a root partition in NVMe M2 PCIe

Scenario:

I
have installed a test machine
Opensuse
Leap 15.3
Motherboard
Asustek M4A89TD
PRO USB3
CPU
AMD Athlon™ II X3 450 Processor
8
GB memory DDR3
Nvidia
GF108 [GeForce GT 630]
“Hard disk” Crucial CT1000P2SSD8 NVMe
M.2 1TB connected
through Mzhou M.2 SSD M-key NVM PCIe 3.0 x1 adapter
USB
3.1 flash drive Magic stealth 32GB

I installed leap 15.2 using the usb 3.1 for /boot and the M.2 for / (everything else).
The use of the usb for boot is because the motherboard is not capable of booting from the PCIe

The installation went fine, everything worked ok, then I left the system unused some weeks and leap 15.3 appeared, so I did a online upgrade, it worked fine to. I have been working with it testing with no error. But in some moment I began to have issues with the system becoming read only. I detected that the system was switching to RO when I was using firefox and I remembered I had changed firefox from the version in opensuse repos to the version in mozilla repo, then I was able to “resolve” the issues desinstalling firefox and installing it again (I switched back to the version in the opensuse repo, but once “solved” the error I switched again to the mozilla repo version with no errors). So it seems that firefox was installed in a trouble maker area of the disk and causing the issues.

So, here I am, the filesystem still has errors, and I’m trying to fix them

Booting the system and running smartctl


 ***pruebas:~ #**** smartctl --xall /dev/nvme0n1                
smartctl 7.0 2019-05-21 r4917 [x86_64-linux-5.3.18-59.13-default] (SUSE RPM) 
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org 

=== START OF INFORMATION SECTION === 
Model Number:                       CT1000P2SSD8 
Serial Number:                      2051E4DF0793 
Firmware Version:                   P2CR031 
PCI Vendor/Subsystem ID:            0xc0a9 
IEEE OUI Identifier:                0x6479a7 
Total NVM Capacity:                 1,000,204,886,016 [1.00 TB] 
Unallocated NVM Capacity:           0 
Controller ID:                      1 
Number of Namespaces:               1 
Namespace 1 Size/Capacity:          1,000,204,886,016 [1.00 TB] 
Namespace 1 Formatted LBA Size:     512 
Namespace 1 IEEE EUI-64:            6479a7 fff0000000 
Local Time is:                      Wed Jul 21 20:22:03 2021 CEST 
Firmware Updates (0x12):            1 Slot, no Reset required 
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test 
Optional NVM Commands (0x005e):     Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp 
Maximum Data Transfer Size:         64 Pages 
Warning  Comp. Temp. Threshold:     70 Celsius 
Critical Comp. Temp. Threshold:     85 Celsius 

Supported Power States 
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat 
0 +     3.50W       -        -    0  0  0  0        0       0 
1 +     1.90W       -        -    1  1  1  1        0       0 
2 +     1.50W       -        -    2  2  2  2        0       0 
3 -   0.0700W       -        -    3  3  3  3     5000    1900 
4 -   0.0020W       -        -    4  4  4  4    13000  100000 

Supported LBA Sizes (NSID 0x1) 
Id Fmt  Data  Metadt  Rel_Perf 
0 +     512       0         1 
1 -    4096       0         0 

=== START OF SMART DATA SECTION === 
SMART overall-health self-assessment test result: PASSED 

SMART/Health Information (NVMe Log 0x02) 
Critical Warning:                   0x00 
Temperature:                        45 Celsius 
Available Spare:                    100% 
Available Spare Threshold:          5% 
Percentage Used:                    0% 
Data Units Read:                    438,752 [224 GB] 
Data Units Written:                 421,564 [215 GB] 
Host Read Commands:                 4,129,572 
Host Write Commands:                2,210,069 
Controller Busy Time:               75 
Power Cycles:                       79 
Power On Hours:                     173 
Unsafe Shutdowns:                   16 
Media and Data Integrity Errors:    0 
Error Information Log Entries:      0 
Warning  Comp. Temperature Time:    0 
Critical Comp. Temperature Time:    0 

Error Information (NVMe Log 0x01, max 16 entries) 
No Errors Logged *  

Then* booting the system with a rescue disk, and with btrfs 4.19.1
*

 *#btrfs check --readonly --force /dev/nvme0n1p1*
 *[1/7] checking root items*
 *[2/7] checking extents*
 *data backref 227831808 root 263 owner 7983 offset 0 num_refs 0 not found in extent tree*
 *incorrect local backref count on 227831808 root 263 owner 7983 offset 0 found 1 wanted 0 back 0x5559e0ab7020*
 *incorrect local backref count on 227831808 root 263 owner 140737488363311 offset 0 found 0 wanted 1 back 0x5559dde718d0*
 *backref disk bytenr does not match extent record, bytenr=227831808, ref bytenr=0*
 *backpointer mismatch on [227831808 4096]*
 *ERROR: errors found in extent allocation tree or chunk allocation*
 *[3/7] checking free space cache*
 *[4/7] checking fs roots*
 *[5/7] checking only csums items (without verifying data)*
 *[6/7] checking root refs*
 *[7/7] checking quota groups*
 *Qgroup are marked as inconsistent.*
 *Opening filesystem to check...*
 *Checking filesystem on /dev/nvme0n1p1*
 *UUID: 5b000355-3a1a-49f5-8005-f10668008aa7*
 *Rescan hasn't been initialized, a difference in qgroup accounting is expected*
 *found 51878920192 bytes used, error(s) found*
 *total csum bytes: 48135312*
 *total tree bytes: 991313920*
 *total fs tree bytes: 885358592*
 *total extent tree bytes: 48414720*
 *btree space waste bytes: 151592274*
 *file data blocks allocated: 239972728832*
  *referenced 85539778560*

  

But I’m not sure what should be the next step. If I were using ext4 I will do a fsck to repair the fs, but reading the btrfs doc it seems the –repair option is dangerous.

I don’t really have important information in the system, I’m using it with a “test” user and I’m mounting my info filesystems from a remote system using sshfs, so worst case scenario I will have to reinstall the system and all the programs I installed.
I will appreciate any piece of advice.

The problem shouldn’t have occurred. Check your drive for problems. Use filter: ‘journalctl _KERNEL_SUBSYSTEM=scsi _KERNEL_SUBSYSTEM=usb’.

pruebas:~ # journalctl _KERNEL_SUBSYSTEM=scsi _KERNEL_SUBSYSTEM=usb --no-pager
-- Logs begin at Thu 2021-07-22 10:53:46 CEST, end at Thu 2021-07-22 11:09:31 CEST. --
Jul 22 10:53:47 pruebas kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.03
Jul 22 10:53:47 pruebas kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jul 22 10:53:47 pruebas kernel: usb usb1: Product: EHCI Host Controller
Jul 22 10:53:47 pruebas kernel: usb usb1: Manufacturer: Linux 5.3.18-59.13-default ehci_hcd
Jul 22 10:53:47 pruebas kernel: usb usb1: SerialNumber: 0000:00:12.2
Jul 22 10:53:47 pruebas kernel: scsi host0: pata_jmicron
Jul 22 10:53:47 pruebas kernel: scsi host2: pata_atiixp
Jul 22 10:53:47 pruebas kernel: hub 1-0:1.0: USB hub found
Jul 22 10:53:47 pruebas kernel: hub 1-0:1.0: 5 ports detected
Jul 22 10:53:47 pruebas kernel: scsi host1: ahci
Jul 22 10:53:47 pruebas kernel: scsi host3: pata_jmicron
Jul 22 10:53:47 pruebas kernel: scsi host4: pata_atiixp
Jul 22 10:53:47 pruebas kernel: scsi host5: ahci
Jul 22 10:53:47 pruebas kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.03
Jul 22 10:53:47 pruebas kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jul 22 10:53:47 pruebas kernel: usb usb2: Product: xHCI Host Controller
Jul 22 10:53:47 pruebas kernel: usb usb2: Manufacturer: Linux 5.3.18-59.13-default xhci-hcd
Jul 22 10:53:47 pruebas kernel: usb usb2: SerialNumber: 0000:03:00.0
Jul 22 10:53:47 pruebas kernel: scsi host6: ahci
Jul 22 10:53:47 pruebas kernel: hub 2-0:1.0: USB hub found
Jul 22 10:53:47 pruebas kernel: scsi host7: ahci
Jul 22 10:53:47 pruebas kernel: hub 2-0:1.0: 2 ports detected
Jul 22 10:53:47 pruebas kernel: usb usb3: We don't know the algorithms for LPM for this host, disabling LPM.
Jul 22 10:53:47 pruebas kernel: usb usb3: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.03
Jul 22 10:53:47 pruebas kernel: usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jul 22 10:53:47 pruebas kernel: usb usb3: Product: xHCI Host Controller
Jul 22 10:53:47 pruebas kernel: usb usb3: Manufacturer: Linux 5.3.18-59.13-default xhci-hcd
Jul 22 10:53:47 pruebas kernel: usb usb3: SerialNumber: 0000:03:00.0
Jul 22 10:53:47 pruebas kernel: hub 3-0:1.0: USB hub found
Jul 22 10:53:47 pruebas kernel: hub 3-0:1.0: 2 ports detected
Jul 22 10:53:47 pruebas kernel: scsi host8: ahci
Jul 22 10:53:47 pruebas kernel: usb usb4: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.03
Jul 22 10:53:47 pruebas kernel: usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jul 22 10:53:47 pruebas kernel: usb usb4: Product: EHCI Host Controller
Jul 22 10:53:47 pruebas kernel: usb usb4: Manufacturer: Linux 5.3.18-59.13-default ehci_hcd
Jul 22 10:53:47 pruebas kernel: usb usb4: SerialNumber: 0000:00:13.2
Jul 22 10:53:47 pruebas kernel: hub 4-0:1.0: USB hub found
Jul 22 10:53:47 pruebas kernel: hub 4-0:1.0: 5 ports detected
Jul 22 10:53:47 pruebas kernel: scsi host9: ahci
Jul 22 10:53:48 pruebas kernel: usb usb5: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.03
Jul 22 10:53:48 pruebas kernel: usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jul 22 10:53:48 pruebas kernel: usb usb5: Product: EHCI Host Controller
Jul 22 10:53:48 pruebas kernel: usb usb5: Manufacturer: Linux 5.3.18-59.13-default ehci_hcd
Jul 22 10:53:48 pruebas kernel: usb usb5: SerialNumber: 0000:00:16.2
Jul 22 10:53:48 pruebas kernel: hub 5-0:1.0: USB hub found
Jul 22 10:53:48 pruebas kernel: hub 5-0:1.0: 4 ports detected
Jul 22 10:53:48 pruebas kernel: usb usb6: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 5.03
Jul 22 10:53:48 pruebas kernel: usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jul 22 10:53:48 pruebas kernel: usb usb6: Product: OHCI PCI host controller
Jul 22 10:53:48 pruebas kernel: usb usb6: Manufacturer: Linux 5.3.18-59.13-default ohci_hcd
Jul 22 10:53:48 pruebas kernel: usb usb6: SerialNumber: 0000:00:12.0
Jul 22 10:53:48 pruebas kernel: hub 6-0:1.0: USB hub found
Jul 22 10:53:48 pruebas kernel: hub 6-0:1.0: 5 ports detected
Jul 22 10:53:48 pruebas kernel: usb 4-1: new high-speed USB device number 2 using ehci-pci
Jul 22 10:53:48 pruebas kernel: usb usb7: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 5.03
Jul 22 10:53:48 pruebas kernel: usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jul 22 10:53:48 pruebas kernel: usb usb7: Product: OHCI PCI host controller
Jul 22 10:53:48 pruebas kernel: usb usb7: Manufacturer: Linux 5.3.18-59.13-default ohci_hcd
Jul 22 10:53:48 pruebas kernel: usb usb7: SerialNumber: 0000:00:13.0
Jul 22 10:53:48 pruebas kernel: hub 7-0:1.0: USB hub found
Jul 22 10:53:48 pruebas kernel: hub 7-0:1.0: 5 ports detected
Jul 22 10:53:49 pruebas kernel: usb usb8: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 5.03
Jul 22 10:53:49 pruebas kernel: usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jul 22 10:53:49 pruebas kernel: usb usb8: Product: OHCI PCI host controller
Jul 22 10:53:49 pruebas kernel: usb usb8: Manufacturer: Linux 5.3.18-59.13-default ohci_hcd
Jul 22 10:53:49 pruebas kernel: usb usb8: SerialNumber: 0000:00:14.5
Jul 22 10:53:49 pruebas kernel: hub 8-0:1.0: USB hub found
Jul 22 10:53:49 pruebas kernel: hub 8-0:1.0: 2 ports detected
Jul 22 10:53:49 pruebas kernel: usb usb9: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 5.03
Jul 22 10:53:49 pruebas kernel: usb usb9: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Jul 22 10:53:49 pruebas kernel: usb usb9: Product: OHCI PCI host controller
Jul 22 10:53:49 pruebas kernel: usb usb9: Manufacturer: Linux 5.3.18-59.13-default ohci_hcd
Jul 22 10:53:49 pruebas kernel: usb usb9: SerialNumber: 0000:00:16.0
Jul 22 10:53:49 pruebas kernel: hub 9-0:1.0: USB hub found
Jul 22 10:53:49 pruebas kernel: hub 9-0:1.0: 4 ports detected
Jul 22 10:53:49 pruebas kernel: usb 6-3: new full-speed USB device number 2 using ohci-pci
Jul 22 10:53:49 pruebas kernel: usb 6-3: New USB device found, idVendor=145f, idProduct=02cc, bcdDevice= 2.00
Jul 22 10:53:49 pruebas kernel: usb 6-3: New USB device strings: Mfr=0, Product=1, SerialNumber=0
Jul 22 10:53:49 pruebas kernel: usb 6-3: Product: Trust Deskset
Jul 22 10:53:49 pruebas kernel: usb 4-1: New USB device found, idVendor=13fe, idProduct=6300, bcdDevice= 1.00
Jul 22 10:53:49 pruebas kernel: usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jul 22 10:53:49 pruebas kernel: usb 4-1: Product: USB DISK 3.0
Jul 22 10:53:49 pruebas kernel: usb 4-1: Manufacturer:          
Jul 22 10:53:49 pruebas kernel: usb 4-1: SerialNumber: 070A119A8067E650
Jul 22 10:53:49 pruebas kernel: usb-storage 4-1:1.0: USB Mass Storage device detected
Jul 22 10:53:49 pruebas kernel: scsi host10: usb-storage 4-1:1.0
Jul 22 10:53:51 pruebas kernel: scsi 10:0:0:0: Direct-Access              USB DISK 3.0     PMAP PQ: 0 ANSI: 6
Jul 22 10:53:51 pruebas kernel: scsi 10:0:0:0: Attached scsi generic sg0 type 0
Jul 22 10:53:51 pruebas kernel: sd 10:0:0:0: [sda] 60628992 512-byte logical blocks: (31.0 GB/28.9 GiB)
Jul 22 10:53:51 pruebas kernel: sd 10:0:0:0: [sda] Write Protect is off
Jul 22 10:53:51 pruebas kernel: sd 10:0:0:0: [sda] Mode Sense: 23 00 00 00
Jul 22 10:53:51 pruebas kernel: sd 10:0:0:0: [sda] No Caching mode page found
Jul 22 10:53:51 pruebas kernel: sd 10:0:0:0: [sda] Assuming drive cache: write through
Jul 22 10:53:51 pruebas kernel: sd 10:0:0:0: [sda] Attached SCSI removable disk

I can’t see anything strange

I think the information of the PCIe M.2 should be here, but I can’t see any error either.

pruebas:~ # journalctl _KERNEL_SUBSYSTEM=pci  --no-pager                          
-- Logs begin at Thu 2021-07-22 10:53:46 CEST, end at Thu 2021-07-22 11:16:40 CEST. --
 pci 0000:00:00.0: [1002:5a11] type 00 class 0x060000
 pci 0000:00:02.0: [1002:5a16] type 01 class 0x060400
 pci 0000:00:02.0: enabling Extended Tags
 pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
 pci 0000:00:04.0: [1002:5a18] type 01 class 0x060400
 pci 0000:00:04.0: enabling Extended Tags
 pci 0000:00:04.0: PME# supported from D0 D3hot D3cold
 pci 0000:00:05.0: [1002:5a19] type 01 class 0x060400
 pci 0000:00:05.0: enabling Extended Tags
 pci 0000:00:05.0: PME# supported from D0 D3hot D3cold
 pci 0000:00:06.0: [1002:5a1a] type 01 class 0x060400
 pci 0000:00:06.0: enabling Extended Tags
 pci 0000:00:06.0: PME# supported from D0 D3hot D3cold
 pci 0000:00:07.0: [1002:5a1b] type 01 class 0x060400
 pci 0000:00:07.0: enabling Extended Tags
 pci 0000:00:07.0: PME# supported from D0 D3hot D3cold
 pci 0000:00:0a.0: [1002:5a1d] type 01 class 0x060400
 pci 0000:00:0a.0: enabling Extended Tags
 pci 0000:00:0a.0: PME# supported from D0 D3hot D3cold
 pci 0000:00:11.0: [1002:4390] type 00 class 0x01018f
 pci 0000:00:11.0: reg 0x10: [io  0xa000-0xa007]
 pci 0000:00:11.0: reg 0x14: [io  0x9000-0x9003]
 pci 0000:00:11.0: reg 0x18: [io  0x8000-0x8007]
 pci 0000:00:11.0: reg 0x1c: [io  0x7000-0x7003]
 pci 0000:00:11.0: reg 0x20: [io  0x6000-0x600f]
 pci 0000:00:11.0: reg 0x24: [mem 0xfcafe000-0xfcafe3ff]
 pci 0000:00:11.0: set SATA to AHCI mode
 pci 0000:00:12.0: [1002:4397] type 00 class 0x0c0310
 pci 0000:00:12.0: reg 0x10: [mem 0xfcaf7000-0xfcaf7fff]
 pci 0000:00:12.2: [1002:4396] type 00 class 0x0c0320
 pci 0000:00:12.2: reg 0x10: [mem 0xfcafe400-0xfcafe4ff]
 pci 0000:00:12.2: supports D1 D2
 pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot
 pci 0000:00:13.0: [1002:4397] type 00 class 0x0c0310
 pci 0000:00:13.0: reg 0x10: [mem 0xfcafc000-0xfcafcfff]
 pci 0000:00:13.2: [1002:4396] type 00 class 0x0c0320
 pci 0000:00:13.2: reg 0x10: [mem 0xfcafe800-0xfcafe8ff]
 pci 0000:00:13.2: supports D1 D2
 pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
 pci 0000:00:14.0: [1002:4385] type 00 class 0x0c0500
 pci 0000:00:14.1: [1002:439c] type 00 class 0x01018a
 pci 0000:00:14.1: reg 0x10: [io  0x0000-0x0007]
 pci 0000:00:14.1: reg 0x14: [io  0x0000-0x0003]
 pci 0000:00:14.1: reg 0x18: [io  0x0000-0x0007]
 pci 0000:00:14.1: reg 0x1c: [io  0x0000-0x0003]
 pci 0000:00:14.1: reg 0x20: [io  0xff00-0xff0f]
 pci 0000:00:14.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
 pci 0000:00:14.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
 pci 0000:00:14.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
 pci 0000:00:14.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
 pci 0000:00:14.2: [1002:4383] type 00 class 0x040300
 pci 0000:00:14.2: reg 0x10: [mem 0xfcaf8000-0xfcafbfff 64bit]
 pci 0000:00:14.2: PME# supported from D0 D3hot D3cold
 pci 0000:00:14.3: [1002:439d] type 00 class 0x060100
 pci 0000:00:14.4: [1002:4384] type 01 class 0x060401
 pci 0000:00:14.5: [1002:4399] type 00 class 0x0c0310
 pci 0000:00:14.5: reg 0x10: [mem 0xfcafd000-0xfcafdfff]
 pci 0000:00:16.0: [1002:4397] type 00 class 0x0c0310
 pci 0000:00:16.0: reg 0x10: [mem 0xfcaff000-0xfcafffff]
 pci 0000:00:16.2: [1002:4396] type 00 class 0x0c0320
 pci 0000:00:16.2: reg 0x10: [mem 0xfcafec00-0xfcafecff]
 pci 0000:00:16.2: supports D1 D2
 pci 0000:00:16.2: PME# supported from D0 D1 D2 D3hot
 pci 0000:00:18.0: [1022:1200] type 00 class 0x060000
 pci 0000:00:18.1: [1022:1201] type 00 class 0x060000
 pci 0000:00:18.2: [1022:1202] type 00 class 0x060000
 pci 0000:00:18.3: [1022:1203] type 00 class 0x060000
 pci 0000:00:18.4: [1022:1204] type 00 class 0x060000
 pci 0000:07:00.0: [10de:0f00] type 00 class 0x030000
 pci 0000:07:00.0: reg 0x10: [mem 0xfd000000-0xfdffffff]
 pci 0000:07:00.0: reg 0x14: [mem 0xd8000000-0xdfffffff 64bit pref]
 pci 0000:07:00.0: reg 0x1c: [mem 0xd6000000-0xd7ffffff 64bit pref]
 pci 0000:07:00.0: reg 0x24: [io  0xec00-0xec7f]
 pci 0000:07:00.0: reg 0x30: [mem 0xfe980000-0xfe9fffff pref]
 pci 0000:07:00.0: enabling Extended Tags
 pci 0000:07:00.1: [10de:0bea] type 00 class 0x040300
 pci 0000:07:00.1: reg 0x10: [mem 0xfe97c000-0xfe97ffff]
 pci 0000:07:00.1: enabling Extended Tags
 pci 0000:00:02.0: PCI bridge to [bus 07]
 pci 0000:00:02.0:   bridge window [io  0xe000-0xefff]
 pci 0000:00:02.0:   bridge window [mem 0xfd000000-0xfe9fffff]
 pci 0000:00:02.0:   bridge window [mem 0xd6000000-0xdfffffff 64bit pref]
 pci 0000:06:00.0: [197b:2361] type 00 class 0x010185
 pci 0000:06:00.0: reg 0x24: [mem 0xfcffe000-0xfcffffff]
 pci 0000:06:00.0: PME# supported from D3hot
 pci 0000:06:00.1: [197b:2361] type 00 class 0x010185
 pci 0000:06:00.1: reg 0x10: [io  0xdc00-0xdc07]
 pci 0000:06:00.1: reg 0x14: [io  0xd880-0xd883]
 pci 0000:06:00.1: reg 0x18: [io  0xd800-0xd807]
 pci 0000:06:00.1: reg 0x1c: [io  0xd480-0xd483]
 pci 0000:06:00.1: reg 0x20: [io  0xd400-0xd40f]
 pci 0000:06:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
 pci 0000:00:04.0: PCI bridge to [bus 06]
 pci 0000:00:04.0:   bridge window [io  0xd000-0xdfff]
 pci 0000:00:04.0:   bridge window [mem 0xfcf00000-0xfcffffff]
 pci 0000:05:00.0: [1106:3403] type 00 class 0x0c0010
 pci 0000:05:00.0: reg 0x10: [mem 0xfceff800-0xfcefffff 64bit]
 pci 0000:05:00.0: reg 0x18: [io  0xc800-0xc8ff]
 pci 0000:05:00.0: supports D2
 pci 0000:05:00.0: PME# supported from D2 D3hot D3cold
 pci 0000:00:05.0: PCI bridge to [bus 05]
 pci 0000:00:05.0:   bridge window [io  0xc000-0xcfff]
 pci 0000:00:05.0:   bridge window [mem 0xfce00000-0xfcefffff]
 pci 0000:04:00.0: [10ec:8168] type 00 class 0x020000
 pci 0000:04:00.0: reg 0x10: [io  0xb800-0xb8ff]
 pci 0000:04:00.0: reg 0x18: [mem 0xd5fff000-0xd5ffffff 64bit pref]
 pci 0000:04:00.0: reg 0x20: [mem 0xd5ff8000-0xd5ffbfff 64bit pref]
 pci 0000:04:00.0: reg 0x30: [mem 0xfcde0000-0xfcdfffff pref]
 pci 0000:04:00.0: supports D1 D2
 pci 0000:04:00.0: PME# supported from D0 D1 D2 D3hot D3cold
 pci 0000:00:06.0: PCI bridge to [bus 04]
 pci 0000:00:06.0:   bridge window [io  0xb000-0xbfff]
 pci 0000:00:06.0:   bridge window [mem 0xfcd00000-0xfcdfffff]
 pci 0000:00:06.0:   bridge window [mem 0xd5f00000-0xd5ffffff 64bit pref]
 pci 0000:03:00.0: [1033:0194] type 00 class 0x0c0330
 pci 0000:03:00.0: reg 0x10: [mem 0xfccfe000-0xfccfffff 64bit]
 pci 0000:03:00.0: PME# supported from D0 D3hot D3cold
 pci 0000:00:07.0: PCI bridge to [bus 03]
 pci 0000:00:07.0:   bridge window [mem 0xfcc00000-0xfccfffff]
 pci 0000:02:00.0: [c0a9:540a] type 00 class 0x010802
 pci 0000:02:00.0: reg 0x10: [mem 0xfcbfc000-0xfcbfffff 64bit]
 pci 0000:02:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:0a.0 (capable of 3
1.504 Gb/s with 8 GT/s x4 link)
 pci 0000:00:0a.0: PCI bridge to [bus 02]
 pci 0000:00:0a.0:   bridge window [mem 0xfcb00000-0xfcbfffff]
 pci 0000:00:14.4: PCI bridge to [bus 01] (subtractive decode)
 pci 0000:00:14.4:   bridge window [io  0x0000-0x0cf7 window] (subtractive decode)
 pci 0000:00:14.4:   bridge window [io  0x0d00-0xffff window] (subtractive decode)
 pci 0000:00:14.4:   bridge window [mem 0x000a0000-0x000bffff window] (subtractive decode)
 pci 0000:00:14.4:   bridge window [mem 0x000d0000-0x000dffff window] (subtractive decode)
 pci 0000:00:14.4:   bridge window [mem 0xcff00000-0xdfffffff window] (subtractive decode)
 pci 0000:00:14.4:   bridge window [mem 0xf0000000-0xfebfffff window] (subtractive decode)
 pci 0000:07:00.0: vgaarb: setting as boot VGA device
 pci 0000:07:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
 pci 0000:07:00.0: vgaarb: bridge control possible
 pci 0000:00:02.0: PCI bridge to [bus 07]
 pci 0000:00:02.0:   bridge window [io  0xe000-0xefff]
 pci 0000:00:02.0:   bridge window [mem 0xfd000000-0xfe9fffff]
 pci 0000:00:02.0:   bridge window [mem 0xd6000000-0xdfffffff 64bit pref]
 pci 0000:00:04.0: PCI bridge to [bus 06]
 pci 0000:00:04.0:   bridge window [io  0xd000-0xdfff]
 pci 0000:00:04.0:   bridge window [mem 0xfcf00000-0xfcffffff]
 pci 0000:00:05.0: PCI bridge to [bus 05]
 pci 0000:00:05.0:   bridge window [io  0xc000-0xcfff]
 pci 0000:00:05.0:   bridge window [mem 0xfce00000-0xfcefffff]
 pci 0000:00:06.0: PCI bridge to [bus 04]
 pci 0000:00:06.0:   bridge window [io  0xb000-0xbfff]
 pci 0000:00:06.0:   bridge window [mem 0xfcd00000-0xfcdfffff]
 pci 0000:00:06.0:   bridge window [mem 0xd5f00000-0xd5ffffff 64bit pref]
 pci 0000:00:07.0: PCI bridge to [bus 03]
 pci 0000:00:07.0:   bridge window [mem 0xfcc00000-0xfccfffff]
 pci 0000:00:0a.0: PCI bridge to [bus 02]
 pci 0000:00:0a.0:   bridge window [mem 0xfcb00000-0xfcbfffff]
 pci 0000:00:14.4: PCI bridge to [bus 01]
 pci 0000:00:12.0: quirk_usb_early_handoff+0x0/0x690 took 768782 usecs
 pci 0000:00:13.0: quirk_usb_early_handoff+0x0/0x690 took 27203 usecs
 pci 0000:00:14.5: quirk_usb_early_handoff+0x0/0x690 took 27208 usecs
 pci 0000:00:16.0: quirk_usb_early_handoff+0x0/0x690 took 27320 usecs
 pci 0000:07:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
 pci 0000:07:00.1: D0 power state depends on 0000:07:00.0
 pci 0000:06:00.0: async suspend disabled to avoid multi-function power-on ordering issue
 pci 0000:06:00.1: async suspend disabled to avoid multi-function power-on ordering issue
 pcieport 0000:00:02.0: PME: Signaling with IRQ 26
 pcieport 0000:00:04.0: PME: Signaling with IRQ 27
 pcieport 0000:00:05.0: PME: Signaling with IRQ 28
 pcieport 0000:00:06.0: PME: Signaling with IRQ 29
 pcieport 0000:00:07.0: PME: Signaling with IRQ 30
 pcieport 0000:00:0a.0: PME: Signaling with IRQ 32
  pata_jmicron 0000:06:00.1: enabling device (0000 -> 0001)
  ehci-pci 0000:00:12.2: EHCI Host Controller
  ehci-pci 0000:00:12.2: new USB bus registered, assigned bus number 1
  ehci-pci 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
  ehci-pci 0000:00:12.2: debug port 1
  ehci-pci 0000:00:12.2: irq 17, io mem 0xfcafe400
  firewire_ohci 0000:05:00.0: added OHCI v1.10 device as card 0, 4 IR + 8 IT contexts, quirks 0x11
  ahci 0000:00:11.0: version 3.0
  ahci 0000:00:11.0: AHCI 0001.0200 32 slots 4 ports 6 Gbps 0xf impl SATA mode
  ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part  
  ehci-pci 0000:00:12.2: USB 2.0 started, EHCI 1.00
  xhci_hcd 0000:03:00.0: xHCI Host Controller
  xhci_hcd 0000:03:00.0: new USB bus registered, assigned bus number 2
  xhci_hcd 0000:03:00.0: hcc params 0x014042cb hci version 0x96 quirks 0x0000000000000004
  ahci 0000:06:00.0: JMB361 has only one port
  ahci 0000:06:00.0: forcing port_map 0x3 -> 0x1
  xhci_hcd 0000:03:00.0: xHCI Host Controller
  xhci_hcd 0000:03:00.0: new USB bus registered, assigned bus number 3
  xhci_hcd 0000:03:00.0: Host supports USB 3.0 SuperSpeed
  ehci-pci 0000:00:13.2: EHCI Host Controller
  ehci-pci 0000:00:13.2: new USB bus registered, assigned bus number 4
  ehci-pci 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
  ehci-pci 0000:00:13.2: debug port 1
  ehci-pci 0000:00:13.2: irq 17, io mem 0xfcafe800
  ahci 0000:06:00.0: AHCI 0001.0000 32 slots 2 ports 3 Gbps 0x1 impl SATA mode
  ahci 0000:06:00.0: flags: 64bit ncq pm led clo pmp pio slum part  
  ehci-pci 0000:00:13.2: USB 2.0 started, EHCI 1.00
  ehci-pci 0000:00:16.2: EHCI Host Controller
  ehci-pci 0000:00:16.2: new USB bus registered, assigned bus number 5
  ehci-pci 0000:00:16.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
  ehci-pci 0000:00:16.2: debug port 1
  ehci-pci 0000:00:16.2: irq 17, io mem 0xfcafec00
  ehci-pci 0000:00:16.2: USB 2.0 started, EHCI 1.00
  ohci-pci 0000:00:12.0: OHCI PCI host controller
  ohci-pci 0000:00:12.0: new USB bus registered, assigned bus number 6
  ohci-pci 0000:00:12.0: irq 18, io mem 0xfcaf7000
  ohci-pci 0000:00:13.0: OHCI PCI host controller
  ohci-pci 0000:00:13.0: new USB bus registered, assigned bus number 7
  ohci-pci 0000:00:13.0: irq 18, io mem 0xfcafc000
  ohci-pci 0000:00:14.5: OHCI PCI host controller
  ohci-pci 0000:00:14.5: new USB bus registered, assigned bus number 8
  ohci-pci 0000:00:14.5: irq 18, io mem 0xfcafd000
  ohci-pci 0000:00:16.0: OHCI PCI host controller
  ohci-pci 0000:00:16.0: new USB bus registered, assigned bus number 9
  ohci-pci 0000:00:16.0: irq 18, io mem 0xfcaff000
  r8169 0000:04:00.0 eth0: RTL8168e/8111e, 00:e0:12:34:56:78, XID 2c1, IRQ 52
  r8169 0000:04:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
  snd_hda_intel 0000:07:00.1: Disabling MSI
  snd_hda_intel 0000:07:00.1: Handle vga_switcheroo audio client
  nvidia 0000:07:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
  r8169 0000:04:00.0 eth0: Link is Down
  r8169 0000:04:00.0 eth0: Link is Up - 100Mbps/Full - flow control off


Te info of the NVMe device, from lshw is


*-pci:5 
             description: PCI bridge 
             product: RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 5) 
             vendor: Advanced Micro Devices, Inc. [AMD/ATI] 
             physical id: a 
             bus info: pci@0000:00:0a.0 
             version: 00 
             width: 32 bits 
             clock: 33MHz 
             capabilities: pci pm pciexpress msi ht normal_decode bus_master cap_list 
             configuration: driver=pcieport 
             resources: irq:32 memory:fcb00000-fcbfffff 
           *-nvme 
                description: NVMe device 
                product: CT1000P2SSD8 
                vendor: Micron/Crucial Technology 
                physical id: 0 
                bus info: pci@0000:02:00.0 
                logical name: /dev/nvme0 
                version: P2CR031 
                serial: 2051E4DF0793 
                width: 64 bits 
                clock: 33MHz 
                capabilities: nvme pciexpress msix msi pm nvm_express bus_master cap_list 
                configuration: driver=nvme latency=0 nqn=nqn.2014.08.org.nvmexpress:c0a9c0a92051E4DF0793        CT1000P2SSD8 state=live 
                resources: irq:35 memory:fcbfc000-fcbfffff 
              *-namespace:0 
                   description: NVMe disk 
                   physical id: 0 
                   logical name: hwmon0 
              *-namespace:1 
                   description: NVMe disk 
                   physical id: 1 
                   bus info: nvme@0:1 
                   logical name: /dev/nvme0n1 
                   size: 931GiB (1TB) 
                   capabilities: gpt-1.00 partitioned partitioned:gpt 
                   configuration: guid=bba2b0ce-25a8-4cd1-8048-a9feea9ea341 logicalsectorsize=512 sectorsize=512 wwid=eui.6479a7fff0000000 
                 *-volume 
                      description: EFI partition 
                      physical id: 1 
                      bus info: nvme@0:1,1 
                      logical name: /dev/nvme0n1p1 
                      logical name: / 
                      logical name: /.snapshots 
                      logical name: /home 
                      logical name: /opt 
                      logical name: /root 
                      logical name: /srv 
                      logical name: /tmp 
                      logical name: /usr/local 
                      logical name: /var 
                      serial: 66c4394f-dbf2-4e45-b1be-49b8ca6c7154 
                      capacity: 931GiB 
                      configuration: mount.fstype=btrfs mount.options=rw,relatime,ssd,space_cache,subvolid=257,subvol=/@/var state=mounted


7983            == 0x000000001F2F
140737488363311 == 0x**8**00000001F2F

This is one bit flip. Content of memory was changed, one bit went from 0 to 1 and was written to disk. It should be possible to fix without data loss, but you should ask on btrfs mailing list where there are developers who actually understand what they are talking about. Post btrfs check output, you may mention that bitflip was suggested and ask if btrfs check can repair this. If not, it is quite likely that one of developers creates custom patch to address this specific problem.

https://btrfs.wiki.kernel.org/index.php/Btrfs_mailing_list

And it makes sense to test your RAM. Leave it for extended period, overnight. This is indication that something may be failing.

I am not sure. I experienced some issues: https://forums.opensuse.org/showthread.php/555649-Extra-Fun-With-Backup-To-External-Disk?p=3051625#post3051625 The pertaining journal is:

erlangen:~ # journalctl -o short-monotonic _KERNEL_SUBSYSTEM=scsi -g sdb --since 2021-07-19 --no-pager  
-- Journal begins at Sat 2021-07-17 20:44:14 CEST, ends at Thu 2021-07-22 13:05:26 CEST. -- 
 2013.862995] erlangen kernel: sd 3:0:0:0: [sdb] Synchronizing SCSI cache 
 2013.863140] erlangen kernel: sd 3:0:0:0: [sdb] Stopping disk 
 2014.599008] erlangen kernel: sd 3:0:0:0: [sdb] Starting disk 
[12386.693892] erlangen kernel: sd 3:0:0:0: [sdb] Synchronizing SCSI cache 
[12386.694028] erlangen kernel: sd 3:0:0:0: [sdb] Stopping disk 
[12387.726669] erlangen kernel: sd 3:0:0:0: [sdb] Starting disk 
-- Boot f67105b07f28450cab51ccea07fa56ee -- 
    0.759344] erlangen kernel: sd 3:0:0:0: [sdb] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB) 
    0.759353] erlangen kernel: sd 3:0:0:0: [sdb] 4096-byte physical blocks 
    0.759369] erlangen kernel: sd 3:0:0:0: [sdb] Write Protect is off 
    0.759374] erlangen kernel: sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00 
    0.759400] erlangen kernel: sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA 
    0.868054] erlangen kernel: sd 3:0:0:0: [sdb] Attached SCSI disk 
-- Boot 40744f5dac8d474a84dc6a699a882a5a -- 
    0.758953] erlangen kernel: sd 3:0:0:0: [sdb] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB) 
    0.758961] erlangen kernel: sd 3:0:0:0: [sdb] 4096-byte physical blocks 
    0.758977] erlangen kernel: sd 3:0:0:0: [sdb] Write Protect is off 
    0.758982] erlangen kernel: sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00 
    0.759006] erlangen kernel: sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA 
    0.859723] erlangen kernel: sd 3:0:0:0: [sdb] Attached SCSI disk 
[33817.360693] erlangen kernel: sd 3:0:0:0: [sdb] Synchronizing SCSI cache 
[33817.360830] erlangen kernel: sd 3:0:0:0: [sdb] Stopping disk 
[33818.281445] erlangen kernel: sd 3:0:0:0: [sdb] Starting disk 
[61742.697032] erlangen kernel: sd 3:0:0:0: [sdb] tag#26 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s 
[61742.697040] erlangen kernel: sd 3:0:0:0: [sdb] tag#26 Sense Key : Medium Error [current]  

........          #shortened to fit forum limit of 15000 characters.

[62772.213626] erlangen kernel: sd 3:0:0:0: [sdb] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s 
[62772.213635] erlangen kernel: sd 3:0:0:0: [sdb] tag#2 Sense Key : Medium Error [current]  
[62772.213640] erlangen kernel: sd 3:0:0:0: [sdb] tag#2 Add. Sense: Unrecovered read error - auto reallocate failed 
[62772.213646] erlangen kernel: sd 3:0:0:0: [sdb] tag#2 CDB: Read(16) 88 00 00 00 00 00 00 b1 ca d8 00 00 00 08 00 00 
[62772.381694] erlangen kernel: sd 3:0:0:0: [sdb] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s 
[62772.381702] erlangen kernel: sd 3:0:0:0: [sdb] tag#16 Sense Key : Medium Error [current]  
[62772.381708] erlangen kernel: sd 3:0:0:0: [sdb] tag#16 Add. Sense: Unrecovered read error - auto reallocate failed 
[62772.381714] erlangen kernel: sd 3:0:0:0: [sdb] tag#16 CDB: Read(16) 88 00 00 00 00 00 00 b1 ca d8 00 00 00 08 00 00 
[62992.002389] erlangen kernel: sd 3:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s 
[62992.002397] erlangen kernel: sd 3:0:0:0: [sdb] tag#0 Sense Key : Medium Error [current]  
[62992.002403] erlangen kernel: sd 3:0:0:0: [sdb] tag#0 Add. Sense: Unrecovered read error - auto reallocate failed 
[62992.002408] erlangen kernel: sd 3:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 b1 c8 00 00 00 04 00 00 00 
[62993.542295] erlangen kernel: sd 3:0:0:0: [sdb] tag#8 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s 
[62993.542303] erlangen kernel: sd 3:0:0:0: [sdb] tag#8 Sense Key : Medium Error [current]  
[62993.542308] erlangen kernel: sd 3:0:0:0: [sdb] tag#8 Add. Sense: Unrecovered read error - auto reallocate failed 
[62993.542314] erlangen kernel: sd 3:0:0:0: [sdb] tag#8 CDB: Read(16) 88 00 00 00 00 00 00 b1 ca d8 00 00 00 08 00 00 
[62993.734443] erlangen kernel: sd 3:0:0:0: [sdb] tag#6 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s 
[62993.734451] erlangen kernel: sd 3:0:0:0: [sdb] tag#6 Sense Key : Medium Error [current]  
[62993.734456] erlangen kernel: sd 3:0:0:0: [sdb] tag#6 Add. Sense: Unrecovered read error - auto reallocate failed 
[62993.734462] erlangen kernel: sd 3:0:0:0: [sdb] tag#6 CDB: Read(16) 88 00 00 00 00 00 00 b1 ca d8 00 00 00 08 00 00 
-- Boot 4ae4e327192b463880fc404de6424088 -- 
    0.762570] erlangen kernel: sd 3:0:0:0: [sdb] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB) 
    0.762579] erlangen kernel: sd 3:0:0:0: [sdb] 4096-byte physical blocks 
    0.762595] erlangen kernel: sd 3:0:0:0: [sdb] Write Protect is off 
    0.762599] erlangen kernel: sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00 
    0.762625] erlangen kernel: sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA 
    0.899394] erlangen kernel: sd 3:0:0:0: [sdb] Attached SCSI disk 
   82.587665] erlangen kernel: sd 3:0:0:0: [sdb] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=2s 
   82.587666] erlangen kernel: sd 3:0:0:0: [sdb] tag#2 Sense Key : Medium Error [current]  
   82.587668] erlangen kernel: sd 3:0:0:0: [sdb] tag#2 Add. Sense: Unrecovered read error - auto reallocate failed 
   82.587669] erlangen kernel: sd 3:0:0:0: [sdb] tag#2 CDB: Read(16) 88 00 00 00 00 00 00 b1 c8 00 00 00 04 00 00 00 
   83.862378] erlangen kernel: sd 3:0:0:0: [sdb] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s 
   83.862386] erlangen kernel: sd 3:0:0:0: [sdb] tag#12 Sense Key : Medium Error [current]  
   83.862392] erlangen kernel: sd 3:0:0:0: [sdb] tag#12 Add. Sense: Unrecovered read error - auto reallocate failed 
   83.862398] erlangen kernel: sd 3:0:0:0: [sdb] tag#12 CDB: Read(16) 88 00 00 00 00 00 00 b1 ca d8 00 00 00 08 00 00 
   84.031682] erlangen kernel: sd 3:0:0:0: [sdb] tag#20 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s 
   84.031691] erlangen kernel: sd 3:0:0:0: [sdb] tag#20 Sense Key : Medium Error [current]  
   84.031696] erlangen kernel: sd 3:0:0:0: [sdb] tag#20 Add. Sense: Unrecovered read error - auto reallocate failed 
   84.031702] erlangen kernel: sd 3:0:0:0: [sdb] tag#20 CDB: Read(16) 88 00 00 00 00 00 00 b1 ca d8 00 00 00 08 00 00 
 1629.818909] erlangen kernel: sd 3:0:0:0: [sdb] tag#28 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s 
 1629.818917] erlangen kernel: sd 3:0:0:0: [sdb] tag#28 Sense Key : Medium Error [current]  
 1629.818922] erlangen kernel: sd 3:0:0:0: [sdb] tag#28 Add. Sense: Unrecovered read error - auto reallocate failed 
 1629.818928] erlangen kernel: sd 3:0:0:0: [sdb] tag#28 CDB: Read(16) 88 00 00 00 00 00 00 b1 d4 00 00 00 0a 00 00 00 
 1633.479151] erlangen kernel: sd 3:0:0:0: [sdb] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s 
 1633.479159] erlangen kernel: sd 3:0:0:0: [sdb] tag#1 Sense Key : Medium Error [current]  
 1633.479165] erlangen kernel: sd 3:0:0:0: [sdb] tag#1 Add. Sense: Unrecovered read error - auto reallocate failed 
 1633.479171] erlangen kernel: sd 3:0:0:0: [sdb] tag#1 CDB: Read(16) 88 00 00 00 00 00 00 b1 de 00 00 00 0a 00 00 00 
 1636.989827] erlangen kernel: sd 3:0:0:0: [sdb] tag#21 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s 
 1636.989836] erlangen kernel: sd 3:0:0:0: [sdb] tag#21 Sense Key : Medium Error [current]  
 1636.989841] erlangen kernel: sd 3:0:0:0: [sdb] tag#21 Add. Sense: Unrecovered read error - auto reallocate failed 
 1636.989847] erlangen kernel: sd 3:0:0:0: [sdb] tag#21 CDB: Read(16) 88 00 00 00 00 00 00 b1 d7 88 00 00 00 08 00 00 
 1647.570525] erlangen kernel: sd 3:0:0:0: [sdb] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=4s 
 1647.570533] erlangen kernel: sd 3:0:0:0: [sdb] tag#9 Sense Key : Medium Error [current]  
 1647.570539] erlangen kernel: sd 3:0:0:0: [sdb] tag#9 Add. Sense: Unrecovered read error - auto reallocate failed 
 1647.570544] erlangen kernel: sd 3:0:0:0: [sdb] tag#9 CDB: Read(16) 88 00 00 00 00 00 00 b2 0b 88 00 00 0a 00 00 00 
-- Boot 716ba3d80d4341e093dd10a2eeed54f7 -- 
    0.759082] erlangen kernel: sd 3:0:0:0: [sdb] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB) 
    0.759090] erlangen kernel: sd 3:0:0:0: [sdb] 4096-byte physical blocks 
    0.759107] erlangen kernel: sd 3:0:0:0: [sdb] Write Protect is off 
    0.759111] erlangen kernel: sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00 
    0.759136] erlangen kernel: sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA 
    0.887829] erlangen kernel: sd 3:0:0:0: [sdb] Attached SCSI disk 
[17335.268390] erlangen kernel: sd 3:0:0:0: [sdb] Synchronizing SCSI cache 
[17335.305144] erlangen kernel: sd 3:0:0:0: [sdb] Stopping disk 
[17336.836309] erlangen kernel: sd 3:0:0:0: [sdb] Starting disk 
-- Boot 300d9904a90d4b77b4eb6bc9c6f812f4 -- 
    0.751009] erlangen kernel: sd 3:0:0:0: [sdb] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB) 
    0.751018] erlangen kernel: sd 3:0:0:0: [sdb] 4096-byte physical blocks 
    0.751034] erlangen kernel: sd 3:0:0:0: [sdb] Write Protect is off 
    0.751039] erlangen kernel: sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00 
    0.751064] erlangen kernel: sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA 
    0.863800] erlangen kernel: sd 3:0:0:0: [sdb] Attached SCSI disk 
erlangen:~ #

The above shows the disk to be clean for some time. Suddenly errors occurred: They got fixed and the disk is now great again.

NVMe may be queried as:

**erlangen:~ #** journalctl -o short-monotonic _KERNEL_SUBSYSTEM=nvme --since 2021-07-19 
-- Journal begins at Sat 2021-07-17 20:44:14 CEST, ends at Thu 2021-07-22 13:16:13 CEST. -- 
 2014.611144] erlangen kernel: nvme nvme0: 8/0/0 default/read/poll queues 
[12387.739559] erlangen kernel: nvme nvme0: 8/0/0 default/read/poll queues 
**-- Boot f67105b07f28450cab51ccea07fa56ee --**
    2.063071] erlangen kernel: nvme nvme0: pci function 0000:04:00.0 
    2.082843] erlangen kernel: nvme nvme0: 8/0/0 default/read/poll queues 
**-- Boot 40744f5dac8d474a84dc6a699a882a5a --**
    2.058663] erlangen kernel: nvme nvme0: pci function 0000:04:00.0 
    2.074967] erlangen kernel: nvme nvme0: 8/0/0 default/read/poll queues 
[33818.294257] erlangen kernel: nvme nvme0: 8/0/0 default/read/poll queues 
**-- Boot 4ae4e327192b463880fc404de6424088 --**
    2.076510] erlangen kernel: nvme nvme0: pci function 0000:04:00.0 
    2.089620] erlangen kernel: nvme nvme0: 8/0/0 default/read/poll queues 
**-- Boot 716ba3d80d4341e093dd10a2eeed54f7 --**
    2.104451] erlangen kernel: nvme nvme0: pci function 0000:04:00.0 
    2.116239] erlangen kernel: nvme nvme0: 8/0/0 default/read/poll queues 
[17336.856350] erlangen kernel: nvme nvme0: 8/0/0 default/read/poll queues 
**-- Boot 300d9904a90d4b77b4eb6bc9c6f812f4 --**
    2.065883] erlangen kernel: nvme nvme0: pci function 0000:04:00.0 
    2.078140] erlangen kernel: nvme nvme0: 8/0/0 default/read/poll queues 
**erlangen:~ #**



A bit flip may cause no messages in the journal. However consistency checks such as btrfs scrub will detect them on used blocks. ‘dd if=/dev/sdX of=/dev/null bs=4M’ will try to read every block of the device regardless whether it is used or unused by the file system and stop on errors encountered.

No, scrub won’t detect them. And “scrub” is not consistency check at all.

Yes, btrfs consistency checks detect inconsistent data and so refuse to mount this filesystem (at least, modify it). How exactly is this revelation useful?

There are data and metadata. From: https://github.com/kdave/btrfs-progs/blob/devel/Documentation/btrfs-scrub.asciidoc

Scrub is not a filesystem checker (fsck) and does not verify nor repair structural damage in the filesystem. It really only checks checksums of data and tree blocks, it doesn’t ensure the content of tree blocks is valid and consistent. There’s some validation performed when metadata blocks are read from disk but it’s not extensive and cannot substitute full btrfs check run.

The above mentioned HDD partition passed the fsck despite some sectors being unreadable. A full truly full check requires both scrub and fsck.

I forgot to mention it. I had issues with RAM memory last week. I plug some memory modules and I had some program crashes, then I tested them and found one of them was faulty. I removed it and passed a RAM test several hours with no errors, since then I have had no more program crashes.

I have tested the NVME booting with the rescue system and running

#badblocks -svn /dev/nvme0n1

I took 6 hours and reported 0 errors.

**pruebas:~ #** journalctl -o short-monotonic _KERNEL_SUBSYSTEM=nvme 
-- Logs begin at Thu 2021-07-22 19:04:50 CEST, end at Thu 2021-07-22 19:13:59 CEST. -- 
    4.765249] pruebas kernel: nvme nvme0: pci function 0000:02:00.0 
    4.773382] pruebas kernel: **nvme nvme0: missing or invalid SUBNQN field.**
    5.105673] pruebas kernel: nvme nvme0: allocated 128 MiB host memory buffer. 
    5.262326] pruebas kernel: nvme nvme0: 8/0/0 default/read/poll queues 
**pruebas:~ #**

This confirms that the issue was caused by bad RAM.

#badblocks -svn /dev/nvme0n1

You misunderstand. Your storage device does not have any problems. The data in memory was corrupted and then written to disk. From the point of view of btrfs checksum was computed against bad data so checksum verification won’t show any issues. What now fails is logical consistency check. The content of written data does not match the rest of filesystem. You have two copies of the same data (metadata to please our btrfs expert) where you should have only one, and both copies contain some logical error.