btrfss errors and trying to recover

I have a 15,3 with snapshots, typical installation.
One day booting systems crash. On reboot there are errors in the filesystem like this and I get no graphical login.

Feb 09 10:55:35 Jarvis kernel: **BTRFS error (device sda2): parent transid verify failed on 860618752 wanted 61777 found 60884**
Feb 09 10:55:35 Jarvis kernel: **BTRFS error (device sda2): parent transid verify failed on 860618752 wanted 61777 found 60884**
Feb 09 10:55:35 Jarvis kernel: **BTRFS error (device sda2): parent transid verify failed on 860618752 wanted 61777 found 60884**

It seems to me the HD may be dying (machanical), so I did a dd_rescue of the whole disk. The dd_rescue finished with no errors, then I dd the image to a new (SD) disk, and all I’m trying is from this second disk, I have the dd image, so I can risk.
First I boot with the new disk and it boots the same that with the old one. I try to change to a old snapshot, it seems to boot ok, the I do a snapper rollback, when I boot again it fails again.

I think I should not have done the snapper rollback but repair the failsystem before, but I’m not really sure if I did notice there was errors in the btrfs fs before of after the dd copy and the rollback.

I try to scrub the system, but it fails as it is mounted ro

Jarvis:~ # btrfs scrub start /
WARNING: failed to open the progress status socket at /var/lib/btrfs/scrub.progress.c16ea39a-1427-444a-9bf3-d90f44d223ca: Read-only file system. Progress cannot be queried
WARNING: failed to write the progress status file: Read-only file system. Status recording disabled
scrub started on /, fsid c16ea39a-1427-444a-9bf3-d90f44d223ca (pid=5519)

So I try to remount the fs rw

Jarvis:~ # cat /etc/fstab
UUID=c16ea39a-1427-444a-9bf3-d90f44d223ca  /                       btrfs  defaults                      0  0
UUID=c16ea39a-1427-444a-9bf3-d90f44d223ca  /var                    btrfs  subvol=/@/var                 0  0
UUID=c16ea39a-1427-444a-9bf3-d90f44d223ca  /usr/local              btrfs  subvol=/@/usr/local           0  0
UUID=c16ea39a-1427-444a-9bf3-d90f44d223ca  /tmp                    btrfs  subvol=/@/tmp                 0  0
UUID=c16ea39a-1427-444a-9bf3-d90f44d223ca  /srv                    btrfs  subvol=/@/srv                 0  0
UUID=c16ea39a-1427-444a-9bf3-d90f44d223ca  /root                   btrfs  subvol=/@/root                0  0
UUID=c16ea39a-1427-444a-9bf3-d90f44d223ca  /opt                    btrfs  subvol=/@/opt                 0  0
UUID=c16ea39a-1427-444a-9bf3-d90f44d223ca  /home                   btrfs  subvol=/@/home                0  0
UUID=c16ea39a-1427-444a-9bf3-d90f44d223ca  /boot/grub2/x86_64-efi  btrfs  subvol=/@/boot/grub2/x86_64-efi  0  0
UUID=c16ea39a-1427-444a-9bf3-d90f44d223ca  /boot/grub2/i386-pc     btrfs  subvol=/@/boot/grub2/i386-pc  0  0
UUID=c16ea39a-1427-444a-9bf3-d90f44d223ca  /.snapshots             btrfs  subvol=/@/.snapshots          0  0
UUID=65470549-4f94-42f9-967e-7b029a37b7b0  swap                    swap   defaults                      0  0
Jarvis:~ # mount -o remount,rw /
mount: /: mount point not mounted or bad option.
Jarvis:~ #

I don’t know what I’m doing wrong.

The fs is mounting ro because of the errors during boot

 09 10:55:36 Jarvis kernel: BTRFS warning (device sda2): error accounting new delayed refs extent (err code: -5), quota inconsistent
Feb 09 10:55:36 Jarvis kernel: BTRFS warning (device sda2): error accounting new delayed refs extent (err code: -5), quota inconsistent
Feb 09 10:55:36 Jarvis kernel: BTRFS warning (device sda2): error accounting new delayed refs extent (err code: -5), quota inconsistent
Feb 09 10:55:36 Jarvis kernel: BTRFS warning (device sda2): error accounting new delayed refs extent (err code: -5), quota inconsistent
Feb 09 10:55:36 Jarvis kernel: BTRFS warning (device sda2): error accounting new delayed refs extent (err code: -5), quota inconsistent
Feb 09 10:55:36 Jarvis kernel: ------------ cut here ]------------
Feb 09 10:55:36 Jarvis kernel: WARNING: CPU: 3 PID: 1099 at ../fs/btrfs/qgroup.c:2510 btrfs_qgroup_account_extents+0x1a9/0x260 [btrfs]
Feb 09 10:55:36 Jarvis kernel: Modules linked in: dmi_sysfs hwmon_vid nvidia_drm(POEX) nvidia_modeset(POEX) nvidia_uvm(POEX) snd_hda_codec_hdmi nvidia(POEX) snd_hda_intel snd_int>
Feb 09 10:55:36 Jarvis kernel: Supported: No, Proprietary and Unsupported modules are loaded
Feb 09 10:55:36 Jarvis kernel: CPU: 3 PID: 1099 Comm: snapd Tainted: P           OE  X  N 5.3.18-59.34-default #1 SLE15-SP3
Feb 09 10:55:36 Jarvis kernel: Hardware name: System manufacturer System Product Name/M4A77TD, BIOS 2104    06/28/2010
Feb 09 10:55:36 Jarvis kernel: RIP: 0010:btrfs_qgroup_account_extents+0x1a9/0x260 [btrfs]
Feb 09 10:55:36 Jarvis kernel: Code: 7e 08 49 83 c6 18 4c 89 fa 48 89 ee e8 e0 e0 3d c6 49 8b 0e 48 85 c9 75 e5 e9 c2 fe ff ff 48 c7 c7 40 1f 46 c0 e8 e5 97 8e c5 <0f> 0b 49 8b 5>
Feb 09 10:55:36 Jarvis kernel: RSP: 0018:ffffbc1780d03da0 EFLAGS: 00010286
Feb 09 10:55:36 Jarvis kernel: RAX: 0000000000000024 RBX: ffff9b0409315550 RCX: 0000000000000000
Feb 09 10:55:36 Jarvis kernel: RDX: 0000000000000000 RSI: ffff9b04a7ad9558 RDI: ffff9b04a7ad9558
Feb 09 10:55:36 Jarvis kernel: RBP: ffff9b040a147000 R08: 0000000000000446 R09: 0000000000000001
Feb 09 10:55:36 Jarvis kernel: R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
Feb 09 10:55:36 Jarvis kernel: R13: 00000000000000f0 R14: 0000000000000000 R15: ffff9b048b64ebc0
Feb 09 10:55:36 Jarvis kernel: FS:  00007f8892b6d700(0000) GS:ffff9b04a7ac0000(0000) knlGS:0000000000000000
Feb 09 10:55:36 Jarvis kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 09 10:55:36 Jarvis kernel: CR2: 00005557d3a3d1d8 CR3: 000000020be50000 CR4: 00000000000006e0
Feb 09 10:55:36 Jarvis kernel: Call Trace:
Feb 09 10:55:36 Jarvis kernel:  btrfs_commit_transaction+0x52e/0xa60 [btrfs]
Feb 09 10:55:36 Jarvis kernel:  ? dput+0xcb/0x300
Feb 09 10:55:36 Jarvis kernel:  ? btrfs_log_dentry_safe+0x54/0x70 [btrfs]
Feb 09 10:55:36 Jarvis kernel:  btrfs_sync_file+0x393/0x430 [btrfs]
Feb 09 10:55:36 Jarvis kernel:  do_fsync+0x38/0x60
Feb 09 10:55:36 Jarvis kernel:  __x64_sys_fsync+0x10/0x20
Feb 09 10:55:36 Jarvis kernel:  do_syscall_64+0x5b/0x1e0
Feb 09 10:55:36 Jarvis kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Feb 09 10:55:36 Jarvis kernel: RIP: 0033:0x55ddade24a4b
Feb 09 10:55:36 Jarvis kernel: Code: a3 fa ff e9 67 ff ff ff cc cc cc cc cc cc cc e8 9b 28 f8 ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 f>
Feb 09 10:55:36 Jarvis kernel: RSP: 002b:000000c00059f7e8 EFLAGS: 00000212 ORIG_RAX: 000000000000004a
Feb 09 10:55:36 Jarvis kernel: RAX: ffffffffffffffda RBX: 000000c000047800 RCX: 000055ddade24a4b
Feb 09 10:55:36 Jarvis kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000a
Feb 09 10:55:36 Jarvis kernel: RBP: 000000c00059f828 R08: 0000000000000001 R09: 0000000000000000
Feb 09 10:55:36 Jarvis kernel: R10: 000000c00003c7a0 R11: 0000000000000212 R12: ffffffffffffffff
Feb 09 10:55:36 Jarvis kernel: R13: 000000000000003e R14: 000000000000003d R15: 0000000000000100
Feb 09 10:55:36 Jarvis kernel: --- end trace 4b12f8de8a3334c1 ]---
Feb 09 10:55:36 Jarvis cupsd[998]: Unable to open log file "/var/log/cups/error_log" - Read-only file system
Feb 09 10:55:36 Jarvis kernel: BTRFS warning (device sda2): Skipping commit of aborted transaction.
Feb 09 10:55:36 Jarvis kernel: BTRFS: error (device sda2) in cleanup_transaction:1818: errno=-5 IO failure
Feb 09 10:55:36 Jarvis kernel: BTRFS info (device sda2): forced readonly
Feb 09 10:55:36 Jarvis kernel: BTRFS info (device sda2): delayed_refs has NO entry
Feb 09 10:55:36 Jarvis systemd[1]: Starting Modem Manager...
Feb 09 10:55:36 Jarvis cupsd[998]: Unable to open log file "/var/log/cups/error_log" - Read-only file system
Feb 09 10:55:36 Jarvis systemd[1]: Starting firewalld - dynamic firewall daemon...
Feb 09 10:55:36 Jarvis cupsd[998]: Unable to open log file "/var/log/cups/error_log" - Read-only file system

any help, please?

I have mounted the fs from a recovery disk

**[root@sysrescue ~]#** fdisk -l 
**Disk /dev/sda: 447.13 GiB, 480103981056 bytes, 937703088 sectors** 
Disk model: KINGSTON SA400S3 
Units: sectors of 1 * 512 = 512 bytes 
Sector size (logical/physical): 512 bytes / 512 bytes 
I/O size (minimum/optimal): 512 bytes / 512 bytes 
Disklabel type: gpt 
Disk identifier: 26DAFF91-68E5-4A18-8E3F-0E45EA01D2A6 

**Device    ** **    Start** **      End** **  Sectors** **  Size** **Type** 
/dev/sda1       2048     18431     16384     8M BIOS boot 
/dev/sda2      18432 765136895 765118464 364.8G Linux filesystem 
/dev/sda3  904914944 937701375  32786432  15.6G Linux swap



mounted it to a directory and try to scrub it. It didn’t finished and reported 122 unrecoverable errors.

**[root@sysrescue ~]#** btrfs scrub status /nacho
[FONT=monospace]**[root@sysrescue ~]#** btrfs scrub status /nacho 
UUID:             c16ea39a-1427-444a-9bf3-d90f44d223ca 
Scrub started:    Sat Feb 12 12:25:45 2022 
Status:           aborted 
Duration:         0:01:30 
Total to scrub:   45.51GiB 
Rate:             253.31MiB/s 
Error summary:    verify=122 
  Corrected:      0 
  Uncorrectable:  122 
  Unverified:     0 
**[root@sysrescue ~]#** 


[/FONT]

The following step should be to do a --repair?

I did a btrfs check …lots of errors


#btrfs check /dev/sda2

https://susepaste.org/67153289

and a restore …errors too

**[root@sysrescue ~]#** btrfs restore /dev/sda2 /usb 
parent transid verify failed on 861143040 wanted 61777 found 60882 
parent transid verify failed on 861143040 wanted 61777 found 60882 
parent transid verify failed on 861143040 wanted 61777 found 60882 
Ignoring transid failure 
leaf parent key incorrect 861143040 
parent transid verify failed on 861143040 wanted 61777 found 60882 
Ignoring transid failure 
leaf parent key incorrect 861143040 
ERROR: searching extent data returned -1 
Error copying data for /usb/@/home/nacho/.cache/mesa_shader_cache/21/ec5a6b20786ca80f427cabec22dc0bc1593427 
Error searching /usb/@/home/nacho/.cache/mesa_shader_cache/21/ec5a6b20786ca80f427cabec22dc0bc1593427 
Error searching /usb/@/home/nacho/.cache/mesa_shader_cache/21/ec5a6b20786ca80f427cabec22dc0bc1593427 
Error searching /usb/@/home/nacho/.cache/mesa_shader_cache/21/ec5a6b20786ca80f427cabec22dc0bc1593427 
Error searching /usb/@/home/nacho/.cache/mesa_shader_cache/21/ec5a6b20786ca80f427cabec22dc0bc1593427 
Error searching /usb/@/home/nacho/.cache/mesa_shader_cache/21/ec5a6b20786ca80f427cabec22dc0bc1593427 
Error searching /usb/@/home/nacho/.cache/mesa_shader_cache/21/ec5a6b20786ca80f427cabec22dc0bc1593427 
**[root@sysrescue ~]#**