BTRFS error: parent transid verify failed. MOK related?

Small backup server running syncthing, KDE desktop and one 1TB HDD, no SSD, secureboot enabled.

Today on boot it came up with the key enroll dialog, but it timed out before I could connect a keyboard.

dmesg and journalctl list btrf errors, which I hadn’t seen before:

# dmesg| grep error
   97.555130] BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 found 12750
   97.561897] BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 found 12750
   97.565832] BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 found 12750
   97.573780] BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 found 12750
   97.577778] BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 found 12750
   97.578669] BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 found 12750
   97.579479] BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 found 12750
   97.581830] BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 found 12750
   97.585840] BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 found 12750
   97.589841] BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 found 12750
#
# journalctl -b --grep btrfs
-- Logs begin at Tue 2022-02-01 19:07:02 -03, end at Tue 2022-02-01 19:20:10 -03. --
fev 01 19:07:02 PCeM dracut-cmdline[165]: Using kernel command line parameters: root=UUID=3a21d6e5-651e-43c1-86cb>
fev 01 19:07:04 PCeM kernel: Btrfs loaded, crc32c=crc32c-intel, assert=on
fev 01 19:07:04 PCeM kernel: BTRFS: device label root devid 1 transid 12804 /dev/sda2
fev 01 19:07:05 PCeM kernel: BTRFS info (device sda2): disk space caching is enabled
fev 01 19:07:05 PCeM kernel: BTRFS info (device sda2): has skinny extents
fev 01 19:07:11 PCeM kernel: BTRFS info (device sda2): disk space caching is enabled
fev 01 19:07:15 PCeM systemd[1]: Started Watch /etc/sysconfig/btrfsmaintenance.
fev 01 19:07:15 PCeM systemd[1]: Started Balance block groups on a btrfs filesystem.
fev 01 19:07:15 PCeM systemd[1]: Started Scrub btrfs filesystem, verify block checksums.
fev 01 19:08:41 PCeM kernel: BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 fou>
fev 01 19:08:41 PCeM kernel: BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 fou>
fev 01 19:08:41 PCeM kernel: BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 fou>
fev 01 19:08:41 PCeM kernel: BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 fou>
fev 01 19:08:41 PCeM kernel: BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 fou>
fev 01 19:08:41 PCeM kernel: BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 fou>
fev 01 19:08:41 PCeM kernel: BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 fou>
fev 01 19:08:41 PCeM kernel: BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 fou>
fev 01 19:08:41 PCeM kernel: BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 fou>
fev 01 19:08:41 PCeM kernel: BTRFS error (device sda2): parent transid verify failed on 56705024 wanted 12751 fou>
fev 01 19:14:25 PCeM kernel: BTRFS error (device sda2): parent transid verify failed on 58392576 wanted 12751 fou>
fev 01 19:14:25 PCeM kernel: BTRFS error (device sda2): parent transid verify failed on 58392576 wanted 12751 fou>
fev 01 19:14:25 PCeM kernel: BTRFS: error (device sda2) in __btrfs_free_extent:3013: errno=-5 IO failure
fev 01 19:14:25 PCeM kernel: BTRFS info (device sda2): forced readonly
fev 01 19:14:25 PCeM kernel: BTRFS: error (device sda2) in btrfs_run_delayed_refs:2129: errno=-5 IO failure
#

In a check of the HDD with a liveCD, gsmart show all atributes normal. I tried to repair the root partiton sda2 using gparted but it froze, no error message, but the desktop was normally resoponsive.

The root partition is small, but there’s almost no software installed besides the standard packages.

# btrfs filesystem usage -T /
Overall:
    Device size:                  16.81GiB
    Device allocated:             10.57GiB
    Device unallocated:            6.24GiB
    Device missing:                  0.00B
    Used:                          6.37GiB
    Free (estimated):              8.24GiB      (min: 5.12GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               17.83MiB      (used: 0.00B)

             Data    Metadata  System              
Id Path      single  DUP       DUP      Unallocated
-- --------- ------- --------- -------- -----------
 1 /dev/sda2 8.01GiB   2.50GiB 64.00MiB     6.24GiB
-- --------- ------- --------- -------- -----------
   Total     8.01GiB   1.25GiB 32.00MiB     6.24GiB
   Used      6.00GiB 186.16MiB 16.00KiB            
#
# btrfs filesystem df /
Data, single: total=8.01GiB, used=6.00GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.25GiB, used=186.14MiB
GlobalReserve, single: total=17.83MiB, used=0.00B
# 

At boot the btfrs fs is mounted rw, but a couple attempts to write to it (I tried zypper), the fs changes to ro.

Berore writing:

:~> mount | grep sda
/dev/sda2 on / type btrfs (rw,relatime,space_cache,subvolid=256,subvol=/@)
/dev/sda2 on /boot/grub2/i386-pc type btrfs (rw,relatime,space_cache,subvolid=265,subvol=/@/boot/grub2/i386-pc)
/dev/sda2 on /opt type btrfs (rw,relatime,space_cache,subvolid=263,subvol=/@/opt)
/dev/sda2 on /boot/grub2/x86_64-efi type btrfs (rw,relatime,space_cache,subvolid=264,subvol=/@/boot/grub2/x86_64-efi)
/dev/sda2 on /root type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/@/root) 
/dev/sda2 on /srv type btrfs (rw,relatime,space_cache,subvolid=261,subvol=/@/srv)
/dev/sda2 on /tmp type btrfs (rw,relatime,space_cache,subvolid=260,subvol=/@/tmp)
/dev/sda2 on /var type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/@/var)
/dev/sda2 on /usr/local type btrfs (rw,relatime,space_cache,subvolid=259,subvol=/@/usr/local)
/dev/sda3 on /home type ext4 (rw,noatime,nodiratime,stripe=32743,data=writeback)
/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro) 
:~>

After:

:~> mount | grep sda
/dev/sda2 on / type btrfs (**ro**,relatime,space_cache,subvolid=256,subvol=/@)
/dev/sda2 on /boot/grub2/i386-pc type btrfs (**ro**,relatime,space_cache,subvolid=265,subvol=/@/boot/grub2/i386-pc) 
/dev/sda2 on /opt type btrfs (**ro**,relatime,space_cache,subvolid=263,subvol=/@/opt)
/dev/sda2 on /boot/grub2/x86_64-efi type btrfs (**ro**,relatime,space_cache,subvolid=264,subvol=/@/boot/grub2/x86_64-efi)
/dev/sda2 on /root type btrfs (**ro**,relatime,space_cache,subvolid=262,subvol=/@/root)
/dev/sda2 on /srv type btrfs (**ro**,relatime,space_cache,subvolid=261,subvol=/@/srv)
/dev/sda2 on /tmp type btrfs (**ro**,relatime,space_cache,subvolid=260,subvol=/@/tmp)
/dev/sda2 on /var type btrfs (**ro**,relatime,space_cache,subvolid=258,subvol=/@/var)
/dev/sda2 on /usr/local type btrfs (**ro**,relatime,space_cache,subvolid=259,subvol=/@/usr/local)
/dev/sda3 on /home type ext4 (rw,noatime,nodiratime,stripe=32743,data=writeback)
/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro) :~>

And scrub fails (since the system is mounted ro?):

PCeM:/home/blimmer # btrfs scrub start -B /
WARNING: failed to open the progress status socket at /var/lib/btrfs/scrub.progress.3a21d6e5-651e-43c1-86cb-83d8c6e15135: Read-only file system. Progress cannot be queried
WARNING: failed to write the progress status file: Read-only file system. Status recording disabled
ERROR: scrubbing / failed for device id 1: ret=-1, errno=30 (Read-only file system)
scrub canceled for 3a21d6e5-651e-43c1-86cb-83d8c6e15135
        scrub started at Tue Feb  1 19:22:10 2022 and was aborted after 00:00:00
        total bytes scrubbed: 0.00B with 0 errors
#

Any suggestions?

Thanks,

Bruno

:~> lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 931,5G  0 disk
|-sda1   8:1    0   199M  0 part /boot/efi
|-sda2   8:2    0  16,8G  0 part /usr/local
|-sda3   8:3    0 910,5G  0 part /home
|-sda4   8:4    0     4G  0 part [SWAP]
sr0     11:0    1  1024M  0 rom
:~>

Any more info I should supply?

Help, please?

To me, it does not look to be MOK related. However, I am not a “btrfs” user, so I don’t have any other comments.

Hi nrickert, thanks for replying.

I don’t think so too, but just to rule that out, do you know how I can re-enroll the key? I’ve done this once with a nvidia key as the proprietary blob didn’t work after an update, IINM. I posted about this some time ago.

But this box is a vanilla Intel Dual-Core Processor J1800 welded to a Asrock micro ATX Motherboard and built-in Intel graphics. I’ve no idea how or what I should try to re-enroll.

P.S.: I looked at openSUSE documentation on mok keys but couldn’t discern how to do this.

The key should be a file in “/etc/uefi/certs”.

You can find which keys are currently enrolled with

mokutil -l

That list shows the fingerprint of the key, and the file name in “/etc/uefi/certs” contains part of the fingerprint (makes it easy to recognize).

To enroll a missing key:

mokutil --root-pw --import /etc/uefi/certs/"keyfilename"

and then reboot to complete the operation.

You can use

man mokutil

or

mokutil --help

for more information.

Thanks, nrickert. As you suspected, it didn’t make any difference on the btrfs issue. I could enroll two keys:

# ls -l /etc/uefi/certs
total 12
-rw-r--r-- 1 root root 1288 jan 24 08:44 4AAA0B54.crt
-rw-r--r-- 1 root root 1257 abr 23  2021 BCA4E38E-shim.crt
-rw-r--r-- 1 root root 1177 mai  3  2021 BDD31A9E-kmp.crt

# mokutil -l
MokListRT is empty

# mokutil --root-pw --import /etc/uefi/certs/BDD31A9E-kmp.crt
# mokutil --root-pw --import /etc/uefi/certs/BCA4E38E-shim.crt
# mokutil --root-pw --import /etc/uefi/certs/4AAA0B54.crt
Already in kernel trusted keyring. Skip /etc/uefi/certs/4AAA0B54.crt
#

After reboot I got the dialog and enrolled the two keys, BUT still:

# mokutil -l
MokListRT is empty
#

Using zypper in errors out downloading a number of repos, BUT not all if the fs is still rw, or none if the fs is ro:

# LANG=C zypper in gsmart
The target filesystem is mounted as read-only. Please make sure the target filesystem is writeable
#

The only difference, regardless if the fs is in rw or ro state, is that scrub reports no errors:

 # btrfs scrub start -B /
#

Besides reinstalling the root system, I’m out of ideas. :frowning:

Using zypper in <anything> errors out downloading a number of repos, BUT not all if the fs is still rw, or none if the fs is ro:

I’m guessing there is a damaged area in the partition, but not all as when the fs is rw, zypper did refresh the first two or three repos.

Anyway, back to square one.

You have a faulty version of “shim”. I ran into that issue, too. The faulty version has a higher version number than the good version, so it won’t automatically update.

Use Yast Software Management, and for an unconditional update of “shim” to fix that.

Again, I don’t think it will change your “btrfs” issues. But you should never see “MokListRT is empty”. It should at least list the key that is built into “shim” itself.

In a good box had to disable “updates from SUSE Linux Enterprise 15” repo to start Yast Software Manager, to see what shim versions are available… I think I hit this bug:

https://bugzilla.suse.com/show_bug.cgi?id=1194626

The version installed on the good box is 15.4-7.19.1 (probably from the disabled update repo), and the available version is 15.4-2.1 from the main repo

Both boxes (good and bad) are using the same version, there is no difference in the output of:


**[FONT=monospace]**#** LANG=C zypper info shim 
Loading repository data... 
Reading installed packages... 


Information for package shim: 
----------------------------- 
Repository     : @System 
Name           : shim 
Version        : 15.4-7.19.1 
Arch           : x86_64 
Vendor         : SUSE LLC <https://www.suse.com/> 
Installed Size : 1.8 MiB 
Installed      : Yes 
Status         : up-to-date 
Source package : shim-15.4-7.19.1.src 
Summary        : UEFI shim loader 
Description    :  
    shim is a trivial EFI application that, when run, attempts to open and 
    execute another application.
**[/FONT]

On a side note, the repo situation has been a PITA since LEAP and SLE repos were mixed. I hope the developers can reach a stable situation, but from what I’ve been reading around I’m not hopeful.

Anyway, I’m tired of this. I think I’ll trundle along as it is, without updates, until LEAP 15.4 or disk failure, whatever happens first.

Thank you for your help.

Yes, that’s the broken version of “shim” that I had until I did an unconditional update of shim (via Yast).