BTRFS file system switched into ro mode during software update

English Install/Boot/Login
#1

Hi,

after the last kernel update the following reproducible error occurs on my computer (Leap 15.2).

The software update shows new packages. When starting the software update the following error message appears in the journal and the root partition (SSD /dev/sda2 with BTRFS) is mounted only RO.

Mär 12 13:33:05 enterprise kernel: ------------ cut here ]------------
Mär 12 13:33:05 enterprise kernel: WARNING: CPU: 4 PID: 8227 at ../fs/btrfs/extent-tree.c:898 lookup_inline_extent_backref+0x5a0/0x640 [btrfs]
Mär 12 13:33:05 enterprise kernel: Modules linked in: fuse af_packet nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat i>
Mär 12 13:33:05 enterprise kernel:  pcspkr fam15h_power ledtrig_audio k10temp snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep drm_kms_helper snd_pcm sp5100_tco drm snd_timer i2c_piix4 r8169 snd fb_sys_fops realtek syscopyarea sysfillrect sysimgblt >
Mär 12 13:33:05 enterprise kernel: CPU: 4 PID: 8227 Comm: snapperd Tainted: P           OE      5.3.18-lp152.66-default #1 openSUSE Leap 15.2
Mär 12 13:33:05 enterprise kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-UD3P, BIOS F1 08/06/2013
Mär 12 13:33:05 enterprise kernel: RIP: 0010:lookup_inline_extent_backref+0x5a0/0x640 [btrfs]
Mär 12 13:33:05 enterprise kernel: Code: 48 8b 5c 24 38 4c 8b 74 24 48 e9 17 fe ff ff 48 8b 5c 24 38 b8 8b ff ff ff e9 62 fe ff ff 48 c7 c7 68 06 6a c0 e8 ee f7 72 f9 <0f> 0b b8 fb ff ff ff e9 4a fe ff ff 48 8b 7c 24 18 48 c7 c6 e8 06
Mär 12 13:33:05 enterprise kernel: RSP: 0018:ffffb6c5c420f850 EFLAGS: 00010282
Mär 12 13:33:05 enterprise kernel: RAX: 0000000000000024 RBX: ffff8b66d82adbd0 RCX: 0000000000000000
Mär 12 13:33:05 enterprise kernel: RDX: 0000000000000000 RSI: ffff8b66deb19a18 RDI: ffff8b66deb19a18
Mär 12 13:33:05 enterprise kernel: RBP: ffff8b66dc617dd0 R08: 00000000000004c8 R09: 0000000000000001
Mär 12 13:33:05 enterprise kernel: R10: ffff8b62c0000000 R11: 0000000000000001 R12: 0000000000001000
Mär 12 13:33:05 enterprise kernel: R13: 0000000000000000 R14: 0000000035ad9000 R15: 000000000000000d
Mär 12 13:33:05 enterprise kernel: FS:  00007f65ff8ef700(0000) GS:ffff8b66deb00000(0000) knlGS:0000000000000000
Mär 12 13:33:05 enterprise kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mär 12 13:33:05 enterprise kernel: CR2: 00005576dace75f8 CR3: 00000002923e4000 CR4: 00000000000406e0
Mär 12 13:33:05 enterprise kernel: Call Trace:
Mär 12 13:33:05 enterprise kernel:  insert_inline_extent_backref+0x55/0xe0 [btrfs]
Mär 12 13:33:05 enterprise kernel:  __btrfs_inc_extent_ref.isra.46+0x88/0x260 [btrfs]
Mär 12 13:33:05 enterprise kernel:  __btrfs_run_delayed_refs+0x83c/0x1190 [btrfs]
Mär 12 13:33:05 enterprise kernel:  ? btrfs_reserve_metadata_bytes+0x7fa/0x950 [btrfs]
Mär 12 13:33:05 enterprise kernel:  btrfs_run_delayed_refs+0x62/0x1f0 [btrfs]
Mär 12 13:33:05 enterprise kernel:  btrfs_commit_transaction+0x50/0xa10 [btrfs]
Mär 12 13:33:05 enterprise kernel:  ? start_transaction+0xc5/0x550 [btrfs]
Mär 12 13:33:05 enterprise kernel:  btrfs_mksubvol+0x5cb/0x5e0 [btrfs]
Mär 12 13:33:05 enterprise kernel:  ? _cond_resched+0x15/0x40
Mär 12 13:33:05 enterprise kernel:  ? _cond_resched+0x15/0x40
Mär 12 13:33:05 enterprise kernel:  btrfs_ioctl_snap_create_transid+0x16f/0x1a0 [btrfs]
Mär 12 13:33:05 enterprise kernel:  btrfs_ioctl_snap_create_v2+0x126/0x180 [btrfs]
Mär 12 13:33:05 enterprise kernel:  btrfs_ioctl+0x1398/0x2fb0 [btrfs]
Mär 12 13:33:05 enterprise kernel:  ? do_filp_open+0xaf/0x110
Mär 12 13:33:05 enterprise kernel:  ? _copy_to_user+0x22/0x30
Mär 12 13:33:05 enterprise kernel:  ? cp_new_stat+0x150/0x180
Mär 12 13:33:05 enterprise kernel:  ? do_vfs_ioctl+0xa0/0x680
Mär 12 13:33:05 enterprise kernel:  do_vfs_ioctl+0xa0/0x680
Mär 12 13:33:05 enterprise kernel:  ksys_ioctl+0x70/0x80
Mär 12 13:33:05 enterprise kernel:  __x64_sys_ioctl+0x16/0x20
Mär 12 13:33:05 enterprise kernel:  do_syscall_64+0x65/0x1f0
Mär 12 13:33:05 enterprise kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Mär 12 13:33:05 enterprise kernel: RIP: 0033:0x7f66026929e7
Mär 12 13:33:05 enterprise kernel: Code: b3 66 90 48 8b 05 b1 14 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 81 14 2c 00 f7 d8 64 89 01 48
Mär 12 13:33:05 enterprise kernel: RSP: 002b:00007f65ff8ec438 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
Mär 12 13:33:05 enterprise kernel: RAX: ffffffffffffffda RBX: 00007f65f800d790 RCX: 00007f66026929e7
Mär 12 13:33:05 enterprise kernel: RDX: 00007f65ff8ec440 RSI: 0000000050009417 RDI: 0000000000000007
Mär 12 13:33:05 enterprise kernel: RBP: 00007f65ff8ec440 R08: 000000000000000f R09: 00007f66026dd3e0
Mär 12 13:33:05 enterprise kernel: R10: 000000000000055b R11: 0000000000000202 R12: 0000000000000006
Mär 12 13:33:05 enterprise kernel: R13: 0000000000000007 R14: 00007f65ff8ee5d0 R15: 0001000000000000
Mär 12 13:33:05 enterprise kernel: --- end trace 22cd536acdb32f40 ]---
Mär 12 13:33:05 enterprise kernel: BTRFS: error (device sda2) in btrfs_run_delayed_refs:2188: errno=-5 IO failure
Mär 12 13:33:05 enterprise kernel: BTRFS info (device sda2): forced readonly

Checking the SSD with smartctl does not result in any errors.

btrfs check results:

btrfs check --force /dev/sda2
Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
Checking filesystem on /dev/sda2
UUID: 26f39cac-64fe-48c8-8c59-bc9248e97f6d
[1/7] checking root items
[2/7] checking extents
ref mismatch on [900567040 4096] extent item 0, found 4
data backref 900567040 parent 22499704832 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 900567040 parent 22499704832 owner 0 offset 0 found 1 wanted 0 back 0x5652292521a0
data backref 900567040 parent 1385725952 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 900567040 parent 1385725952 owner 0 offset 0 found 1 wanted 0 back 0x5652278459c0
data backref 900567040 root 266 owner 38341 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 900567040 root 266 owner 38341 offset 0 found 1 wanted 0 back 0x5652250e5410
data backref 900567040 parent 1773486080 owner 0 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 900567040 parent 1773486080 owner 0 offset 0 found 1 wanted 0 back 0x565226e9c350
backpointer mismatch on [900567040 4096]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
root 266 inode 38341 errors 1000, some csum missing
root 458 inode 38341 errors 1000, some csum missing
root 461 inode 38341 errors 1000, some csum missing
root 464 inode 38341 errors 1000, some csum missing
root 466 inode 38341 errors 1000, some csum missing
root 467 inode 38341 errors 1000, some csum missing
root 468 inode 38341 errors 1000, some csum missing
root 505 inode 38341 errors 1000, some csum missing
root 506 inode 38341 errors 1000, some csum missing
root 509 inode 38341 errors 1000, some csum missing
root 510 inode 38341 errors 1000, some csum missing
root 511 inode 38341 errors 1000, some csum missing
root 512 inode 38341 errors 1000, some csum missing
root 513 inode 38341 errors 1000, some csum missing
root 514 inode 38341 errors 1000, some csum missing
root 515 inode 38341 errors 1000, some csum missing
root 516 inode 38341 errors 1000, some csum missing
root 517 inode 38341 errors 1000, some csum missing
root 518 inode 38341 errors 1000, some csum missing
ERROR: errors found in fs roots
found 22353182720 bytes used, error(s) found
total csum bytes: 20129884
total tree bytes: 1044054016
total fs tree bytes: 978796544
total extent tree bytes: 38780928
btree space waste bytes: 168055593
file data blocks allocated: 153981341696
 referenced 70521249792

A remount failed:

BTRFS error (device sda2): Remounting read-write after error is not allowed

With a live system I executed

btrfs scrub

btrfs rescue chunk-recover

btrfs rescue super-recover

After a reboot, the partition works normally and is RW-mounted. Only again when trying to update the software the fun starts all over again.

Does anyone have an idea?

#2

My guess is that you ran out of available space and don’t bother.

#3

I don’t think your recovery with a live system made a difference at all. Simply rebooting can get it mounted RW again.

If it is lack of free space, I would remove stale files, snapshots before attempting another update.

For better troubleshooting I’d reach out to Linux-BTRFS mailing list.

#4

I don’t think so, because it’s a 500 GB SSD and I have only the system files on this disk. I have a 5% usage.

#5

Show the output of the following command:

**erlangen:~ #** btrfs filesystem usage -T /   
Overall: 
    Device size:                  51.69GiB 
    Device allocated:             33.04GiB 
    Device unallocated:           18.65GiB 
    Device missing:                  0.00B 
    Used:                         23.90GiB 
    Free (estimated):             26.06GiB      (min: 26.06GiB) 
    Free (statfs, df):            26.06GiB 
    Data ratio:                       1.00 
    Metadata ratio:                   1.00 
    Global reserve:               69.27MiB      (used: 0.00B) 
    Multiple profiles:                  no 

                  Data     Metadata System               
Id Path           single   single   single   Unallocated 
-- -------------- -------- -------- -------- ----------- 
 1 /dev/nvme0n1p3 30.01GiB  3.00GiB 32.00MiB    18.65GiB 
-- -------------- -------- -------- -------- ----------- 
   Total          30.01GiB  3.00GiB 32.00MiB    18.65GiB 
   Used           22.59GiB  1.30GiB 16.00KiB             
**erlangen:~ #**
#6

If there are hardware issues with the drive, they may show up


dos@DOS1:~> sudo btrfs device stats / 
[/dev/nvme0n1p2].write_io_errs    0 
[/dev/nvme0n1p2].read_io_errs     0 
[/dev/nvme0n1p2].flush_io_errs    0 
[/dev/nvme0n1p2].corruption_errs  0 
[/dev/nvme0n1p2].generation_errs  0
#7

The same is happening to me.

**pruebas:~ #** btrfs filesystem usage -T / 
Overall: 
    Device size:                 931.51GiB 
    Device allocated:             51.05GiB 
    Device unallocated:          880.46GiB 
    Device missing:                  0.00B 
    Used:                         47.09GiB 
    Free (estimated):            883.29GiB      (min: 883.29GiB) 
    Data ratio:                       1.00 
    Metadata ratio:                   1.00 
    Global reserve:               97.61MiB      (used: 0.00B) 

                  Data     Metadata  System               
Id Path           single   single    single   Unallocated 
-- -------------- -------- --------- -------- ----------- 
 1 /dev/nvme0n1p1 49.01GiB   2.01GiB 32.00MiB   880.46GiB 
-- -------------- -------- --------- -------- ----------- 
   Total          49.01GiB   2.01GiB 32.00MiB   880.46GiB 
   Used           46.19GiB 926.98MiB 16.00KiB             
**pruebas:~ #**
#8

I have tried to make some space, i rebooted, I tested that the fs was mounted rw and deled firefox cache (1GB), and then


**pruebas:~ #** systemctl start btrfs-balance.service 
**pruebas:~ #** journalctl -u btrfs-balance.service 
-- Logs begin at Sun 2021-07-18 21:19:11 CEST, end at Sun 2021-07-18 21:22:37 CEST. -- 
Jul 18 21:22:36 pruebas systemd[1]: Started Balance block groups on a btrfs filesystem. 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: Before balance of / 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: Data, single: total=49.01GiB, used=46.55GiB 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: System, single: total=32.00MiB, used=16.00KiB 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: Metadata, single: total=2.01GiB, used=937.84MiB 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: GlobalReserve, single: total=98.00MiB, used=0.00B 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: Filesystem      Size  Used Avail Use% Mounted on 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: /dev/nvme0n1p1  1.1T   52G  949G   6% / 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: ERROR: error during balancing '/': Read-only file system 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: There may be more info in syslog - try dmesg | tail 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: ERROR: error during balancing '/': Read-only file system 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: There may be more info in syslog - try dmesg | tail 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: Dumping filters: flags 0x1, state 0x0, force is off 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]:   DATA (flags 0x2): balancing, usage=5 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: ERROR: error during balancing '/': Read-only file system 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: There may be more info in syslog - try dmesg | tail 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: Dumping filters: flags 0x1, state 0x0, force is off 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]:   DATA (flags 0x2): balancing, usage=10 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: ERROR: error during balancing '/': Read-only file system 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: There may be more info in syslog - try dmesg | tail 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: ERROR: error during balancing '/': Read-only file system 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: There may be more info in syslog - try dmesg | tail 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: Dumping filters: flags 0x6, state 0x0, force is off 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]:   METADATA (flags 0x2): balancing, usage=3 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]:   SYSTEM (flags 0x2): balancing, usage=3 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: After balance of / 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: Data, single: total=49.01GiB, used=46.55GiB 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: System, single: total=32.00MiB, used=16.00KiB 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: Metadata, single: total=2.01GiB, used=937.84MiB 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: GlobalReserve, single: total=98.00MiB, used=0.00B 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: Filesystem      Size  Used Avail Use% Mounted on 
Jul 18 21:22:37 pruebas btrfs-balance.sh[3009]: /dev/nvme0n1p1  1.1T   52G  949G   6% / 
Jul 18 21:22:37 pruebas systemd[1]: btrfs-balance.service: Succeeded. 
**pruebas:~ #**
#9

Umm. This sounds very much similar to the issue I had. In my case somehow the btrfs “/” root entry in /etc/fstab got lost (which results in the rootfs remaing mounted as readonly). And once it’s lost, yast seems to happy to keep ignoring it:
https://forums.opensuse.org/showthread.php/557179-btrfs-rootfs-mounted-readonly

Just saying. Maybe check your /etc/fstab contents. I have no idea how the entry got lost on my system. This may well have happened after some update - since it clearly was working for the first few weeks after I had upgraded my system to 15.2.

#10

fstab is ok

**pruebas:~ #** cat /etc/fstab 
UUID=5b000355-3a1a-49f5-8005-f10668008aa7  /            btrfs  defaults              0  0 
UUID=5b000355-3a1a-49f5-8005-f10668008aa7  /.snapshots  btrfs  subvol=/@/.snapshots  0  0 
UUID=5b000355-3a1a-49f5-8005-f10668008aa7  /var         btrfs  subvol=/@/var         0  0 
UUID=5b000355-3a1a-49f5-8005-f10668008aa7  /usr/local   btrfs  subvol=/@/usr/local   0  0 
UUID=5b000355-3a1a-49f5-8005-f10668008aa7  /tmp         btrfs  subvol=/@/tmp         0  0 
UUID=5b000355-3a1a-49f5-8005-f10668008aa7  /srv         btrfs  subvol=/@/srv         0  0 
UUID=5b000355-3a1a-49f5-8005-f10668008aa7  /root        btrfs  subvol=/@/root        0  0 
UUID=5b000355-3a1a-49f5-8005-f10668008aa7  /opt         btrfs  subvol=/@/opt         0  0 
UUID=5b000355-3a1a-49f5-8005-f10668008aa7  /home        btrfs  subvol=/@/home        0  0 
UUID=d8840035-b312-4034-8e0f-54a67505de9a  /boot        ext4   data=ordered          0  2 
UUID=67e3f0e9-f93c-40cd-a13d-aa2a36ba91ab  swap         swap   sw,pri=2              0  0 
UUID=9369acab-a71c-4274-b8d6-f9ab9da6e7a9  swap         swap   sw,pri=1              0  0 
**pruebas:~ #**

and the fs is mounted rw on boot and it keep woking during a while, but then it changes to ro

BTW can I remount it rw when that happen instead of rebooting?

#11

You can. Whether it succeeds depends on the cause it mounted r-o in the first place.

Some commands to run and report back here:


sudo btrfs device stats /
dmesg | grep -C 5 -i btrfs
#12 

**pruebas:~ #** btrfs device stats / 
[/dev/nvme0n1p1].write_io_errs    0 
[/dev/nvme0n1p1].read_io_errs     0 
[/dev/nvme0n1p1].flush_io_errs    0 
[/dev/nvme0n1p1].corruption_errs  4 
[/dev/nvme0n1p1].generation_errs  0 
**pruebas:~ #**

This one seems prety ugly

**pruebas:~ #** dmesg | grep -C 5 -i btrfs 
...
    5.839496] **Btrfs** loaded, crc32c=crc32c-generic, assert=on 
    5.846331] **BTRFS**: device fsid 5b000355-3a1a-49f5-8005-f10668008aa7 devid 1 transid 12161 /dev/nvme0n1p1 
...       
--
    8.393000] sd 10:0:0:0: [sda] No Caching mode page found 
    8.398819] sd 10:0:0:0: [sda] Assuming drive cache: write through 
    8.458919]  sda: sda1 sda2 sda3 sda4 
    8.467533] sd 10:0:0:0: [sda] Attached SCSI removable disk 
    8.845951] PM: Image not found (code -22) 
    8.929889] **BTRFS** info (device nvme0n1p1): disk space caching is enabled 
    8.929890] **BTRFS** info (device nvme0n1p1): has skinny extents 
    8.932601] **BTRFS** info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 
    8.976779] **BTRFS** info (device nvme0n1p1): enabling ssd optimizations 
    9.463110] systemd-journald[147]: Received SIGTERM from PID 1 (systemd). 
    9.696313] systemd[1]: systemd 246.13+suse.141.gac812b8fa0 running in system mode. (+PAM +AUDIT +SELINUX -IMA +APPARMOR -SMACK +SYSVINIT +
UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid) 
    9.705462] systemd[1]: Detected architecture x86-64. 
    9.711638] systemd[1]: Set hostname to <pruebas>. 
    9.979431] systemd-sysv-generator[439]: SysV service '/etc/init.d/vmware-USBArbitrator' lacks a native systemd unit file. Automatically ge
nerating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust. 
--
   10.853337] systemd[1]: Listening on Syslog Socket. 
   10.867353] systemd[1]: Starting Journal Service... 
   10.880372] systemd[1]: Starting Load Kernel Modules... 
   10.893261] systemd[1]: Starting Remount Root and Kernel File Systems... 
   10.900991] systemd[1]: Starting Coldplug All udev Devices... 
   10.902607] **BTRFS** info (device nvme0n1p1): disk space caching is enabled 
   10.915239] systemd[1]: sysroot.mount: Succeeded. 
   10.918475] systemd[1]: Activated swap /dev/disk/by-uuid/67e3f0e9-f93c-40cd-a13d-aa2a36ba91ab. 
   10.933213] systemd[1]: dev-disk-by\x2duuid-67e3f0e9\x2df93c\x2d40cd\x2da13d\x2daa2a36ba91ab.swap: Succeeded. 
   10.933462] systemd[1]: Activated swap /dev/disk/by-uuid/9369acab-a71c-4274-b8d6-f9ab9da6e7a9. 
   10.952037] systemd[1]: Mounted Huge Pages File System. 
--
   23.449696] Initialized host personality 
   23.492470] NET: Registered protocol family 40 
   23.534642] ppdev: user-space parallel port driver 
   24.681050] resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000d0000-0x000dffff
 window] 
   24.681302] caller _nv001015rm+0x1bf/0x1f0 [nvidia] mapping multiple BARs 
   84.590495] **BTRFS** info (device nvme0n1p1): qgroup scan completed (inconsistency flag cleared) 
  169.593279] **BTRFS** info (device nvme0n1p1): qgroup scan completed (inconsistency flag cleared) 
  266.186543] **BTRFS** info (device nvme0n1p1): qgroup scan completed (inconsistency flag cleared) 
  426.347661] ------------ cut here ]------------ 
  426.347750] WARNING: CPU: 1 PID: 383 at ../fs/**btrfs**/extent-tree.c:3068 __**btrfs**_free_extent.isra.49+0x201/0xbc0 **btrfs**] 
  426.347751] Modules linked in: ppdev parport_pc parport vmw_vsock_vmci_transport vsock vmw_vmci af_packet nft_fib_inet nft_fib_ipv4 nft_fib
_ipv6 nft_fib nft_reject_inet nft_reject nft_ct nft_chain_nat nf_tables ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw i
p6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set iscsi_ibft i
scsi_boot_sysfs nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter dmi_sysfs ext4 
crc16 mbcache jbd2 squashfs loop nvidia_drm(POEXX) nvidia_modeset(POEXX) nvidia_uvm(POEXX) nvidia(POEXX) edac_mce_amd snd_hda_codec_realtek kv
m_amd snd_hda_codec_hdmi ccp snd_hda_codec_generic ledtrig_audio kvm snd_hda_intel snd_intel_dspcfg soundwire_intel soundwire_generic_allocati
on soundwire_cadence irqbypass soundwire_bus drm_kms_helper snd_hda_codec cec joydev wmi_bmof snd_hda_core pcspkr rc_core k10temp r8169 snd_hw
dep realtek 
  426.347780]  ipmi_devintf snd_soc_core libphy ipmi_msghandler syscopyarea snd_compress sysfillrect sysimgblt snd_pcm_dmaengine fb_sys_fops 
asus_atk0110(NN) snd_pcm sp5100_tco(NN) snd_timer snd i2c_piix4 soundcore acpi_cpufreq drm fuse configfs sd_mod uas usb_storage **btrfs** libcrc32
c hid_generic xor usbhid raid6_pq ohci_pci ata_generic xhci_pci xhci_hcd nvme pata_jmicron ehci_pci ohci_hcd firewire_ohci ehci_hcd pata_atiix
p ahci libahci usbcore libata firewire_core serio_raw nvme_core t10_pi crc_itu_t wmi button sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc sc
si_dh_alua scsi_mod msr 
  426.347804] Supported: No, Proprietary and Unsupported modules are loaded 
  426.347808] CPU: 1 PID: 383 Comm: **btrfs**-transacti Tainted: P           OE  X  N 5.3.18-59.13-default #1 SLE15-SP3 
  426.347809] Hardware name: System manufacturer System Product Name/M4A89TD PRO USB3, BIOS 3029    09/07/2012 
  426.347831] RIP: 0010:__**btrfs**_free_extent.isra.49+0x201/0xbc0 **btrfs**] 
  426.347834] Code: 89 e6 e8 a2 cc ff ff 41 89 c5 58 45 85 ed 0f 84 90 00 00 00 41 83 fd fe 0f 85 11 ff ff ff 48 c7 c7 b0 58 7a c0 e8 7d a3 a
2 e2 <0f> 0b 49 8b 3c 24 e8 f4 6b 00 00 ff b4 24 b8 00 00 00 48 8b 7c 24 
  426.347835] RSP: 0018:ffffa82040593c58 EFLAGS: 00010282
....
  426.347845] Call Trace: 
  426.347872]  ? alloc_reserved_file_extent+0x268/0x300 **btrfs**] 
  426.347894]  __**btrfs**_run_delayed_refs+0x54c/0x1180 **btrfs**] 
  426.347899]  ? __switch_to_asm+0x40/0x70 
  426.347900]  ? __switch_to_asm+0x34/0x70 
  426.347902]  ? __switch_to_asm+0x40/0x70 
  426.347903]  ? __switch_to_asm+0x40/0x70 
  426.347906]  ? __switch_to+0x330/0x4d0 
  426.347908]  ? __switch_to_asm+0x34/0x70 
  426.347929]  **btrfs**_run_delayed_refs+0x62/0x1f0 **btrfs**] 
  426.347953]  **btrfs**_commit_transaction+0x50/0xa10 **btrfs**] 
  426.347976]  ? start_transaction+0xc5/0x550 **btrfs**] 
  426.347999]  transaction_kthread+0x148/0x160 **btrfs**] 
  426.348022]  ? **btrfs**_cleanup_transaction+0x5c0/0x5c0 **btrfs**] 
  426.348024]  kthread+0x10d/0x130 
  426.348027]  ? kthread_park+0xa0/0xa0 
  426.348029]  ret_from_fork+0x22/0x40 
  426.348033] --- end trace 459b7a9b08f0a188 ]--- 
  426.348038] **BTRFS** info (device nvme0n1p1): leaf 1424736256 gen 12186 total ptrs 186 free space 2186 owner 2 
  426.348040]  item 0 key (227254272 168 8192) itemoff 16233 itemsize 50 
  426.348041]          extent refs 2 gen 7850 flags 1 
  426.348042]          ref#0: shared data backref parent 49787895808 count 1 
  426.348044]          ref#1: shared data backref parent 49660346368 count 1 
  426.348046]  item 1 key (227262464 168 12288) itemoff 16183 itemsize 50 
--
  426.348798]          extent refs 1 gen 9106 flags 1 
  426.348798]          ref#0: extent data backref root 263 objectid 19735 offset 0 count 1 
  426.348800]  item 185 key (230137856 168 32768) itemoff 6836 itemsize 53 
  426.348801]          extent refs 1 gen 9106 flags 1 
  426.348802]          ref#0: extent data backref root 263 objectid 19736 offset 0 count 1 
  426.348805] **BTRFS** error (device nvme0n1p1): unable to find ref byte nr 227831808 parent 0 root 263  owner 7983 offset 0 
  426.348805] ------------ cut here ]------------ 
  426.348806] **BTRFS**: Transaction aborted (error -2) 
  426.348851] WARNING: CPU: 1 PID: 383 at ../fs/**btrfs**/extent-tree.c:3074 __**btrfs**_free_extent.isra.49+0x25a/0xbc0 **btrfs**] 
  426.348851] Modules linked in: ppdev parport_pc parport vmw_vsock_vmci_transport vsock vmw_vmci af_packet nft_fib_inet nft_fib_ipv4 nft_fib
_ipv6 nft_fib nft_reject_inet nft_reject nft_ct nft_chain_nat nf_tables ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw i
p6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set iscsi_ibft i
scsi_boot_sysfs nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter dmi_sysfs ext4 
crc16 mbcache jbd2 squashfs loop nvidia_drm(POEXX) nvidia_modeset(POEXX) nvidia_uvm(POEXX) nvidia(POEXX) edac_mce_amd snd_hda_codec_realtek kv
m_amd snd_hda_codec_hdmi ccp snd_hda_codec_generic ledtrig_audio kvm snd_hda_intel snd_intel_dspcfg soundwire_intel soundwire_generic_allocati
on soundwire_cadence irqbypass soundwire_bus drm_kms_helper snd_hda_codec cec joydev wmi_bmof snd_hda_core pcspkr rc_core k10temp r8169 snd_hw
dep realtek 
  426.348874]  ipmi_devintf snd_soc_core libphy ipmi_msghandler syscopyarea snd_compress sysfillrect sysimgblt snd_pcm_dmaengine fb_sys_fops 
asus_atk0110(NN) snd_pcm sp5100_tco(NN) snd_timer snd i2c_piix4 soundcore acpi_cpufreq drm fuse configfs sd_mod uas usb_storage **btrfs** libcrc32
c hid_generic xor usbhid raid6_pq ohci_pci ata_generic xhci_pci xhci_hcd nvme pata_jmicron ehci_pci ohci_hcd firewire_ohci ehci_hcd pata_atiix
p ahci libahci usbcore libata firewire_core serio_raw nvme_core t10_pi crc_itu_t wmi button sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc sc
si_dh_alua scsi_mod msr 
  426.348890] Supported: No, Proprietary and Unsupported modules are loaded 
  426.348893] CPU: 1 PID: 383 Comm: **btrfs**-transacti Tainted: P        W  OE  X  N 5.3.18-59.13-default #1 SLE15-SP3 
  426.348893] Hardware name: System manufacturer System Product Name/M4A89TD PRO USB3, BIOS 3029    09/07/2012 
  426.348914] RIP: 0010:__**btrfs**_free_extent.isra.49+0x25a/0xbc0 **btrfs**] 
  426.348915] Code: 48 8b 44 24 08 48 8b 40 50 f0 48 0f ba a8 98 09 00 00 03 0f 92 c0 84 c0 5f 75 11 44 89 ee 48 c7 c7 a0 59 7a c0 e8 16 b2 9
b e2 <0f> 0b 48 8b 3c 24 b9 fe ff ff ff ba 02 0c 00 00 48 c7 c6 80 71 79 
  426.348916] RSP: 0018:ffffa82040593c58 EFLAGS: 00010286 
...
  426.348924] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 
  426.348925] CR2: 00007fae483f8000 CR3: 00000001b6c6c000 CR4: 00000000000006e0 
  426.348925] Call Trace: 
  426.348947]  ? alloc_reserved_file_extent+0x268/0x300 **btrfs**] 
  426.348969]  __**btrfs**_run_delayed_refs+0x54c/0x1180 **btrfs**] 
  426.348972]  ? __switch_to_asm+0x40/0x70 
  426.348973]  ? __switch_to_asm+0x34/0x70 
  426.348974]  ? __switch_to_asm+0x40/0x70 
  426.348976]  ? __switch_to_asm+0x40/0x70 
  426.348978]  ? __switch_to+0x330/0x4d0 
  426.348979]  ? __switch_to_asm+0x34/0x70 
  426.349000]  **btrfs**_run_delayed_refs+0x62/0x1f0 **btrfs**] 
  426.349024]  **btrfs**_commit_transaction+0x50/0xa10 **btrfs**] 
  426.349047]  ? start_transaction+0xc5/0x550 **btrfs**] 
  426.349069]  transaction_kthread+0x148/0x160 **btrfs**] 
  426.349092]  ? **btrfs**_cleanup_transaction+0x5c0/0x5c0 **btrfs**] 
  426.349094]  kthread+0x10d/0x130 
  426.349095]  ? kthread_park+0xa0/0xa0 
  426.349097]  ret_from_fork+0x22/0x40 
  426.349101] --- end trace 459b7a9b08f0a189 ]--- 
  426.349104] **BTRFS**: error (device nvme0n1p1) in __**btrfs**_free_extent:3074: errno=-2 No such entry 
  426.349105] **BTRFS** info (device nvme0n1p1): forced readonly 
  426.349108] **BTRFS**: error (device nvme0n1p1) in **btrfs**_run_delayed_refs:2188: errno=-2 No such entry 
**pruebas:~ #**
#13

The first command shows corruption was reported for this drive, the second one shows a btrfs error leading to force readonly. These 2 things might be related. It doesn’t show what triggered the error at the 426s mark.

Some things you can do about it:

  1. Run a system check (RAM, drives, etc)
  2. Format partition and reinstall system, restore data from backup

You can boot the system in r/w mode, I believe it’s not a structural damage to the filesystem. It’s a possible hardware/firmware issue that affected a non-critical area of the fs. Do further troubleshooting if you want to salvage it https://en.opensuse.org/SDB:BTRFS#How_to_repair_a_broken.2Funmountable_btrfs_filesystem

#14

I have passed a full ram test with no errors

and, booting with a system rescue disk and testing the btrfs fs

[root@sysrescue ~]# btrfs check /dev/nvme0n1p1
Opening filesystem to check...
Checking filesystem on /dev/nvme0n1p1
UUID: 5b000355-3a1a-49f5-8005-f10668008aa7
[1/7] checking root items
[2/7] checking extents
data backref 227831808 root 263 owner 7983 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 227831808 root 263 owner 7983 offset 0 found 1 wanted 0 back 0x614f330
incorrect local backref count on 227831808 root 263 owner 140737488363311 offset 0 found 0 wanted 1 back 0x5415380
backref disk bytenr does not match extent record, bytenr=227831808, ref bytenr=0
backpointer mismatch on [227831808 4096]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups
found 50062684160 bytes used, error(s) found
total csum bytes: 46482488
total tree bytes: 931463168
total fs tree bytes: 829816832
total extent tree bytes: 45875200
btree space waste bytes: 141986007
file data blocks allocated: 213846175744
 referenced 79108595712
[root@sysrescue ~]#

yes, there are errors, but i don’t understand which errors… I guess I have to read a lot about btrfs

I have noticed two items:

1st: ERROR: errors found in extent allocation tree or chunk allocation

2nd: btree space waste bytes: 141986007 it is a lot of space!

#15

At this point you can reach out to btrfs developers at https://btrfs.wiki.kernel.org/index.php/Btrfs_mailing_list, they usually help with these.

#16

I have found what the problem was … firefox!!! :open_mouth:

I don’t know why, but what was causing the btrfs filesystem to report an error and go RO was the version 90.0.1 of firefox from the mozilla repo.

Now with firefox 78.12.0 from the opensuse update repo everything works fine.

It is not something related to the program being installed or libraries or something but the error happened when the program was running, I detected it by trial and error and I found that if I used opera instead of firefox the problem did not happen, but launching firefoz and some minutes after the fs go read only… even without using firefox at all, jut open it.

I don’t know if it should be reported to btrfs or firefox developers.

best regards

#17

I don’t see how a program could cause this issue. To confirm this, reinstall FF 90.

That said, do not ignore the corruption errors that were reported by “btrfs device stats”. It’s more of a hardware thing rather than software. Some failure happen in the storage occupied by FF 90, installing the other version used other storage space.

#18

You are right. I have reinstalled FF 90 and it works fine.
So I have to find why I had the errors and try to fix them… btrfs check and repair sound pretty scary