BTRFS full: forced readonly

English Install/Boot/Login
1

Hello,

I have my system files on a btrfs. The machine suddenly did not boot normally. Systemd did not start the login, network and graphical display leaving me with the tty.
The reason is a full btrfs filesystem, with no space to make files on the /tmp directory. This frustrates systemd, giving strange errors.

I tried to recover, but got stuck. The space is 0% on the btrfs, giving me only a read only mount. Removing some files is therefore impossible. I tried this also via a recovery system (boot via USB). This gives the same result and now i am stuck.

Hopefully someone can help me to mount the partition as read-write, so i can remove some files and continue.

Mounting the btrfs partition (/dev/sdd1) on the recovery system gives:

# mount /dev/sdd1 temp-mount/
 3049.751115] BTRFS info (device sdd1): disk space caching is enabled
 3049.751117] BTRFS info (device sdd1): has skinny extents
 3049.790239] BTRFS info (device sdd1): enabling ssd optimizations
 3049.795922] BTRFS info (device sdd1): checking UUID tree
 3049.854980] BTRFS warning (device sdd1): btrfs_uuid_scan_kthread failed -28

(the disk is then read-only)

# mount 
/dev/sdd1 on /home/linux/temp-mount type btrfs (ro,relatime,ssd,space_cache,skip_balance,subvolid=257,subvol=/@)

and trying to remount as a read write partition gives:

# mount -o remount,rw,skip_balance /dev/sdd1 
 3076.761128] BTRFS info (device sdd1): disk space caching is enabled
 3078.943146] BTRFS: error (device sdd1) in btrfs_drop_snapshot:5505: errno=-28 No space left
 3078.943151] BTRFS info (device sdd1): forced readonly

The partition is then mounted read only, giving me no option to remove some files. Unmounting and checking gives:

# umount /dev/sdd1

# btrfs check /dev/sdd1 
Opening filesystem to check...
Checking filesystem on /dev/sdd1
UUID: 11e837a4-2ab2-4a08-a6ca-e681ac45afce
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 31472496640 bytes used, no error found
total csum bytes: 29688304
total tree bytes: 1064960000
total fs tree bytes: 988725248
total extent tree bytes: 38453248
btree space waste bytes: 175403793
file data blocks allocated: 338649542656
 referenced 77958098944

The kernel version on the recovery USB is:

# uname -r
5.7.5-1-default
2

I solved this by mouting the btrfs via ‘clear_cache’ option and then delete some files immediately after the mount.

# umount /dev/sdd1
# mount -o rw,clear_cache /dev/sdd1 temp-mount/
 5812.772627] BTRFS info (device sdd1): force clearing of disk cache
 5812.772629] BTRFS info (device sdd1): disk space caching is enabled
 5812.772630] BTRFS info (device sdd1): has skinny extents
 5812.813089] BTRFS info (device sdd1): enabling ssd optimizations
 5812.818800] BTRFS info (device sdd1): checking UUID tree
 5812.880389] BTRFS warning (device sdd1): btrfs_uuid_scan_kthread failed -28

It looked OK. I checked with the mount command and the system was read-write.

# mount
/dev/sdd1 on /home/linux/temp-mount type btrfs (rw,relatime,ssd,space_cache,clear_cache,subvolid=257,subvol=/@)

Checking again to post the output in this forum gave me a read-only filesystem:

# mount
/dev/sdd1 on /home/linux/temp-mount type btrfs (ro,relatime,ssd,space_cache,clear_cache,subvolid=257,subvol=/@)

What happened? The kernel log shows:

 5844.656825] BTRFS: error (device sdd1) in btrfs_drop_snapshot:5505: errno=-28 No space left
 5844.656829] BTRFS info (device sdd1): forced readonly

This was no good because the system became a read-only filesystem, just by checking?

I found in the kernel log the following:

 5597.812766] BTRFS info (device sdd1): force clearing of disk cache
 5597.812768] BTRFS info (device sdd1): disk space caching is enabled
 5597.812769] BTRFS info (device sdd1): has skinny extents
 5597.852945] BTRFS info (device sdd1): enabling ssd optimizations
 5597.858599] BTRFS info (device sdd1): checking UUID tree
 5597.918148] BTRFS warning (device sdd1): btrfs_uuid_scan_kthread failed -28
 5632.003644] BTRFS: error (device sdd1) in btrfs_drop_snapshot:5505: errno=-28 No space left
 5632.003649] BTRFS info (device sdd1): forced readonly

and

 5812.772627] BTRFS info (device sdd1): force clearing of disk cache
 5812.772629] BTRFS info (device sdd1): disk space caching is enabled
 5812.772630] BTRFS info (device sdd1): has skinny extents
 5812.813089] BTRFS info (device sdd1): enabling ssd optimizations
 5812.818800] BTRFS info (device sdd1): checking UUID tree
 5812.880389] BTRFS warning (device sdd1): btrfs_uuid_scan_kthread failed -28
 5844.656825] BTRFS: error (device sdd1) in btrfs_drop_snapshot:5505: errno=-28 No space left
 5844.656829] BTRFS info (device sdd1): forced readonly

There appears to be a 32-34 seconds of delay between the mount and the drive becoming read-only!
So i must be quick.

Trying again did give me a read-write filesystem.

# umount /dev/sdd1
# mount -o rw,clear_cache /dev/sdd1 temp-mount/
 5914.987809] BTRFS info (device sdd1): force clearing of disk cache
 5914.987811] BTRFS info (device sdd1): disk space caching is enabled
 5914.987812] BTRFS info (device sdd1): has skinny extents
 5915.029363] BTRFS info (device sdd1): enabling ssd optimizations
 5915.033179] BTRFS info (device sdd1): checking UUID tree
 5915.092812] BTRFS warning (device sdd1): btrfs_uuid_scan_kthread failed -28

Immediately after the mount, i deleted some files in the /tmp directory. This works!

# cd temp-mount/tmp/
# find . -type f -mtime +30 -exec rm -f {} \;

Solved. The system stays read-write.

There were a lot of “old” files in the tmp directory…

# df -h
/dev/sdd1        30G   26G  4.5G  86% /home/linux/temp-mount