btrfs problems

English Install/Boot/Login
#1

Hi,

I have run into a problem today with btrfs which is incredibly frustrating. I have a habit of keeping a lot of junk arround and today my root (/) file system and today it has filled up so I cant login.

So I logged in with the command line and saw this:

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        81G   80G  740K 100% /

I thought easy I have a 30GB file on there that I dont need any more, lets delete that. After deleting the 30GB file I now have a 100% full file system, see:

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        81G   80G  740K 100% /

Having read other posts about btrfs and its chunk allocation I guesed I needed to run:

btrfs balance start -dusage=55 /

Which gave me the output of:

Done, had to relocate 0 out of 84 chunks

and left my file system 100% full.

So I thought humm I should change 55, so I have tried 1, 2, 3, 10, 20, 30, 40, 50, 60,70, 80, 90, 95, 96, 97, 98, 99 all of which get me the same output.

The other thing I have found is running this

 btrfs balance start -m /

Problem is that this cant complete because there is not enough space on the root file system!

Anyone have any ideas on how to fix this?

#2

It’s most likely due to snapshots filling your entire drive.

Do a:
snapper list
then look at how many snapshots you have. Leave a few just in case and issue:
snapper delete 1-xxx

Where xxx is a few snapshots “in the past”, just in case you need to rollback something.

#3

Cool beans thats the badger. Is there a way to turn off snapshots, its not something I really use as I have other backup systems in place (as of course snapshots dont protect me from hardware failure)?

Thanks

#4

Take a peek here; https://forums.opensuse.org/showthread.php/503019-How-to-disable-auto-snapshots-in-13-2-or-limit-the-disk-space-used-by-snapshots-files

#5

This is impressive, for two reasons: One, it’s really good news that this file system is being this tolerant in this particular case being this full. Btrfs has has been on a very long road to stability due to ENOSPC issues, so getting to this point and being able to recover from it is a very good sign. Two, not good that snapper isn’t doing a better job cleaning up after itself.

Suggestions: Next time, you can do:

btrfs device add /dev/sdX <mp>

Even if this is a e.g. 16GB USB stick, or a spare partition on a USB drive, or whatever, this immediately gives the file breathing room. Now you can safely start deleting snapshots and files. This is really important because in copy on write filesystems, deleting consumes space before it creates free space. Deleting anything requires metadata writes to reflect the changes in the filesystem, only once those writes are totally finished and committed to disk, will blocks be freed.

So once the file system has breathing room, now you can delete files and subvolumes. Once that’s done, you can

btrfs device delete /dev/sdX <mp>

And any metadata that needed to be written to that temp block device will be migrated to remaining devices.

More suggestions: File a bug for snapper. It really needs to expire older snapshots much sooner. This is kinda difficult in certain cases where very large files are on Btrfs being snapshot. It can cause a lot more space being used than expected. There’s on-going work to try to make this more predictable but it’s kindof a difficult problem.

Also, file a feature request (or possibly a bug) for the DE you’re using, so that it provides notifications of low free space on mounted volumes. At least, / and /home probably should, by default, evaluate free space periodically by the DE, so that the user can be warned. Btrfs might be more sensitive to these issues, but any file system could very easily faceplant in a really bad way by totally running out of free space. The DE should tell the user.

#6

I’m impressed! 81GB (of Btrfs) its full. Doesn’t matter whatever size from the beginning (20-40-80GB) It cause problems.Why in h**l was it allowed and set as default for openSUSE DE users? What can we learn from this?

regards

#7

The problem has very little to do with BTRFS but everything to do with Snapper which is a SUSE product. You can completely disable Snapper snapshots upon installation which completely eliminates the problem.

Their defaults for managing the number of snapshots as well as automatic clean up wasn’t very well thought out. It’s great if you don’t do a lot of installations, updates or removals but breaks into pieces when you have hundreds or thousands of updates on constant basis.

Edit:
On that note, Snapper should have a fuzzy logic that removes snapshots if the disk usage is above X % and this behaviour should also be applied to the zypp version of it. In theory you can run into a situation where you’re installing an update but because snapper snapshots things, the update fails to install since the disk is at full capacity, leaving you with a crippled system. In theory.

#8

Also the default snapshot retention and cleanup settings are really easy to change by editing the config file and/or using snapper commands to change it and to perform clean up tasks dynamically. That way I usually manage to keep Tumbleweed’s root partition within 20GB usage give or take of its 35GB overall size.

#9

Not mine of the sober…

Would chose Btrfs if knowing the problems of it?

Ok. Nice to have on a corporate level but… ?

Me sins 16 years of user DE in reality. 20GB. It takes is man to maintain the / root… Btrfs I know. Could be working if…

regards

#10

Thanks for the snapper info. I was about to pull my hair out wondering where all the disk space was going.

I do not understand how to configure snapper. This is an excerpt from the config file:

# run daily number cleanup
NUMBER_CLEANUP="yes"


# limit for number cleanup
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="10"
NUMBER_LIMIT_IMPORTANT="10"

I had more than 10 entries listed by “snapper list”.

What units are used for the age? Does this say it will only clean up if the entries are 1800 hours old (75 days)? So there may be an unlimited number of snapshots less than the min age, but only 10 older?

#11

NUMBER_MIN_AGE=“1800” is in seconds, so it is 30 minutes not 75 days. Also, by default snapper only runs once a day.

#12

On Fri 06 Mar 2015 09:26:01 PM CST, J Andrew wrote:

mattm3a;2698459 Wrote:
> Thanks for the snapper info. I was about to pull my hair out wondering
> where all the disk space was going.
>
> I do not understand how to configure snapper. This is an excerpt from
> the config file:
>
> >
Code:

> > # run daily number cleanup
> NUMBER_CLEANUP=“yes”
>
>
> # limit for number cleanup
> NUMBER_MIN_AGE=“1800”
> NUMBER_LIMIT=“10”
> NUMBER_LIMIT_IMPORTANT=“10”
>

> >
> I had more than 10 entries listed by “snapper list”.
>
> What units are used for the age? Does this say it will only clean up
> if the entries are 1800 hours old (75 days)? So there may be an
> unlimited number of snapshots less than the min age, but only 10
> older?

NUMBER_MIN_AGE=“1800” is in seconds, so it is 30 minutes not 75 days.
Also, by default snapper only runs once a day.

Hi
You mean the snapper clean up job to remove all the snapshots (well
most of it as it leaves some breadcumbs and the folder, which is a bug
AFAIK) based on your config rules.

SO if you install a lot of stuff in bits and pieces you can create
pre/post snapshots (run snapper list).

–
Cheers Malcolm °¿° LFCS, SUSE Knowledge Partner (Linux Counter #276890)
SUSE Linux Enterprise Desktop 12 GNOME 3.10.1 Kernel 3.12.36-38-default
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below… Thanks!