Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: btrfs, snapper, and recovering lost space after a rollback?

  1. #1

    Question btrfs, snapper, and recovering lost space after a rollback?

    I'm running Tumbleweed with Snapper/snapshots. X.org broke one time and I used the snapshots to recover my system (and wait until the nVidia drivers were fixed). However, I now seem to have a lot of space missing on my main partition, and I'm trying to work out how to recover it.

    I *think* what has happened is that my original "/" subvolume and the snapshot subvolume that I'm now booting from have diverged a lot (because it's Tumbleweed) but I don't know if there's a way to recover the space.

    Code:
    $ mount | grep ^/dev
    /dev/mapper/main-root on / type btrfs (rw,relatime,ssd,space_cache,subvolid=1017,subvol=/@/.snapshots/685/snapshot)
    /dev/mapper/main-root on /srv type btrfs (rw,relatime,ssd,space_cache,subvolid=259,subvol=/@/srv)
    /dev/mapper/main-root on /boot/grub2/i386-pc type btrfs (rw,relatime,ssd,space_cache,subvolid=260,subvol=/@/boot/grub2/i386-pc)
    /dev/mapper/main-root on /.snapshots type btrfs (rw,relatime,ssd,space_cache,subvolid=272,subvol=/@/.snapshots)
    /dev/mapper/main-root on /opt type btrfs (rw,relatime,ssd,space_cache,subvolid=258,subvol=/@/opt)
    /dev/mapper/main-root on /boot/grub2/x86_64-efi type btrfs (rw,relatime,ssd,space_cache,subvolid=261,subvol=/@/boot/grub2/x86_64-efi)
    /dev/sdb4 on /boot/efi type vfat (rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro)
    /dev/sdb3 on /var type ext4 (rw,relatime,data=ordered)
    /dev/sdc1 on /mnt/backup type ext4 (rw,relatime,stripe=32750,data=ordered)
    /dev/sdb2 on /mnt/media type ext4 (rw,relatime,stripe=32747,data=ordered)
    /dev/sdb5 on /tmp type ext4 (rw,relatime,data=ordered)
    /dev/mapper/main-home on /home type ext4 (rw,noatime,discard,stripe=32692,data=ordered)
    
    
    $ df -h /
    Filesystem             Size  Used Avail Use% Mounted on
    /dev/mapper/main-root   25G   16G  9.1G  63% /
    
    
    $ sudo snapper list
    Type   | #   | Pre # | Date                         | User | Cleanup  | Description | Userdata
    -------+-----+-------+------------------------------+------+----------+-------------+---------
    single | 0   |       |                              | root |          | current     |         
    single | 685 |       | Tue 13 Feb 2018 21:10:48 GMT | root |          |             |         
    single | 770 |       | Fri 02 Mar 2018 09:00:01 GMT | root | timeline | timeline    |         
    single | 771 |       | Fri 02 Mar 2018 10:00:01 GMT | root | timeline | timeline    |    
         
    
    $ sudo btrfs qgroup show -p /
    qgroupid         rfer         excl parent               
    --------         ----         ---- ------               
    0/5          16.00KiB     16.00KiB ---                  
    0/257         9.61GiB      5.06GiB ---                  
    0/258       184.82MiB    184.82MiB ---                  
    0/259        16.00KiB     16.00KiB ---                  
    0/260        16.00KiB     16.00KiB ---                  
    0/261         3.44MiB      3.44MiB ---                  
    0/272        16.00KiB     16.00KiB ---                  
    0/1017        9.71GiB     16.00KiB ---                  
    0/1107        9.71GiB      1.58MiB ---                  
    0/1108        9.71GiB    144.00KiB ---                  
    0/1109        9.71GiB     16.00KiB ---                  
    1/0           9.71GiB      1.83MiB 0/1107,0/1108,0/1109
    
    
    $ sudo btrfs subvolume list  /
    ID 257 gen 29387 top level 5 path @
    ID 258 gen 29300 top level 257 path @/opt
    ID 259 gen 29166 top level 257 path @/srv
    ID 260 gen 29332 top level 257 path @/boot/grub2/i386-pc
    ID 261 gen 29332 top level 257 path @/boot/grub2/x86_64-efi
    ID 272 gen 29392 top level 257 path @/.snapshots
    ID 1017 gen 29391 top level 272 path @/.snapshots/685/snapshot
    ID 1107 gen 29360 top level 272 path @/.snapshots/770/snapshot
    ID 1108 gen 29379 top level 272 path @/.snapshots/771/snapshot
    ID 1109 gen 29391 top level 272 path @/.snapshots/772/snapshot
    
    
    $ sudo btrfs sub get-default /
    ID 1017 gen 29391 top level 272 path @/.snapshots/685/snapshot
    The quotas say that I'm using <10GB, but df says I'm using 16GB. Subvolume 0/257 is my old "/", but 0/1017 is my new "/". 0/257 now shows >5GB exclusive data. I'd like to recover that space, but I didn't want to experiment and break anything! I thought I might be able to delete 0/257, but all of my subvolumes have it as their top level (directly or indirectly).

    Is there a correct way to safely recover that 5GB of space? I'm happy to assume that I'll never roll back to 0/257!

    Thanks.

    Note 1: I've got SSD and HDDs with partitions across them, hence the lack of some normal subvolumes.
    Note 2: I've manually deleted most of the snapshots to try to see where the disk usage is and save what space I can, hence the short list

  2. #2

    Default Re: btrfs, snapper, and recovering lost space after a rollback?

    I do not recommend that you follow these instructions - this is merely me exploring an option while understanding how btrfs works


    I've done some experimental changes and I think I have a safe-but-unapproved way to recover space.

    Check your disk usage for brtfs:
    Code:
    sudo btrfs qgroup show -p /
    This showed 5GB used by 0/257 as "excl" (i.e. no other subvolume shared the data)

    Mount the old subvolume (I used "/media" because nothing else was mounted there):
    Code:
    $ sudo mount -o subvolid=257 /dev/mapper/main-root /media
    Then find something that hasn't changed and delete it (e.g. Tango icons)
    Code:
    $ sudo rm -rf /media/usr/share/icons/Tango
    Check the quotas and it hasn't gone down, because it hasn't changed and so the blocks are shared rather than exclusive.

    So, what we need to do is find what changed. Someone made a btrfs-diff script, but it assumes that you haven't touched your old snapshot since it was taken (which isn't the case for me, because a) it's the old r/w root rather than a r/o snapshot and b) I've been poking it and deleting things, which updated its "generation"!)

    Assuming you've not added files to the old snapshot, run:
    Code:
    $ sudo btrfs subvolume find-new /media 0 | cut -f 14 -d" "|sort -nr|head -n1
    This gives the last transaction ID/generation that something was added to the old subvolume (mounted at "/media")

    Now we use that to find what changed in our current snapshot since then.

    First, find the current base snapshot:
    Code:
    sudo btrfs subvolume get-default / | sed 's/.*@//'
    Then use that path and the last transaction ID from earlier to find the files that have changed in the current snapshot since the last addition in the old snapshot, e.g.:
    Code:
    sudo btrfs subvolume find-new /.snapshots/774/snapshot 27955  | cut -f17 -d" " | sort | uniq
    (Field #17 is the file name, and we sort and uniq in case a file was updated multiple times)

    From that list, find a large file or directory of files that have changed. e.g. I had upgraded Visual Studio Code. Preferably something not critical to the system that you can reinstall if necessary, in case you get something wrong! Delete them from the old subvolume:
    Code:
    $ rm -rf /media/usr/share/code
    Now check the quotas again and the "excl" for 0/257 should have gone down.

    It should be possible to take all of the output from the "btrfs subvolume find-new" command and programmatically delete them from the old subvolume, but remember that a) these are new files in the newer snapshot so they could be updates/replacements (which will exist) or they could be new files (which won't exist) and b) scripting deletion from unknown output is dangerous!

    I do not recommend that you follow these instructions - this is merely me exploring an option while understanding how btrfs works

    There must be a better way to do this and free up the space from subvolume 0/257 when I've done a rollback and have a new default!

  3. #3
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    12,040
    Blog Entries
    2

    Default Re: btrfs, snapper, and recovering lost space after a rollback?

    Do you really store Visual Studio code on BTRFS?

    I wonder if you might have been able to recover space <safely> by simply removing snapshots using snapper.
    Snapper should be tried and/or used first always because by doing so you can ensure that anything you do would be safely done, eg merging when necessary.
    When you don't use snapper, you might have orphaned snapshots and possibly affected your snapper directory's integrity.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  4. #4
    Join Date
    Oct 2008
    Location
    Glasgow, Scotland
    Posts
    1,257

    Default Re: btrfs, snapper, and recovering lost space after a rollback?

    Quote Originally Posted by IBBoard View Post
    I'm running Tumbleweed with Snapper/snapshots. X.org broke one time and I used the snapshots to recover my system (and wait until the nVidia drivers were fixed). However, I now seem to have a lot of space missing on my main partition, ...
    What makes you think that there is “missing” space? I have limited vision and cannot follow all of your post, but it seems mostly irrelevant. The “df” command is not very applicable to Btrfs. Snapshots in Tumbleweed tend to be enormous because most of the root filesystem gets replaced on version upgrade, but you only have two.

    What output do you get from:
    Code:
     # btrfs fi show /
    
     # btrfs fi df /
    If there is a large difference between the “Used” values try:
    Code:
     # btrfs balance start /
    (this can take a while)
    then repeat the “btrfs fi” commands.
    Check out the Btrfs Wiki, e.g. https://btrfs.wiki.kernel.org/index...._lots_of_space
    --
    slàinte mhath,
    rayH

    ~ knowing the right answer is easier than knowing the right question.

  5. #5

    Default Re: btrfs, snapper, and recovering lost space after a rollback?

    Quote Originally Posted by tsu2 View Post
    Do you really store Visual Studio code on BTRFS?
    Yes. VS Code is an IDE. It installs as an RPM. So it does the Right Thing and puts all of its files under /usr.

    Quote Originally Posted by tsu2 View Post
    I wonder if you might have been able to recover space <safely> by simply removing snapshots using snapper.
    Snapper should be tried and/or used first always because by doing so you can ensure that anything you do would be safely done, eg merging when necessary.
    When you don't use snapper, you might have orphaned snapshots and possibly affected your snapper directory's integrity.

    TSU
    I already tried that, and it won't recover all of the space. Hence why I'm asking about how to recover the space that subvolume 0/257 is using now that I've got a different default subvolume due to a rollback.


    Quote Originally Posted by eng-int View Post
    What makes you think that there is “missing” space?
    "df" says I've used 16GB, the quotas say I'm using ~10GB on the live snapshot and ~5GB is "exclusive" to the old base 0/257 snapshot. That 5GB of exclusive space is lost, wasted space. I'm never going to go back to it (because I have a new default snapshot) but it is taking up space. Because it isn't a snapshot then Snapper can't clear it up.

    Quote Originally Posted by eng-int View Post
    I have limited vision and cannot follow all of your post, but it seems mostly irrelevant. The “df” command is not very applicable to Btrfs.
    The commands in the first post were to show my setup, where my system is using space, and what the current state of snapshots is on my machine. I've seen people berated for giving too little information, so I didn't want to describe my problem in the abstract.

    Quote Originally Posted by eng-int View Post
    Snapshots in Tumbleweed tend to be enormous because most of the root filesystem gets replaced on version upgrade, but you only have two.

    What output do you get from:
    Code:
     # btrfs fi show /
    
     # btrfs fi df /
    I only have two snapshots (which are small, according to the qgroups) because I've been deleting them to see how low I can get the disk space usage and whether I can get close to just the actual size used.

    Code:
    $ sudo btrfs fi show /
    Label: none  uuid: 84c68236-6fcb-4647-a744-18a738e57fa5
        Total devices 1 FS bytes used 15.14GiB
        devid    1 size 25.00GiB used 24.91GiB path /dev/mapper/main-root
    
    
    $ sudo btrfs fi df /
    Data, single: total=23.50GiB, used=14.53GiB
    System, single: total=32.00MiB, used=16.00KiB
    Metadata, single: total=1.38GiB, used=624.84MiB
    GlobalReserve, single: total=53.55MiB, used=0.00B
    So they're close but not perfect.

    Quote Originally Posted by eng-int View Post
    If there is a large difference between the “Used” values try:
    Code:
     # btrfs balance start /
    (this can take a while)
    then repeat the “btrfs fi” commands.
    Check out the Btrfs Wiki, e.g. https://btrfs.wiki.kernel.org/index...._lots_of_space
    That's not quite the problem that I have. The problem I have is that I actually run out of space because there are too many old files in old subvolumes. As you said, that is because of the big Tumbleweed updates. I want to do what I can to minimise that.

    I let Snapper auto-clean and I do "snapper delete" when I need to, but there is still ~5GB of space used by files that will never be used again because they're in the original "/" subvolume, which will never get mounted because I've done a rollback and now have a different default subvolume.

    As I understand it, a 10GB root partition with no other snapshots should take up 10GB on Btrfs. Mine isn't. I'm using ~16GB on Btrfs. I want to try and get it back down to 10GB so that I hit problems with disk usage less frequently.

    Thanks.

  6. #6
    Join Date
    Sep 2012
    Posts
    5,372

    Default Re: btrfs, snapper, and recovering lost space after a rollback?

    Quote Originally Posted by IBBoard View Post
    Subvolume 0/257 is my old "/"
    That seems to be the problem. Normally on current openSUSE if you enable snapshots during installation your root will be @/.snapshots/1/snapshot, i.e. the very first snapshot created, and @ subvolume will be nearly empty. When did you install your system? Did you chose to enable snapshots during installation?

    I do not think it is possible to move subvolumes, so what is left is carefully removing everything from @ except actual subvolumes (like @/.snapshot, @/.srv etc).

  7. #7
    Join Date
    Apr 2016
    Location
    North America
    Posts
    537

    Default Re: btrfs, snapper, and recovering lost space after a rollback?


  8. #8

    Default Re: btrfs, snapper, and recovering lost space after a rollback?

    Quote Originally Posted by arvidjaar View Post
    That seems to be the problem. Normally on current openSUSE if you enable snapshots during installation your root will be @/.snapshots/1/snapshot, i.e. the very first snapshot created, and @ subvolume will be nearly empty. When did you install your system? Did you chose to enable snapshots during installation?

    I do not think it is possible to move subvolumes, so what is left is carefully removing everything from @ except actual subvolumes (like @/.snapshot, @/.srv etc).
    Based on Zypper's history, it was installed back in late October. I think I enabled snapshots at install time (because I left it as Btrfs rather than the ext4 that I'm more used to so that I could use snapshots for recovery), but I do remember enabling quotas later.

    Maybe snapshots didn't get enabled because I used custom partitioning (putting /var and /tmp on a HDD instead of the SSD)?

    I'll see what I can delete manually and hopefully not lose something important!

    At least now that I've rolled back then I'm on a new default snapshot and the problem won't happen again in the future 🙂

    Thanks.

  9. #9

    Default Re: btrfs, snapper, and recovering lost space after a rollback?

    I ran:

    Code:
    sudo rm -rf /media/{bin,etc,lib,lib64,sbin,usr}
    sudo rm /media/boot/{b,c,i,s,S,v}*
    sudo rm -rf /media/boot/grub2/{backgrounds,fonts,grub.cfg,grubenv,locale,themes}
    Note: This was probably only safe on my system because some directories that are normally subvolumes are separate partitions! Understand what you're doing before you run this command!

    This has taken subvolume 0/257 down to under 1MB rfer and excl in the qgroup list.

    I now have a ~10GB root partition and one 300MB snapshot (plus btrfs metadata) taking up 11GB according to the main df command. That should leave me plenty of space for snapshots for a while. Hopefully there won't be issues down the line due to not having the correct setup initially (since my default subvolume is now snapshot 774, which looks like it mirrors the setup that I was supposed to have, but with a different snapshot number).

    Thanks!

  10. #10
    Join Date
    Nov 2009
    Location
    West Virginia Sector 13
    Posts
    15,816

    Default Re: btrfs, snapper, and recovering lost space after a rollback?

    Removing snapshots with normal Linux file commands may break snapper.

Page 1 of 2 12 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •