Howto use BTRFS compress?

English Install/Boot/Login
#1

Hi!

I install sda opensuse, sdc fedora.
openSuse not works compress (fedora works)

Howto compress zstd?


**# cat /etc/fstab **
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /                       btrfs  defaults                      0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /var                    btrfs  subvol=/@/var,compress=zstd                 0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /usr/local              btrfs  subvol=/@/usr/local,compress=zstd           0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /tmp                    btrfs  subvol=/@/tmp,compress=zstd                 0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /srv                    btrfs  subvol=/@/srv,compress=zstd                 0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /root                   btrfs  subvol=/@/root,compress=zstd                0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /opt                    btrfs  subvol=/@/opt,compress=zstd                 0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /home                   btrfs  subvol=/@/home,compress=zstd                0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /boot/grub2/x86_64-efi  btrfs  subvol=/@/boot/grub2/x86_64-efi  0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /boot/grub2/i386-pc     btrfs  subvol=/@/boot/grub2/i386-pc  0  0
UUID=A1A4-EBD5                             /boot/efi               vfat   defaults                      0  2
UUID=3b2c07f3-bb6b-4282-ab9f-a1705a11baf1  swap                    swap   defaults                      0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /.snapshots             btrfs  subvol=/@/.snapshots          0  0


**# dmesg | grep -i btrfs**
    3.608105] Btrfs loaded, crc32c=crc32c-intel, assert=on
    3.608586] BTRFS: device label fedora_localhost-live devid 1 transid 102676 /dev/sdc4
    3.608763] BTRFS: device fsid 0ad79315-b215-4f5c-b526-cf0187e60298 devid 1 transid 5156 /dev/sda2
    3.649744] BTRFS info (device sda2): disk space caching is enabled
    3.649745] BTRFS info (device sda2): has skinny extents
    9.646867] BTRFS info (device sda2): disk space caching is enabled
  578.373228] BTRFS info (device sdc4): disk space caching is enabled
  578.373229] BTRFS info (device sdc4): has skinny extents
  578.430712] BTRFS info (device sdc4): enabling ssd optimizations
  578.525204] BTRFS info (device sdc4): disk space caching is enabled
  578.525206] BTRFS info (device sdc4): has skinny extents
  578.555573] BTRFS info (device sdc4): enabling ssd optimizations
  685.141399] BTRFS info (device sda2): qgroup scan completed (inconsistency flag cleared)

**# mount | grep btrfs**
/dev/sda2 on / type btrfs (rw,relatime,space_cache,subvolid=268,subvol=/@/.snapshots/1/snapshot)
/dev/sda2 on /.snapshots type btrfs (rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots)
/dev/sda2 on /srv type btrfs (rw,relatime,space_cache,subvolid=261,subvol=/@/srv)
/dev/sda2 on /boot/grub2/i386-pc type btrfs (rw,relatime,space_cache,subvolid=266,subvol=/@/boot/grub2/i386-pc)
/dev/sda2 on /usr/local type btrfs (rw,relatime,space_cache,subvolid=259,subvol=/@/usr/local)
/dev/sda2 on /opt type btrfs (rw,relatime,space_cache,subvolid=263,subvol=/@/opt)
/dev/sda2 on /boot/grub2/x86_64-efi type btrfs (rw,relatime,space_cache,subvolid=265,subvol=/@/boot/grub2/x86_64-efi)
/dev/sda2 on /var type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/@/var)
/dev/sda2 on /home type btrfs (rw,relatime,space_cache,subvolid=264,subvol=/@/home)
/dev/sda2 on /root type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/@/root)
/dev/sda2 on /tmp type btrfs (rw,relatime,space_cache,subvolid=260,subvol=/@/tmp)

**# btrfs filesystem usage /**
Overall:
    Device size:         465.27GiB
    Device allocated:         104.02GiB
    Device unallocated:         361.25GiB
    Device missing:             0.00B
    Used:             101.25GiB
    Free (estimated):         361.99GiB    (min: 181.36GiB)
    Data ratio:                  1.00
    Metadata ratio:              2.00
    Global reserve:         158.41MiB    (used: 0.00B)


Data,single: Size:100.01GiB, Used:99.27GiB
   /dev/sda2     100.01GiB


Metadata,DUP: Size:2.00GiB, Used:1013.69MiB
   /dev/sda2       4.00GiB


System,DUP: Size:8.00MiB, Used:16.00KiB
   /dev/sda2      16.00MiB


Unallocated:
   /dev/sda2     361.25GiB



**# compsize /home**
Processed 339334 files, 276459 regular extents (277493 refs), 70843 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL      100%       78G          78G          78G       
none       100%       78G          78G          78G

openSUSE Support Database (strong deficiency)
https://en.opensuse.org/SDB:BTRFS

Thx help!

#2

The problem appears to be that the "compress= " option you’re using apparently works only on newly written files, existing files are not affected (Hint - This suggests that you may want to set this option in new installs, when you come to the system layout screen, don’t just accept defaults. Select the Guided Setup which allows you to customize the suggested disk layout. Alternative of course is to select the option to create your partitions and file systems from scratch without a suggestion).

Although also mentioned in the SDB BTRFS you referenced, this is more clearly stated in the ArchWiki BTRFS
https://wiki.archlinux.org/index.php/Btrfs

Only files created or modified after the mount option is added will be compressed.To apply compression to existing files, use the btrfs filesystem defragment -calg command, where alg is either zlib, lzo or zstd. For example, in order to re-compress the whole file system with [FONT=monospace]zstd, run the following command:
[/FONT]

btrfs filesystem defragment -r -v -czstd /

Note that for what you’re doing, you may also want to install compsize which might be a better way to report compression effectiveness.

Give the suggested command a try and see if it works (Remember, if you have anything really critical already, copy or back it up elsewhere in case the system is damaged beyond repair)

TSU

#3

tsu2:

Thanks for the detailed answer but, does not work.

Does not compress for new files and does not compress for defragment (this is shown by the compsize)
Sample


compsize /home
Processed 338427 files, 276212 regular extents (277235 refs), 71000 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL      100%       78G          78G          78G       
none       100%       78G          78G          78G
#4

Fedora32 add fstab “compress=zstd,space_cache=v2” and works


UUID=857546e0-fb88-40df-87e7-a7dd894e6b1b /                       btrfs   **compress=zstd,space_cache=v2**,subvol=root     0 0
UUID=63d6f9ee-2232-470f-bbfe-504b4b48a5ac /boot                   ext4    defaults        1 2
UUID=24D9-65B1          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
UUID=857546e0-fb88-40df-87e7-a7dd894e6b1b /home                   btrfs   **compress=zstd,space**_c**ache=v2**,subvol=home     0 0
UUID=da61eabf-f595-48ba-a6c6-421a433c19e8 none                    swap    defaults        0 0

compsize sample

# compsize /etc
Processed 1720 files, 632 regular extents (632 refs), 1261 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       39%       14M          35M          35M       
none       100%      6.7M         6.7M         6.7M       
zstd        25%      7.5M          29M          29M 

# compsize /usr/
Processed 309176 files, 178334 regular extents (188210 refs), 168447 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       59%      8.8G          14G          15G       
none       100%      5.9G         5.9G         6.0G       
zstd        32%      2.9G         8.9G         9.6G
#5

On a newly installed 15.2, I have nothing in my /home
I ran the compression command anyway, as expected it only found hidden files and determined they weren’t compressible for whatever reason.

Now, a WARNING
Do NOT do THE FOLLOWING UNTIL THERE ARE CERTAIN ANSWERS

I had success running the following command which compresses the root partition

btrfs filesystem defragment -r -v -czstd /

Took awhile, but the command ran through every individual file in the root partition.

Interestingly, I couldn’t run compsize against the root partition but was able to against any directory within the partition, volume or subvolume, so for instance i ran against /bin and it reported 80% compression.
I don’t know if this might apply to your attempt to compress /home, assuming /home is a partition, volume or subvolume and not a directory, try running compsize against your subdirectories in that partition and see if files are reported compressed.

Now, the problem.
When I rebooted, Grub dropped into a CLI and likely can’t read the compressed openSUSE files.
The ArchWiki BTRFS lists several possible causes, many which can be discarded immediately. My guess for the moment is that Grub needs to be modified to be modified to read files using the specified compression algorithm, and I suspect that it’s not possible to fix from the grub console… I’m guessing at the moment if I was really stuck in this situation in a production machine, I might have to try mounting in a chroot or using a LiveCD or a re-install elsewhere, make the modification and then copy the files to the system to be repaired.

In other words, my guess is that this problem should have been avoided by modifying grub before I rebooted, after the reboot it’s no longer a problem that can be easily addressed.

But, this is only a test system that can be discarded… I’m likely just going to throw it away and build another one.
This is one problem which is not going to be fixed by a snapper rollback.

TSU

#6

Thx all help!

Why is there no question during installation?
compression or not
(zstd, zlib, lzo)

:slight_smile:

#7

lzo compress works

**# cat /etc/fstab **
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /                       btrfs  defaults,**compress=lzo**                      0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /var                    btrfs  subvol=/@/var,**compress=lzo**                  0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /usr/local              btrfs  subvol=/@/usr/local,**compress=lzo**            0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /tmp                    btrfs  subvol=/@/tmp,**compress=lzo**                  0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /srv                    btrfs  subvol=/@/srv,**compress=lzo**                  0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /root                   btrfs  subvol=/@/root,**compress=lzo**                 0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /opt                    btrfs  subvol=/@/opt,**compress=lzo**                  0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /home                   btrfs  subvol=/@/home,**compress=lzo**                 0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /boot/grub2/x86_64-efi  btrfs  subvol=/@/boot/grub2/x86_64-efi  0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /boot/grub2/i386-pc     btrfs  subvol=/@/boot/grub2/i386-pc  0  0
UUID=A1A4-EBD5                             /boot/efi               vfat   defaults                      0  2
UUID=3b2c07f3-bb6b-4282-ab9f-a1705a11baf1  swap                    swap   defaults                      0  0
UUID=0ad79315-b215-4f5c-b526-cf0187e60298  /.snapshots             btrfs  subvol=/@/.snapshots,**compress=lzo**           0  0

**# dmesg | grep -i btrfs**
    3.657317] Btrfs loaded, crc32c=crc32c-intel, assert=on
    3.659239] BTRFS: device label fedora_localhost-live devid 1 transid 104013 /dev/sdc4
    3.729477] BTRFS: device fsid 0ad79315-b215-4f5c-b526-cf0187e60298 devid 1 transid 6148 /dev/sda2
    3.776297] BTRFS info (device sda2): disk space caching is enabled
    3.776298] BTRFS info (device sda2): has skinny extents
**   10.193013] BTRFS info (device sda2): use lzo compression, level 0**
   10.193014] BTRFS info (device sda2): disk space caching is enabled
  364.123263] BTRFS info (device sda2): qgroup scan completed (inconsistency flag cleared)
  633.053156] BTRFS info (device sda2): qgroup scan completed (inconsistency flag cleared)

**# mount | grep btrfs**
/dev/sda2 on / type btrfs (rw,relatime,**compress=lzo**,space_cache,subvolid=268,subvol=/@/.snapshots/1/snapshot)
/dev/sda2 on /.snapshots type btrfs (rw,relatime,**compress=lzo**,space_cache,subvolid=267,subvol=/@/.snapshots)
/dev/sda2 on /home type btrfs (rw,relatime,**compress=lzo**,space_cache,subvolid=264,subvol=/@/home)
/dev/sda2 on /var type btrfs (rw,relatime,**compress=lzo**,space_cache,subvolid=258,subvol=/@/var)
/dev/sda2 on /opt type btrfs (rw,relatime,**compress=lzo**,space_cache,subvolid=263,subvol=/@/opt)
/dev/sda2 on /tmp type btrfs (rw,relatime,**compress=lzo**,space_cache,subvolid=260,subvol=/@/tmp)
/dev/sda2 on /usr/local type btrfs (rw,relatime,**compress=lzo**,space_cache,subvolid=259,subvol=/@/usr/local)
/dev/sda2 on /boot/grub2/i386-pc type btrfs (rw,relatime,**compress=lzo**,space_cache,subvolid=266,subvol=/@/boot/grub2/i386-pc)
/dev/sda2 on /boot/grub2/x86_64-efi type btrfs (rw,relatime,**compress=lzo**,space_cache,subvolid=265,subvol=/@/boot/grub2/x86_64-efi)
/dev/sda2 on /srv type btrfs (rw,relatime,**compress=lzo**,space_cache,subvolid=261,subvol=/@/srv)
/dev/sda2 on /root type btrfs (rw,relatime,**compress=lzo**,space_cache,subvolid=262,subvol=/@/root)

**But, not work (compsize zero)
**


/var
/usr/local
/srv
/opt

Realy works:

**# compsize /tmp**
Processed 42 files, 10 regular extents (10 refs), 28 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       99%      1.9M         1.9M         1.9M       
none       100%      1.9M         1.9M         1.9M       
lzo         58%      874B         1.4K         1.4K      

**# compsize /root**
Processed 20 files, 8 regular extents (8 refs), 12 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       99%      281K         281K         277K       
none       100%      280K         280K         276K       
lzo         92%      787B         855B         855B   

**# compsize /home**
Processed 340773 files, 279286 regular extents (280324 refs), 71118 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       99%       78G          79G          78G       
none       100%       78G          78G          78G       
lzo         54%       27M          51M          50M       


**# compsize /.snapshots**
Processed 5863895 files, 406459 regular extents (3389220 refs), 3079614 inline.
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       76%       23G          30G         219G       
none       100%       17G          17G          93G       
lzo         54%      297M         543M         2.8G       
zstd        41%      4.8G          11G         123G
#8

A lot of BTRFS bells and whistles are somewhat experiential so maybe the installer writers decided not to explicitly include the option??? Or maybe they just missed it??

#9

gogalthorp

Of course, the function is missing. A single OS that is actually btrfs installed by default.
And the default is “Snapper”, is set is configured.

#10

I’m not looking at an installation right now, but I seem to remember that compression and encryption are both offered as checkbox options later if you choose the Guided or Expert options on the Layout screen. Of course checkboxes only enable a default and does not allow you to specify the particular algorithm.

But regardless of those checkboxes, I do know that for every object in the layout you have the ability to type whatever you what to specify ad hoc options.
So, you do have the ability to specify a “compress=zstd” or “compress-force=zstd” for example (I haven’t actually run these specific options in an install but see no reason for them not to work).

TSU

#11

A small FYI to this thread…

During a test install,
I verified that during initial installation,
Whether you use Guided Setup or not,
If you then select “Expert Partitioner” you are prsented a checkbox only to encrypt and not to compress.
You can use the Partitioner to set ad hoc “Options” only to the partition, and when I compressed the root partition, to no surprise the system was unbootable for the same reason I gave in my previous post… Grub needs to be configured to read the compression algorithm used to compress your root partition.
Since the Linux system directories have not yet been created at this stage of the Installation, it’s not possible to configure compression for individual directories.

Also,
With openSUSE 15.1 and 15.2 /home is configured as its own BTRFS subvolume and I did not find a way to set compression only for that subvolume (although I may have missed it).

So,
Bottom line is that until configuring grub to read compressed volums has been tested, I’d recommend not compressing any partitions or volumes during installation.

TSU

#12

grub 2.04 supports btrfs compression.