Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: Btrfs: Extremely slow delete of files

  1. #1
    Join Date
    Jan 2014
    Location
    Erlangen
    Posts
    1,314

    Default Btrfs: Extremely slow delete of files

    The following compares performance of 2 partitions sitting on a Samsung 850 EVO 500GB:

    Code:
    fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k  --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based  --end_fsync=1
    
     ext4:
    
    WRITE: bw=340MiB/s (356MB/s), 340MiB/s-340MiB/s (356MB/s-356MB/s), io=20.0GiB (21.5GB), run=60259-60259msec
    
    time rm random-write.0.0: real    0m0.004s, user    0m0.001s, sys     0m0.004s
    
     btrfs:
    
    WRITE: bw=153MiB/s (161MB/s), 153MiB/s-153MiB/s (161MB/s-161MB/s), io=9332MiB (9786MB), run=60849-60849msec
    
    time rm random-write.0.0: real    0m11.166s, user    0m0.001s, sys     0m11.126s
    Deletion of a file is 2,700 times slower with btrfs! Any idea?
    AMD Athlon 4850e (2009), openSUSE 13.1, KDE 4, Intel i3-4130 (2014), i7-6700K (2016), i5-8250U (2018), openSUSE Tumbleweed, KDE Plasma 5

  2. #2
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    2,869

    Default Re: Btrfs: Extremely slow delete of files

    Quote Originally Posted by karlmistelberger View Post
    Any idea?
    Btrfs housekeeping …
    • Btrfs is fine and good but, if the housekeeping (balance and scrub) isn't regularly executed, it may become slow.

  3. #3
    Join Date
    Jun 2008
    Location
    Groningen, Netherlands
    Posts
    20,467
    Blog Entries
    14

    Default Re: Btrfs: Extremely slow delete of files

    Quote Originally Posted by dcurtisfra View Post
    Btrfs housekeeping …
    • Btrfs is fine and good but, if the housekeeping (balance and scrub) isn't regularly executed, it may become slow.
    If the admin doesn't tinker with the settings, on a default setup the system takes care of that fine.
    ° Appreciate my reply? Click the star and let me know why.

    ° Perfection is not gonna happen. No way.

    http://en.opensuse.org/User:Knurpht
    http://nl.opensuse.org/Gebruiker:Knurpht

  4. #4
    Join Date
    Jan 2014
    Location
    Erlangen
    Posts
    1,314

    Default Re: Btrfs: Extremely slow delete of files

    Quote Originally Posted by Knurpht View Post
    If the admin doesn't tinker with the settings, on a default setup the system takes care of that fine.
    I didn't tinker. I forgot about COW. When I issued xattr +C /root and reran fio, I got more reasonable results:
    Code:
    erlangen:~ # fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1
    random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=1
    fio-3.16
    Starting 1 process
    random-write: Laying out IO file (1 file / 4096MiB)
    Jobs: 1 (f=1): [w(1)][100.0%][w=227MiB/s][w=58.1k IOPS][eta 00m:00s]
    random-write: (groupid=0, jobs=1): err= 0: pid=15499: Tue Feb 11 14:09:24 2020
      write: IOPS=53.1k, BW=207MiB/s (218MB/s)(12.2GiB/60450msec); 0 zone resets
        slat (nsec): min=267, max=202829, avg=853.17, stdev=807.54
        clat (nsec): min=124, max=139405k, avg=16615.77, stdev=287021.52
         lat (usec): min=4, max=139405, avg=17.47, stdev=287.02
        clat percentiles (usec):
         |  1.00th=[    6],  5.00th=[    6], 10.00th=[    7], 20.00th=[    8],
         | 30.00th=[    8], 40.00th=[    9], 50.00th=[   10], 60.00th=[   11],
         | 70.00th=[   11], 80.00th=[   12], 90.00th=[   23], 95.00th=[   71],
         | 99.00th=[  118], 99.50th=[  131], 99.90th=[  161], 99.95th=[  180],
         | 99.99th=[  469]
       bw (  KiB/s): min=90387, max=357960, per=100.00%, avg=217505.79, stdev=39424.47, samples=117
       iops        : min=22596, max=89490, avg=54376.43, stdev=9856.15, samples=117
      lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
      lat (usec)   : 2=0.01%, 4=0.03%, 10=57.53%, 20=31.20%, 50=4.37%
      lat (usec)   : 100=4.76%, 250=2.10%, 500=0.01%, 750=0.01%, 1000=0.01%
      lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
      lat (msec)   : 100=0.01%, 250=0.01%
      cpu          : usr=7.70%, sys=10.85%, ctx=3290070, majf=0, minf=24
      IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         issued rwts: total=0,3209977,0,1 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=1
    
    Run status group 0 (all jobs):
      WRITE: bw=207MiB/s (218MB/s), 207MiB/s-207MiB/s (218MB/s-218MB/s), io=12.2GiB (13.1GB), run=60450-60450msec
    erlangen:~ # filefrag random-write.0.0 
    random-write.0.0: 5 extents found
    erlangen:~ # time rm random-write.0.0 
    
    real    0m0.113s
    user    0m0.000s
    sys     0m0.113s
    erlangen:~ #
    With COW enabled fio would generate some 1 million extents.
    AMD Athlon 4850e (2009), openSUSE 13.1, KDE 4, Intel i3-4130 (2014), i7-6700K (2016), i5-8250U (2018), openSUSE Tumbleweed, KDE Plasma 5

  5. #5
    Join Date
    Jun 2008
    Location
    Groningen, Netherlands
    Posts
    20,467
    Blog Entries
    14

    Default Re: Btrfs: Extremely slow delete of files

    @Karl: my reply was re. dcurtis' one.
    ° Appreciate my reply? Click the star and let me know why.

    ° Perfection is not gonna happen. No way.

    http://en.opensuse.org/User:Knurpht
    http://nl.opensuse.org/Gebruiker:Knurpht

  6. #6
    Join Date
    Jan 2014
    Location
    Erlangen
    Posts
    1,314

    Default Re: Btrfs: Extremely slow delete of files

    Quote Originally Posted by Knurpht View Post
    @Karl: my reply was re. dcurtis' one.
    I am angry, because I failed to check attributes. I assumed folders excluded from snapshots would have COW disabled. But this is true only for /var. I added /tmp and /root.
    AMD Athlon 4850e (2009), openSUSE 13.1, KDE 4, Intel i3-4130 (2014), i7-6700K (2016), i5-8250U (2018), openSUSE Tumbleweed, KDE Plasma 5

  7. #7
    Join Date
    Jun 2008
    Location
    Groningen, Netherlands
    Posts
    20,467
    Blog Entries
    14

    Default Re: Btrfs: Extremely slow delete of files

    Quote Originally Posted by karlmistelberger View Post
    I am angry, because I failed to check attributes. I assumed folders excluded from snapshots would have COW disabled. But this is true only for /var. I added /tmp and /root.
    Apparently we experienced the fact that "Anger is the worst consultant" on the same day. I was struggling with what I thought was a btrfs/snapper issue, to find out after a break ( walk through yesterdays storm ) to find out I had not rolled back to snapshot 1560, but to 1568 ....... . After finding out and doing the right thing, all was OK again.
    ° Appreciate my reply? Click the star and let me know why.

    ° Perfection is not gonna happen. No way.

    http://en.opensuse.org/User:Knurpht
    http://nl.opensuse.org/Gebruiker:Knurpht

  8. #8
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    2,869

    Wink Re: Btrfs: Extremely slow delete of files

    Quote Originally Posted by karlmistelberger View Post
    I forgot about COW. When I issued xattr +C /root and reran fio, I got more reasonable results:
    Karl,

    fine, but, for Btrfs, “chattr +C” should only be executed on new or empty files …
    <https://wiki.archlinux.org/index.php...he_file_system> – ArchWiki hint:
    Code:
    $ mv /path/to/dir /path/to/dir_old
    $ mkdir /path/to/dir
    $ chattr +C /path/to/dir
    $ cp -a /path/to/dir_old/* /path/to/dir
    $ rm -rf /path/to/dir_old
    I won't bother quoting the “chattr” man page note related to Btrfs COW …

    Best regards
    DCu

  9. #9
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    12,069
    Blog Entries
    2

    Default Re: Btrfs: Extremely slow delete of files

    CoW is popular for virtual machine disks and a few other scenarios.
    In theory, it should likely decrease latency by allowing processes to move ahead without waiting for disk writes to complete synchronously.

    Of course every situation has its requirements
    But for ordinary use I'd generally recommend CoW.
    Of course if you're slamming the disk running benchmarks, the actual times will be slower (because of the support for lazy writes) but that's looking only at the disk subsystem, and unless you absolutely need that data written to disk in the real world, the test isn't reflective of best system optimization practice.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  10. #10
    Join Date
    Jan 2014
    Location
    Erlangen
    Posts
    1,314

    Default Re: Btrfs: Extremely slow delete of files

    Quote Originally Posted by dcurtisfra View Post
    Karl, fine, but, for Btrfs, “chattr +C” should only be executed on new or empty files …
    I am aware of this. I apply the command to the root folder of the subvolume, which renders its tree as if mounted with option nodatacow.
    AMD Athlon 4850e (2009), openSUSE 13.1, KDE 4, Intel i3-4130 (2014), i7-6700K (2016), i5-8250U (2018), openSUSE Tumbleweed, KDE Plasma 5

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •