Hey everyone 
I was wondering if it was possible to pin metadata to faster SSD drives in a btrfs RAID1 pool with data on HDDs? 
My pool is quite unusual and is made up of 4+2+1+1 TB external HDDs as of now.
Lately though I’ve been getting write errors with the smaller drives being too slow and getting bogged down, timing out on write commands. Just to make sure it wasn’t an enclosure or HDD problem I tried swapping both with new ones and it didn’t make it any better.
Aug 03 17:18:14 suse-pc kernel: sd 5:0:0:0: [sdd] tag#0 timing out command, waited 180s
Aug 03 17:18:14 suse-pc kernel: I/O error, dev sdd, sector 324666880 op 0x1:(WRITE) flags 0x100000 phys_seg 64 prio class 2
Layout and usage:
pavin@suse-pc:~> sudo btrfs filesystem show /usbraid/
Please enter the PIN:
Please touch the device.
Label: 'usbraid' uuid: 378f24e8-6537-4a27-b827-b133f902b3f4
Total devices 4 FS bytes used 1.39TiB
devid 1 size 1.82TiB used 1.08TiB path /dev/mapper/wd2t
devid 4 size 3.64TiB used 1.41TiB path /dev/mapper/seagate4t
devid 5 size 931.50GiB used 173.03GiB path /dev/mapper/seagate_red
devid 6 size 931.50GiB used 165.03GiB path /dev/mapper/seagate_black
pavin@suse-pc:~>
pavin@suse-pc:~>
pavin@suse-pc:~> sudo btrfs filesystem usage -T /usbraid/
Overall:
Device size: 7.28TiB
Device allocated: 2.82TiB
Device unallocated: 4.46TiB
Device missing: 0.00B
Device slack: 0.00B
Used: 2.78TiB
Free (estimated): 2.25TiB (min: 2.25TiB)
Free (statfs, df): 1.51TiB
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no
Data Metadata System
Id Path RAID1 RAID1 RAID1 Unallocated Total Slack
-- ------------------------- --------- -------- --------- ----------- --------- -----
1 /dev/mapper/wd2t 1.08TiB 1.00GiB - 757.97GiB 1.82TiB -
4 /dev/mapper/seagate4t 1.41TiB 2.00GiB 64.00MiB 2.23TiB 3.64TiB -
5 /dev/mapper/seagate_red 173.00GiB - 32.00MiB 758.46GiB 931.50GiB -
6 /dev/mapper/seagate_black 164.00GiB 1.00GiB 32.00MiB 766.46GiB 931.50GiB -
-- ------------------------- --------- -------- --------- ----------- --------- -----
Total 1.41TiB 2.00GiB 64.00MiB 4.46TiB 7.28TiB 0.00B
Used 1.39TiB 1.49GiB 240.00KiB
pavin@suse-pc:~>
I have ordered two additional SSDs to act as a cache, planning to use it with bcachefs, but kind of apprehensive after reading this.
Bcachefs also doesn’t support scrub currently and I would rather try to make do with btrfs if possible. Searching the web provides conflicting information with github issues for adding this feature and Synology website saying they support it. IIRC they use btrfs with bcache as a caching layer. I’m not too fond of this idea either as my recent device replace/rebuild after moving to a new drive/enclosure for troubleshooting took 3 whole days! 
Looking at iostat while it was running it seems to have been caused by metadata update delays.
P.S. I have run btrfs check and there were no errors found, but I do get correctable errors during scrub after encountering write errors (which is common now unfortunately). Also no SMART errors on any drive.
. Bcachefs doesn’t currently support full filesystem rebalance or scrub though patches in Linux kernel 6.11 should fix errors on the fly when reading.