Long story short:
- Had 2 870QVO, they are worse than you think
- Setup OpenSuSE Leap 15.3, 2 md arrays, 1 btrfs RAID1 array
- 1st md array BIOS boot, 2nd array swap
- Performance gets worse on 870QVO, decide to swap to 2 smaller MX500 2TB
- Add two new MX500 SSDs, “sfdisk -d /dev/sdb”, adjust lba and size for smaller drive, load config via fdisk, write
- Add sdc1 and sdc2 to respective md arrays, replace the old partition with new partition, remove old partition
- Run btrfs shrink, btrfs replace /dev/sdb3 with /dev/sdc3, finishes, reboot, everything is fine and everything is mounted right
- Repeat last 3 steps for the /dev/sda with another new ssd. md aray synced, btrfs data moved, server freezes, force reboot
- Not booting into the system, emtpy screen. Via usb stick and “Boot Linux System” I can boot into the system where we are now.
“sfdisk -d /dev/sdb”
label: gpt
label-id: F904AC8D-687A-4A8F-9A27-7DB398BD0BED
device: /dev/sdb
unit: sectors
first-lba: 34
last-lba: 7814037134
sector-size: 512
/dev/sdb1 : start= 2048, size= 20480, type=A19D880F-05FC-4D3B-A006-743F0F84911E, uuid=4F6C8BD4-4D80-418C-A21B-A7D90D612ABD
/dev/sdb2 : start= 22528, size= 4214784, type=A19D880F-05FC-4D3B-A006-743F0F84911E, uuid=74CAAF94-99B9-4417-B596-F3FAAA1B5149
/dev/sdb3 : start= 4237312, size= 7809799823, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, uuid=4A0FA483-33E1-4208-8F86-8BD8E89A0A77
Adjusted “sfdisk -d /dev/sdb” for smaller drive
label: gpt
label-id: F904AC8D-687A-4A8F-9A27-7DB398BD0BED
device: /dev/sdb
unit: sectors
first-lba: 34
last-lba: 3907029134
sector-size: 512
/dev/sdb1 : start= 2048, size= 20480, type=A19D880F-05FC-4D3B-A006-743F0F84911E, uuid=4F6C8BD4-4D80-418C-A21B-A7D90D612ABD
/dev/sdb2 : start= 22528, size= 4214784, type=A19D880F-05FC-4D3B-A006-743F0F84911E, uuid=74CAAF94-99B9-4417-B596-F3FAAA1B5149
/dev/sdb3 : start= 4237312, size= 3902791823, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, uuid=4A0FA483-33E1-4208-8F86-8BD8E89A0A77
“sfdisk -d /dev/sda”
label: gpt
label-id: F9E1C06E-AC29-4061-B2D4-FB1A5EDE0471
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 7814037134
sector-size: 512
/dev/sda1 : start= 2048, size= 20480, type=A19D880F-05FC-4D3B-A006-743F0F84911E, uuid=E1022C8C-4B4E-4262-BBB0-2E204F7BBBCC
/dev/sda2 : start= 22528, size= 4214784, type=A19D880F-05FC-4D3B-A006-743F0F84911E, uuid=8D2F378F-1D50-4D2F-AE1B-78DED1D07F7B
/dev/sda3 : start= 4237312, size= 7809799823, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, uuid=15871739-A3D2-4F86-B1B3-233683F2A532, attrs="LegacyBIOSBootable"
Adjusted “sfdisk -d /dev/sda” for smaller drive
label: gpt
label-id: F9E1C06E-AC29-4061-B2D4-FB1A5EDE0471
device: /dev/sda
unit: sectors
first-lba: 34
last-lba: 3907029134
sector-size: 512
/dev/sda1 : start= 2048, size= 20480, type=A19D880F-05FC-4D3B-A006-743F0F84911E, uuid=E1022C8C-4B4E-4262-BBB0-2E204F7BBBCC
/dev/sda2 : start= 22528, size= 4214784, type=A19D880F-05FC-4D3B-A006-743F0F84911E, uuid=8D2F378F-1D50-4D2F-AE1B-78DED1D07F7B
/dev/sda3 : start= 4237312, size= 3902791823, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, uuid=15871739-A3D2-4F86-B1B3-233683F2A532, attrs="LegacyBIOSBootable"
The new “sda”
fdisk -l /dev/sdaj
Disk /dev/sdaj: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: CT2000MX500SSD1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: F9E1C06E-AC29-4061-B2D4-FB1A5EDE0471
Device Start End Sectors Size Type
/dev/sdaj1 2048 22527 20480 10M Linux RAID
/dev/sdaj2 22528 4237311 4214784 2G Linux RAID
/dev/sdaj3 4237312 3907029134 3902791823 1.8T Linux filesystem
The new “sdb”
fdisk -l /dev/sdak
Disk /dev/sdak: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: CT2000MX500SSD1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: F904AC8D-687A-4A8F-9A27-7DB398BD0BED
Device Start End Sectors Size Type
/dev/sdak1 2048 22527 20480 10M Linux RAID
/dev/sdak2 22528 4237311 4214784 2G Linux RAID
/dev/sdak3 4237312 3907029134 3902791823 1.8T Linux filesystem
“lsblk -f” of both partitions
sdaj
├─sdaj1 linux_raid_member 1.0 any:boot 53eb9b5b-e2fd-2055-fce3-fec47d14f4f4
│ └─md126
│ └─md126p1
├─sdaj2 linux_raid_member 1.0 any:swap 31fa9784-bb4f-faba-154c-aba28d8af0f3
│ └─md127 swap 1 bed2b849-7c5a-4b4f-b202-d91c428ddbda
└─sdaj3 btrfs 56a0a09b-5e35-465c-bd6d-1fb79145de0f 101.6G 88% /
sdak
├─sdak1 linux_raid_member 1.0 any:boot 53eb9b5b-e2fd-2055-fce3-fec47d14f4f4
│ └─md126
│ └─md126p1
├─sdak2 linux_raid_member 1.0 any:swap 31fa9784-bb4f-faba-154c-aba28d8af0f3
│ └─md127 swap 1 bed2b849-7c5a-4b4f-b202-d91c428ddbda
└─sdak3 btrfs 56a0a09b-5e35-465c-bd6d-1fb79145de0f
“BIOS boot” md array
fdisk -l /dev/md126
Disk /dev/md126: 9.94 MiB, 10420224 bytes, 20352 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 7D8B04BD-C682-4DC1-8B8A-E807F3B6A73C
Device Start End Sectors Size Type
/dev/md126p1 2048 20062 18015 8.8M BIOS boot
fstab
cat /etc/fstab
UUID=56a0a09b-5e35-465c-bd6d-1fb79145de0f / btrfs defaults 0 0
UUID=56a0a09b-5e35-465c-bd6d-1fb79145de0f /var btrfs subvol=/@/var 0 0
UUID=56a0a09b-5e35-465c-bd6d-1fb79145de0f /usr/local btrfs subvol=/@/usr/local 0 0
UUID=56a0a09b-5e35-465c-bd6d-1fb79145de0f /tmp btrfs subvol=/@/tmp 0 0
UUID=56a0a09b-5e35-465c-bd6d-1fb79145de0f /srv btrfs subvol=/@/srv 0 0
UUID=56a0a09b-5e35-465c-bd6d-1fb79145de0f /root btrfs subvol=/@/root 0 0
UUID=56a0a09b-5e35-465c-bd6d-1fb79145de0f /opt btrfs subvol=/@/opt 0 0
UUID=56a0a09b-5e35-465c-bd6d-1fb79145de0f /home btrfs subvol=/@/home 0 0
UUID=56a0a09b-5e35-465c-bd6d-1fb79145de0f /boot/grub2/x86_64-efi btrfs subvol=/@/boot/grub2/x86_64-efi 0 0
UUID=56a0a09b-5e35-465c-bd6d-1fb79145de0f /boot/grub2/i386-pc btrfs subvol=/@/boot/grub2/i386-pc 0 0
UUID=56a0a09b-5e35-465c-bd6d-1fb79145de0f /.snapshots btrfs subvol=/@/.snapshots 0 0
History of all actions taken:
mdadm /dev/md126 --add /dev/sdac1
mdadm /dev/md127 --add /dev/sdac2
mdadm /dev/md126 --replace /dev/sdb1 --with /dev/sdac1
mdadm /dev/md127 --replace /dev/sdb2 --with /dev/sdac2
mdadm /dev/md126 --remove /dev/sdb1
mdadm /dev/md127 --remove /dev/sdb2
btrfs filesystem resize 2:-2900G /
btrfs replace start 2 /dev/sdac3 /
reboot
mdadm /dev/md126 --add /dev/sdab1
mdadm /dev/md127 --add /dev/sdab2
mdadm /dev/md126 --replace /dev/sda1 --with /dev/sdab1
mdadm /dev/md127 --replace /dev/sda2 --with /dev/sdab2
mdadm /dev/md126 --remove /dev/sda1
mdadm /dev/md127 --remove /dev/sda2
btrfs filesystem resize 1:-2900G /
btrfs replace start 1 /dev/sdab3 /
"system freeze"
"hard reboot"
"black screen"
Old “/etc/default/grub_installdevice” had the following content:
/dev/disk/by-id/ata-Samsung_SSD_870_QVO_4TB_S5STNJ0R500056K-part3
activate
generic_mbr
Got the system back running via “Boot Linux System”, but I never really worked with mdadm (mainly hw raid) so I’m kinda confused what to poke right now.
Going by the docs on booting problems it wants me to poke yast > boot laoder. Which naturally has problems, since it can’t find the old >>>disk<<< that was removed.
Old grub_installdevice had the following entries “/dev/disk/by-id/ata-Samsung_SSD_870_QVO_4TB_S5STNJ0R500056K-part3”, “activate” and “generic_mbr”
I’m quite rusty on bootloaders and md raids, so… kinda need help to figure out how to get it to normally boot without destroying my disks/partitions/data.
The “yast > boot laoder” has the options of Boot Code Location. But I don’t know how they do it, since it seems that the old setup had only one disk function as boot device(don’t remember, sorry).
“Write to MBR” sounds wrong, since it’s a GPT formatted disk and the “MBR” should actually be a “protected MBR”. “Writing to Partition” will it work with my setup?
I’m pretty sure I’ve seen the old(before disk migration) bootloader config having a “custom boot partition” set, but I can’T really remember.
This is it, if you have a solutin or more questions, do tell. I’m happy for any insight, since I’m unable to figure out what is the proper step with this setup.
Be it a simpler setup, ezpz. But this one messes with my braincells.
https://i.imgur.com/7E41pvT.png
PS: SWAP was removed, so if the easiest solution is to have one didcated clean partition without bells and whistles, then it can be done. The only important part is data, and that can be booted from “Boot Linux System” in case of failure.