BTRFS replace not eliminating failing device, system is still looking for it after reboot

[lavadrop]

I have a 2 disk btrfs RAID1.
One disk is sda/cr_ata-ST8000VN004-3CP101_WWZ7KJ9J
The second disk is sdb/raid2
The third and damaged one is sde/raid1
I used yast partitioner to partition the new drive raw but with encryption, both disk in the RAID1 are encrypted.
I followed this:
https://archive.kernel.org/oldwiki/btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices.html#Replacing_failed_devices
After inputting the sudo btrfs replace start 1 /dev/mapper/cr_ata-ST8000VN004-3CP101_WWZ7KJ9J /mnt/raid
btrfs complains there is a partition already and to use the -f flag to overwrite. And so I did. It finished witouth errors but I had to rebalance to convert chunks to RAID1. That went well too.
I unmounted and remounted, no problem. I edited my crypttab to change the UUID of the old drive to the new drive. Fstab has the btrfs UUID that didn’t change.
Now, when I reboot, the system idles waiting for the OLD drive by UUID to mount. So I put it back online via dock to boot, and now the system complains it can’t mount, so I continue and now after putting sudo btrfs filesystem show

WARNING: adding device /dev/mapper/raid1 gen 43963 but found an existing device /dev/mapper/cr_ata-ST8000VN004-3CP101_WWZ7KJ9J gen 44318
ERROR: cannot scan /dev/mapper/raid1: File exists

What did I do wrong or how can I replace my failing hard drive but also keep encryption and preferably keeping the name “raid1” instead of “cr_ata-ST8000VN004-3CP101_WWZ7KJ9J” ?

[arvidjaar]

@lavadrop

sudo btrfs replace start 1 /dev/mapper/cr_ata-ST8000VN004-3CP101_WWZ7KJ9J /mnt/raid

We have no information about your filesystem, which devices belong(ed) to this filesystem, so we have no idea what device 1 refers to.

We have no information about the state of the device 1 when this command was running so we have no way to guess what could possibly happen.

@lavadrop

the system idles waiting for the OLD drive by UUID to mount.

What does it mean? What is OLD, what is NEW, what message do you see?

@lavadrop

What did I do wrong

It is impossible to even guess because you did not really show us what you did. Taking a wild guess based on

@lavadrop

btrfs complains there is a partition already and to use the -f flag to overwrite.

before replacement you configured the drive /dev/mapper/cr_ata-ST8000VN004-3CP101_WWZ7KJ9J as a filesystem and now on boot this filesystem is missing.

Anyway - on boot system does whatever is defined in /etc/fstab, so edit it and remove whatever line is incorrect (presumably has OLD UUID).

[argentwolf]

The following might be of some help.

Welcome to BTRFS documentation!

[argentwolf]

Setting Up BTRFS RAID

[hui]

@arvidjaar

And why are you answering me?

It is a basic forum function (or flaw) that there are two answer buttons at the end of a thread.

You need to raise a ticket with Discourse if you don‘t like their design.

[argentwolf]

With the length of your initial post, I knew my opine on your issue(s) would be taken out of context, so I thought resources on the subject might help. My bad! I still believe others out there have addressed your problem/confusion much better than I could. Good Luck!

[lavadrop]

Ok, this is my system:

lsblk
NAME                                                   MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                                                      8:0    0   7.3T  0 disk  
└─cr_ata-ST8000VN004-3CP101_WWZ7KJ9J                   254:6    0   7.3T  0 crypt /mnt/raid
sdb                                                      8:16   0   3.6T  0 disk  
└─raid2                                                254:3    0   3.6T  0 crypt 
sdc                                                      8:32   0 232.9G  0 disk  
└─sdc1                                                   8:33   0 232.9G  0 part  
sdd                                                      8:48   0   7.3T  0 disk  /mnt/backup
sde                                                      8:64   0   3.6T  0 disk  
└─raid1                                                254:4    0   3.6T  0 crypt 
sr0                                                     11:0    1  1024M  0 rom   
nvme0n1                                                259:0    0 465.8G  0 disk  
├─nvme0n1p1                                            259:1    0   512M  0 part  /boot/efi
└─nvme0n1p2                                            259:2    0 465.3G  0 part  
  └─cr_nvme-eui.00000000000000016479a7716f0000f7-part2 254:0    0 465.3G  0 crypt 
    ├─system-root                                      254:1    0 159.9G  0 lvm   /var
    │                                                                             /opt
    │                                                                             /root
    │                                                                             /srv
    │                                                                             /usr/local
    │                                                                             /boot/grub2/x86_64-efi
    │                                                                             /boot/grub2/i386-pc
    │                                                                             /.snapshots
    │                                                                             /
    ├─system-swap                                      254:2    0  15.4G  0 lvm   [SWAP]
    └─system-home                                      254:5    0 289.9G  0 lvm   /home

sde/raid1 is the OLD DAMAGED hard drive, sda is the brand new drive I instructed btrfs to replace in the array, sdb/raid2 is the mirrored drive.

This is the btrfs filesystem:

sudo btrfs filesystem show
Label: none  uuid: 4114a69e-5b10-4537-ab86-60773064a389
	Total devices 1 FS bytes used 66.48GiB
	devid    1 size 159.95GiB used 75.07GiB path /dev/mapper/system-root

Label: 'BorgBackup'  uuid: b5dc3d44-c92e-4805-941f-f6ac8aa74dab
	Total devices 1 FS bytes used 1.26TiB
	devid    1 size 7.28TiB used 1.29TiB path /dev/sdd

Label: none  uuid: 3a5fa0a8-1e39-4c4c-ab16-064f211e5c03
	Total devices 2 FS bytes used 1.31TiB
	devid    1 size 7.28TiB used 1.32TiB path /dev/mapper/cr_ata-ST8000VN004-3CP101_WWZ7KJ9J
	devid    2 size 3.64TiB used 1.32TiB path /dev/mapper/raid2
WARNING: adding device /dev/mapper/raid1 gen 43963 but found an existing device /dev/mapper/cr_ata-ST8000VN004-3CP101_WWZ7KJ9J gen 44318
ERROR: cannot scan /dev/mapper/raid1: File exists

The UUID in fstab is 3a5fa0a8-1e39-4c4c-ab16-064f211e5c03

My crypttab:

cr_nvme-eui.00000000000000016479a7716f0000f7-part2  UUID=4834839d-7933-4294-acff-3fa6f5e22909 none x-initrd.attach
cr_ata-ST8000VN004-3CP101_WWZ7KJ9J UUID=6ac85733-ab88-40b7-841a-5babefcf70cc none x-initrd.attach
raid2 UUID=031c7ad8-0776-4601-b774-bc396c6298e1 none x-initrd.attach

When starting up, the system idles with a message like: waiting for diskbyUUID… and unless I reconnect the OLD, DAMAGED drive, it won’t continue booting.
I formatted the new sda drive without a filesystem but with LUKS2 enabled.

[arvidjaar]

@lavadrop

diskbyUUID

Show this UUID. Show the actual message.

@lavadrop

lsblk

As you see, this output does not show any UUID so it would have been useless for comparing with the message anyway. But it shows 4 LUKS devices and your /etc/crypttab shows only 3 lines. Next wild guess - initrd contains /etc/crypttab with no more existing device (which also explains how it gets activated to start with).

Try rebuilding initrd.

[lavadrop]

[    0.874239] [      T1] systemd[1]: Expecting device /dev/disk/by-uuid/031c7ad8-0776-4601-b774-bc396c6298e1...
[    0.874245] [      T1] systemd[1]: Expecting device /dev/disk/by-uuid/0ae86c72-564c-41d1-9333-eedfe64c84ea...
[    0.874248] [      T1] systemd[1]: Expecting device /dev/disk/by-uuid/4834839d-7933-4294-acff-3fa6f5e22909...
[    0.874252] [      T1] systemd[1]: Expecting device /dev/disk/by-uuid/B719-5173...

031c7ad8-0776-4601-b774-bc396c6298e1 -> ../../sdb
0ae86c72-564c-41d1-9333-eedfe64c84ea -> ../../sde
3a5fa0a8-1e39-4c4c-ab16-064f211e5c03 -> ../../dm-3
4114a69e-5b10-4537-ab86-60773064a389 -> ../../dm-1
4834839d-7933-4294-acff-3fa6f5e22909 -> ../../nvme0n1p2
6ac85733-ab88-40b7-841a-5babefcf70cc -> ../../sda
95f203ba-c86e-4069-9c19-66d362c9bd2d -> ../../sdc1
b5dc3d44-c92e-4805-941f-f6ac8aa74dab -> ../../sdd
B719-5173 -> ../../nvme0n1p1
e1403994-22f8-4099-a912-49f4e92a048f -> ../../dm-5
e4ee98e6-2e04-4f04-87ff-3c4368513e43 -> ../../dm-2

[arvidjaar]

@lavadrop

[    0.874245] [      T1] systemd[1]: Expecting device /dev/disk/by-uuid/0ae86c72-564c-41d1-9333-eedfe64c84ea...

If I follow your story correctly, this is the removed device (or, better, the LUKS container for the removed device). Then unplug it and rebuild initrd.

[lavadrop]

I see, thank you. How do I rebuild initrd?

[arvidjaar]

@lavadrop

How do I rebuild initrd?

dracut --regenerate-all --force

[lavadrop]

I got this:
dracut[F]: No permission to write to /boot.

[gogalthorp]

Did you do it as root???

[lavadrop]

No.
Jesus, does every sysadmin assume everyone else is a sysadmin?

[argentwolf]

No, only us recovering Windows users.

[lavadrop]

Just kidding

[windl]

I find it rather surprising that you could set up a BtrFS RAID1 with encrypted disks, but still being not aware that a normal user is not allowed to change the system configuration. Having to use root protects the system from bad software and from user mistakes. Learn to like it!