Three servers have openSUSE Leap 16.0 installed, along with components including cluster-md-kmp-default and dlm-kmp-default.
A 3-replica RAID1 array based on md-cluster was configured. An issue occurs during disk re-add testing that results in permanent data loss. The detailed operation steps are as follows:
- Create the RAID1 array on Node 1
bash
运行
mdadm --create cm --bitmap=clustered --metadata=1.2 --nodes=12 --assume-clean --level=mirror --raid-devices=3 /dev/sdb /dev/sdc /dev/sdd
- Assemble the array on Node 2 and Node 3 Assembly command:
bash
运行
mdadm -A cm /dev/sdb /dev/sdc /dev/sdd
- Create and mount the XFS filesystem on Node 3
bash
运行
mkfs.xfs /dev/md/cm
mount /dev/md/cm /d
plaintext
n3:~ # mdadm -D /dev/md/cm
/dev/md/cm:
Version : 1.2
Creation Time : Fri Jul 17 09:46:44 2026
Raid Level : raid1
Array Size : 41909248 (39.97 GiB 42.92 GB)
Used Dev Size : 41909248 (39.97 GiB 42.92 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Intent Bitmap : Internal(Clustered)
Update Time : Fri Jul 17 10:03:40 2026
State : active
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Consistency Policy : bitmap
Name : n1:cm
Cluster Name : ficusCluster
UUID : 47b9dd84:10f02a14:c5584ad6:5d6f625e
Events : 76
Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
1 8 32 1 active sync /dev/sdc
2 8 48 2 active sync /dev/sdd
plaintext
n3:~ # mkfs.xfs /dev/md/cm -f
meta-data=/dev/md/cm isize=512 agcount=4, agsize=2619328 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=1
= reflink=1 bigtime=1 inobtcount=1 nrext64=1
= exchange=1 metadir=0
data = bsize=4096 blocks=10477312, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1, parent=1
log =internal log bsize=4096 blocks=16384, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
= rgcount=0 rgsize=0 extents
= zoned=0 start=0 reserved=0
Discarding blocks...Done.
- Simulate disk failure
bash
运行
mdadm /dev/md/cm -f /dev/sdb
mdadm --manage /dev/md/cm --remove /dev/sdb
Kernel log snippet:
plaintext
[ 5069.968110] [ T1938] md/raid1:md127: Disk failure on sdb, disabling device.
md/raid1:md127: Operation continuing on 2 devices.
- Write test data on Node 3
bash
运行
dd if=/dev/zero of=/d/1.img bs=1M count=5000
sync
plaintext
n3:~ # sync
n3:~ # dd if=/dev/zero of=/d/1.img bs=1M count=5000
sync
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 25.7616 s, 204 MB/s
n3:~ # sync
n3:~ #
- Re-add /dev/sdb on Node 1 and observe array recovery status
bash
运行
mdadm --manage /dev/md/cm --re-add /dev/sdb
plaintext
n1:~ # date
Fri Jul 17 11:00:32 CST 2026
n1:~ # cat /proc/mdstat
Personalities : [raid1]
md127 : active raid1 sdd[2] sdc[1]
41909248 blocks super 1.2 [3/2] [_UU]
bitmap: 1/1 pages [4KB], 65536KB chunk
unused devices: <none>
n1:~ # mdadm --manage /dev/md/cm --re-add /dev/sdb
mdadm: re-added /dev/sdb
n1:~ # cat /proc/mdstat
Personalities : [raid1]
md127 : active raid1 sdb[0] sdd[2] sdc[1]
41909248 blocks super 1.2 [3/3] [UUU]
bitmap: 1/1 pages [4KB], 65536KB chunk
unused devices: <none>
n1:~ # date
Fri Jul 17 11:00:54 CST 2026
n1:~ # mdadm -D /dev/md/cm
/dev/md/cm:
Version : 1.2
Creation Time : Fri Jul 17 09:46:44 2026
Raid Level : raid1
Array Size : 41909248 (39.97 GiB 42.92 GB)
Used Dev Size : 41909248 (39.97 GiB 42.92 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Intent Bitmap : Internal(Clustered)
Update Time : Fri Jul 17 11:00:46 2026
State : active
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Consistency Policy : bitmap
Name : n1:cm (local to host n1)
Cluster Name : ficusCluster
UUID : 47b9dd84:10f02a14:c5584ad6:5d6f625e
Events : 89
Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
1 8 32 1 active sync /dev/sdc
2 8 48 2 active sync /dev/sdd
n1:~ #
Kernel recovery log:
plaintext
[ 5194.489611] [ T1965] md: recover of RAID array md127
[ 5194.489870] [ T1965] md: md127: c done.
Problem Summary
The entire recovery process completes instantaneously, and no actual data synchronization/recovery is performed during the rebuild.