raid1, crypt, now can't mount....

I’ve been beating my head against this for the second day now, and think I’m going in circles now, so could use some external guidance please.
This is in a laptop with two hard drives running opensuse 11.1.
I setup raid1 (mirror) between the two drives.
md0 mounts root / using sda8 and sdb8
md1 mounts /home user sda9 and sdb9

/home is also encrypted (LUKS password at boot time).
And I (think) I added a second layer of encryption (but not sure at this point) encrypted the specific director /home/user.

This system has been purring along fine with this setup for months. All of a sudden on a boot up yesterday afternoon (it was working earlier on bootup that morning), I couldn’t access /home anymore. It didn’t ask for the LUKS password, and showed errors on the bootup indicating that the md1 was not a device, and so crypt couldn’t load.
md0 loads fine (root).

When the system boots or runs /etc/init.d/boot.crypt-early it fails because md1 fails (or the interdependency between them).

Here is some output, maybe folks can help point me in the right direction, I’ve tried dozens of sites and forums approaches from previous others postings, but not quite getting there.

Here some info:

v9:/ # cat /etc/mdadm.conf
DEVICE partitions
ARRAY /dev/md0 level=raid1 UUID=cc67b93f:c23c38d7:6a446079:1726b629
ARRAY /dev/md1 level=raid1 UUID=adb90daa:7e594eba:f6dc04c6:73a5d54a

cat /etc/crypttab
cr_md1 /dev/md1 none none

mdadm -E /dev/sda9
/dev/sda9:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : adb90daa:7e594eba:f6dc04c6:73a5d54a
Name : 1
Creation Time : Fri Dec 5 20:22:43 2008
Raid Level : raid1
Raid Devices : 2

Avail Dev Size : 104920304 (50.03 GiB 53.72 GB)
Array Size : 104920304 (50.03 GiB 53.72 GB)
Super Offset : 104920432 sectors
State : active
Device UUID : 7d0b8cee:55afb0b5:2f6d4ca3:c820e941

Internal Bitmap : -101 sectors from superblock
Update Time : Tue Apr 7 10:30:12 2009
Checksum : 4dad99ba - correct
Events : 0

Array Slot : 2 (0, 1)

Array State : uu

mdadm -E /dev/sdb9
/dev/sdb9:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x1
Array UUID : adb90daa:7e594eba:f6dc04c6:73a5d54a
Name : 1
Creation Time : Fri Dec 5 20:22:43 2008
Raid Level : raid1
Raid Devices : 2

Avail Dev Size : 104920304 (50.03 GiB 53.72 GB)
Array Size : 104920304 (50.03 GiB 53.72 GB)
Super Offset : 104920432 sectors
State : active
Device UUID : 776ec64a:5a5f3aef:37170c0e:0969cccb

Internal Bitmap : -101 sectors from superblock
Update Time : Tue Apr 7 10:30:12 2009
Checksum : d8149f11 - correct
Events : 17

Array Slot : 1 (0, 1)

Array State : uU

cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md0 : active raid1 sdb8[0] sda8[1]
31463196 blocks super 1.0 [2/2] [UU]
bitmap: 3/241 pages [12KB], 64KB chunk

unused devices: <none>

dv9:/ # /etc/init.d/boot.crypto-early start
Activating crypto devices using /etc/crypttab …
Command failed: Not a block device
/dev/md1… failed

dv9:/ # /etc/init.d/boot.md start
Starting MD Raid mdadm: failed to RUN_ARRAY /dev/md/1: Input/output error
failed

dv9:/ # mdadm -S /dev/md1
mdadm: stopped /dev/md1
dv9:/ # mdadm -A /dev/md1
mdadm: failed to RUN_ARRAY /dev/md/1: Input/output error

/var/log/messages:
Apr 8 12:13:53 dv9 kernel: md: md1 stopped.
Apr 8 12:13:53 dv9 kernel: md: bind<sda9>
Apr 8 12:13:53 dv9 kernel: md: bind<sdb9>
Apr 8 12:13:53 dv9 kernel: md: kicking non-fresh sda9 from array!
Apr 8 12:13:53 dv9 kernel: md: unbind<sda9>
Apr 8 12:13:53 dv9 kernel: md: export_rdev(sda9)
Apr 8 12:13:53 dv9 kernel: md: md1: raid array is not clean – starting background reconstruction
Apr 8 12:13:53 dv9 kernel: raid1: raid set md1 active with 1 out of 2 mirrors
Apr 8 12:13:53 dv9 kernel: md1: bitmap file is out of date, doing full recovery
Apr 8 12:13:53 dv9 kernel: md1: bitmap initialisation failed: -5
Apr 8 12:13:53 dv9 kernel: md1: failed to create bitmap (-5)

Thanks for any suggestions.

What else can I provide? Does anything jump out at anyone as to what’s wrong?

I’ve run the BIOS’s HD test and checked smart and no errors on either drive there.

Oh, and additional information:

mdadm --version
mdadm - v3.0-devel2 - 5th November 2008

uname -a
Linux dv9 2.6.27.19-3.2-default #1 SMP 2009-02-25 15:40:44 +0100 x86_64 x86_64 x86_64 GNU/Linux

rpm -qa | grep crypt
cryptsetup-1.0.5_SVNr46-57.12
cryptconfig-32bit-0.3-63.26
libxcrypt-3.0.1-25.29
libxcrypt-32bit-3.0.1-25.20
libcryptsetup0-1.0.5_SVNr46-57.12
python-crypto-2.0.1-28.81
libgcrypt11-32bit-1.4.1-4.1
libgcrypt-devel-1.4.1-4.1
libgcrypt11-1.4.1-4.1
cryptconfig-0.3-63.28

When attempting to just load with one drive in the array, does not work either…

dv9:/ # mdadm -S /dev/md1
mdadm: stopped /dev/md1
dv9:/ # mdadm --assemble --run /dev/md1 /dev/sdb9
mdadm: failed to RUN_ARRAY /dev/md/1: Input/output error

/var/log/messages:
Apr 8 14:37:53 dv9 kernel: md: md1 stopped.
Apr 8 14:37:53 dv9 kernel: md: bind<sdb9>
Apr 8 14:37:53 dv9 kernel: md: md1: raid array is not clean – starting background reconstruction
Apr 8 14:37:53 dv9 kernel: raid1: raid set md1 active with 1 out of 2 mirrors
Apr 8 14:37:53 dv9 kernel: md1: bitmap file is out of date, doing full recovery
Apr 8 14:37:54 dv9 kernel: md1: bitmap initialisation failed: -5
Apr 8 14:37:54 dv9 kernel: md1: failed to create bitmap (-5)

dv9:/ # mdadm -D /dev/md1
/dev/md1:
Version : 1.00
Creation Time : Fri Dec 5 20:22:43 2008
Raid Level : raid1
Used Dev Size : 52460152 (50.03 GiB 53.72 GB)
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent

Update Time : Tue Apr  7 10:30:12 2009
      State : active, degraded, Not Started

Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

       Name : 1
       UUID : adb90daa:7e594eba:f6dc04c6:73a5d54a
     Events : 17

Number   Major   Minor   RaidDevice State
   0       0        0        0      removed
   1       8       25        1      active sync   /dev/sdb9

dv9:/ # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md1 : inactive sdb9[1]
52460152 blocks super 1.0

md0 : active raid1 sdb8[0] sda8[1]
31463196 blocks super 1.0 [2/2] [UU]
bitmap: 5/241 pages [20KB], 64KB chunk

unused devices: <none>

dv9:/ # mdadm /dev/md1 --fail /dev/sda9 --remove /dev/sda9
mdadm: set device faulty failed for /dev/sda9: No such device
dv9:/ # mdadm /dev/md1 --re-add /dev/sda9
mdadm: add new device failed for /dev/sda9 as 2: Invalid argument

dv9 kernel: md1: ADD_NEW_DISK not supported
dv9 kernel: md1: ADD_NEW_DISK not supported

v9:/ # mdadm --examine-bitmap /dev/sdb9
Filename : /dev/sdb9
Magic : 6d746962
Version : 4
UUID : adb90daa:7e594eba:f6dc04c6:73a5d54a
Events : 17
Events Cleared : 16
State : Out of date
Chunksize : 128 KB
Daemon : 5s flush period
Write Mode : Normal
Sync Size : 52460152 (50.03 GiB 53.72 GB)
Bitmap : 409845 bits (chunks), 391168 dirty (95.4%)

Notice how sda9 looks compared to previosly posted sdb9…

dv9:/ # mdadm --examine-bitmap /dev/sda9
Filename : /dev/sda9
Magic : 6d746962
Version : 4
UUID : adb90daa:7e594eba:f6dc04c6:73a5d54a
Events : 17
Events Cleared : 16
State : Out of date
Chunksize : 128 KB
Daemon : 5s flush period
Write Mode : Normal
Sync Size : 52460152 (50.03 GiB 53.72 GB)
Bitmap : 409845 bits (chunks), 409845 dirty (100.0%)

dv9:/ # fdisk -l /dev/sda

Disk /dev/sda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x282d282d

Device Boot Start End Blocks Id System
/dev/sda1 1 64 514048+ 6 FAT16
/dev/sda2 * 65 14592 116696160 f W95 Ext’d (LBA)
/dev/sda5 65 3888 30716248+ b W95 FAT32
/dev/sda6 3889 3952 514048+ 83 Linux
/dev/sda7 3953 4144 1542208+ 82 Linux swap / Solaris
/dev/sda8 4145 8061 31463271 fd Linux raid autodetect
/dev/sda9 8062 14592 52460226 fd Linux raid autodetect
dv9:/ # fdisk -l /dev/sdb

Disk /dev/sdb: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000ec89e

Device Boot Start End Blocks Id System
/dev/sdb1 1 64 514048+ 7 HPFS/NTFS
/dev/sdb2 65 14592 116696160 f W95 Ext’d (LBA)
/dev/sdb5 65 3888 30716248+ 7 HPFS/NTFS
/dev/sdb6 3889 3952 514048+ 83 Linux
/dev/sdb7 3953 4144 1542208+ 82 Linux swap / Solaris
/dev/sdb8 4145 8061 31463271 fd Linux raid autodetect
/dev/sdb9 8062 14592 52460226 fd Linux raid autodetect

dv9:/ # ls -l /dev/sda9
brw-rw---- 1 root disk 8, 9 Apr 8 18:39 /dev/sda9
dv9:/ # ls -l /dev/sdb9
brw-rw---- 1 root disk 8, 25 Apr 8 04:16 /dev/sdb9

dv9:/ # mdadm -S /dev/md1
mdadm: stopped /dev/md1
dv9:/ # mdadm -A /dev/md1 /dev/sdb9
mdadm: /dev/md/1 assembled from 1 drive - need all 2 to start it (use --run to insist).
dv9:/ # mdadm /dev/md1 -a /dev/sda9
mdadm: cannot get array info for /dev/md1
dv9:/ # mdadm -A /dev/md1 /dev/sdb9 --run
mdadm: cannot open device /dev/sdb9: Device or resource busy
mdadm: /dev/sdb9 has no superblock - assembly aborted
dv9:/ # mdadm /dev/md1 -a /dev/sda9
mdadm: cannot get array info for /dev/md1

dv9:/ # mdadm -S /dev/md1
mdadm: stopped /dev/md1
dv9:/ # mdadm --assemble -v /dev/md1 /dev/sdb9
mdadm: looking for devices for /dev/md1
mdadm: /dev/sdb9 is identified as a member of /dev/md/1, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md/1
mdadm: added /dev/sdb9 to /dev/md/1 as 1
mdadm: /dev/md/1 assembled from 1 drive - need all 2 to start it (use --run to insist).

/var/log/messages:
Apr 8 18:50:51 dv9 kernel: md: md1 stopped.
Apr 8 18:50:51 dv9 kernel: md: unbind<sdb9>
Apr 8 18:50:51 dv9 kernel: md: export_rdev(sdb9)
Apr 8 18:51:13 dv9 kernel: md: md1 stopped.
Apr 8 18:51:13 dv9 kernel: md: bind<sdb9>

dv9:/ # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md0 : active raid1 sdb8[0] sda8[1]
31463196 blocks super 1.0 [2/2] [UU]
bitmap: 9/241 pages [36KB], 64KB chunk

unused devices: <none>
dv9:/ # mdadm -A -v /dev/md1 /dev/sdb9 --run
mdadm: looking for devices for /dev/md1
mdadm: /dev/sdb9 is identified as a member of /dev/md/1, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md/1
mdadm: added /dev/sdb9 to /dev/md/1 as 1
mdadm: failed to RUN_ARRAY /dev/md/1: Input/output error

dv9:/ # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md0 : active raid1 sdb8[0] sda8[1]
31463196 blocks super 1.0 [2/2] [UU]
bitmap: 9/241 pages [36KB], 64KB chunk

unused devices: <none>
dv9:/ # mdadm -A -v /dev/md1 /dev/sdb9 --run
mdadm: looking for devices for /dev/md1
mdadm: /dev/sdb9 is identified as a member of /dev/md/1, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md/1
mdadm: added /dev/sdb9 to /dev/md/1 as 1
mdadm: failed to RUN_ARRAY /dev/md/1: Input/output error

Apr 8 18:52:29 dv9 kernel: md: md1 stopped.
Apr 8 18:52:29 dv9 kernel: md: bind<sdb9>
Apr 8 18:52:29 dv9 kernel: md: md1: raid array is not clean – starting background reconstruction
Apr 8 18:52:29 dv9 kernel: raid1: raid set md1 active with 1 out of 2 mirrors
Apr 8 18:52:29 dv9 kernel: md1: bitmap file is out of date, doing full recovery
Apr 8 18:52:29 dv9 kernel: md1: bitmap initialisation failed: -5
Apr 8 18:52:29 dv9 kernel: md1: failed to create bitmap (-5)

dv9:/ # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md1 : inactive sdb9[1]
52460152 blocks super 1.0

md0 : active raid1 sdb8[0] sda8[1]
31463196 blocks super 1.0 [2/2] [UU]
bitmap: 2/241 pages [8KB], 64KB chunk

unused devices: <none>

I’m about out of ideas. Anyone at all have ANY suggestions?

I hope this isn’t a fatal (to my old data)mistake. Trying to go ahead and (re)create the array…

v9:/ # mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/sdb9 missing
mdadm: /dev/sdb9 appears to be part of a raid array:
level=raid1 devices=2 ctime=Fri Dec 5 20:22:43 2008
mdadm: size set to 52460160K
Continue creating array? y
mdadm: array /dev/md/1 started.

/var/log/messages:
Apr 8 20:06:25 dv9 kernel: md: md1 stopped.
Apr 8 20:06:25 dv9 kernel: md: unbind<sdb9>
Apr 8 20:06:25 dv9 kernel: md: export_rdev(sdb9)
Apr 8 20:23:37 dv9 kernel: md: bind<sdb9>
Apr 8 20:23:37 dv9 kernel: md: md1: raid array is not clean – starting background reconstruction
Apr 8 20:23:38 dv9 kernel: raid1: raid set md1 active with 1 out of 2 mirrors

dv9:/var/log # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md1 : active (auto-read-only) raid1 sdb9[0]
52460160 blocks [2/1] [U_]

md0 : active raid1 sdb8[0] sda8[1]
31463196 blocks super 1.0 [2/2] [UU]
bitmap: 4/241 pages [16KB], 64KB chunk

unused devices: <none>

Now what?

dv9:/ # mdadm -D /dev/md1
/dev/md1:
Version : 0.90
Creation Time : Wed Apr 8 20:23:37 2009
Raid Level : raid1
Array Size : 52460160 (50.03 GiB 53.72 GB)
Used Dev Size : 52460160 (50.03 GiB 53.72 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 1
Persistence : Superblock is persistent

Update Time : Wed Apr  8 20:23:37 2009
      State : clean, degraded

Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

       UUID : 26558511:fa77c7fd:36792f3c:b777964f (local to host dv9)
     Events : 0.1

Number   Major   Minor   RaidDevice State
   0       8       25        0      active sync   /dev/sdb9
   1       0        0        1      removed

dv9:/ # /etc/init.d/boot.crypto-early start
Activating crypto devices using /etc/crypttab …
Please enter passphrase for /dev/md1 (cr_md1):
Please enter passphrase for /dev/md1 (cr_md1): key slot 0 unlocked.
Command successful.

That’s an improvement…
but…

Apr 8 20:26:29 dv9 kernel: ------------ cut here ]------------
Apr 8 20:26:29 dv9 kernel: WARNING: at kernel/softirq.c:137 local_bh_enable+0x3c/0x92()
Apr 8 20:26:29 dv9 kernel: Modules linked in: sha256_generic aes_x86_64 aes_generic cbc nls_iso8859_1 nls_cp437 vfat fat xt_tcpudp xt_pkttype ipt_LOG xt_limit xt_physdev af_packet binfmt_misc sco bridge stp bnep rfcomm l2cap snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device xt_NOTRACK ipt_REJECT xt_state iptable_raw iptable_filter nf_conntrack_netbios_ns nf_conntrack_ipv4 nf_conntrack ip_tables ip6_tables x_tables cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq fuse ohci_hcd dm_crypt ext2 loop dm_mod arc4 ecb crypto_blkcipher mmc_block iwl3945 snd_hda_intel rfkill snd_pcm iTCO_wdt hci_usb(N) mac80211 uvcvideo snd_timer iTCO_vendor_support sdhci_pci rtc_cmos compat_ioctl32 snd_page_alloc sdhci led_class rtc_core video snd_hwdep btusb snd usb_storage nvidia(PX) option rtc_lib mmc_core joydev ohci1394 output i2c_i801 videodev ricoh_mmc battery ac v4l1_compat pcspkr wmi button usbserial ieee1394 sr_mod intel_agp soundcore cfg80211 bluetooth cdrom i2c_core sg e1000e usbhid hid ff_memless raid
Apr 8 20:26:29 dv9 kernel: 456 async_xor async_memcpy async_tx xor raid0 uhci_hcd ehci_hcd sd_mod crc_t10dif usbcore edd raid1 ext3 mbcache jbd fan ide_pci_generic piix ide_core ata_generic ata_piix thermal processor thermal_sys hwmon ahci libata scsi_mod dock
Apr 8 20:26:29 dv9 kernel: Supported: No
Apr 8 20:26:29 dv9 kernel: Pid: 10227, comm: cryptsetup Tainted: P 2.6.27.19-3.2-default #1
Apr 8 20:26:29 dv9 kernel:
Apr 8 20:26:29 dv9 kernel: Call Trace:
Apr 8 20:26:29 dv9 kernel: <ffffffff8020da29>] show_trace_log_lvl+0x41/0x58
Apr 8 20:26:29 dv9 kernel: <ffffffff8049a3da>] dump_stack+0x69/0x6f
Apr 8 20:26:29 dv9 kernel: <ffffffff8023d562>] warn_on_slowpath+0x51/0x77
Apr 8 20:26:29 dv9 kernel: <ffffffff802425b3>] local_bh_enable+0x3c/0x92
Apr 8 20:26:29 dv9 kernel: <ffffffff80433be7>] sk_filter+0x97/0xa0
Apr 8 20:26:29 dv9 kernel: <ffffffff80441a42>] netlink_broadcast+0x220/0x386
Apr 8 20:26:29 dv9 kernel: <ffffffff803614cd>] kobject_uevent_env+0x452/0x520
Apr 8 20:26:29 dv9 kernel: <ffffffff803401d7>] set_disk_ro+0x58/0x8d
Apr 8 20:26:29 dv9 kernel: <ffffffffa0bde1d1>] __bind+0x11d/0x136 [dm_mod]
Apr 8 20:26:29 dv9 kernel: <ffffffffa0bde23e>] dm_swap_table+0x54/0x70 [dm_mod]
Apr 8 20:26:29 dv9 kernel: <ffffffffa0be2766>] do_resume+0xa9/0x122 [dm_mod]
Apr 8 20:26:29 dv9 kernel: <ffffffffa0be317d>] ctl_ioctl+0x1cb/0x213 [dm_mod]
Apr 8 20:26:29 dv9 kernel: <ffffffffa0be31d3>] dm_ctl_ioctl+0xe/0x12 [dm_mod]
Apr 8 20:26:29 dv9 kernel: <ffffffff802bd249>] vfs_ioctl+0x21/0x6c
Apr 8 20:26:29 dv9 kernel: <ffffffff802bd4b6>] do_vfs_ioctl+0x222/0x231
Apr 8 20:26:29 dv9 kernel: <ffffffff802bd516>] sys_ioctl+0x51/0x73
Apr 8 20:26:29 dv9 kernel: <ffffffff8020bfbb>] system_call_fastpath+0x16/0x1b
Apr 8 20:26:29 dv9 kernel: <00007f9f4c742b77>] 0x7f9f4c742b77
Apr 8 20:26:29 dv9 kernel:
Apr 8 20:26:29 dv9 kernel: — end trace 962f83b48f852457 ]—

hmmm…

But still not mounting enabling the encrypted access properly yet:
dv9:/ # ls /home
<notice that it’s empty>

Now doing this:
dv9:/ # mount /dev/mapper/cr_md1 /home
dv9:/ # ls /home
user user.img user.key lost+found

dv9:/ # du -sh /home/*
60K /home/user
44G /home/user.img
4.0K /home/user.key
16K /home/lost+found

Ok. the correct data is in the user.img . Ithink that is the next layer of encyryption I added, but don’t know how to access that next.

Anyone?

Ok, so I _think_the md1 raid1 array is now ok (running in degraded mode):
dv9:/var/log # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md1 : active (auto-read-only) raid1 sdb9[0]
52460160 blocks [2/1] [U_]

md0 : active raid1 sdb8[0] sda8[1]
31463196 blocks super 1.0 [2/2] [UU]
bitmap: 1/241 pages [4KB], 64KB chunk

unused devices: <none>
dv9:/var/log # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md1 : active raid1 sdb9[0]
52460160 blocks [2/1] [U_]

md0 : active raid1 sdb8[0] sda8[1]
31463196 blocks super 1.0 [2/2] [UU]
bitmap: 2/241 pages [8KB], 64KB chunk

unused devices: <none>
dv9:/var/log # mdadm -D /dev/md1
/dev/md1:
Version : 0.90
Creation Time : Wed Apr 8 20:23:37 2009
Raid Level : raid1
Array Size : 52460160 (50.03 GiB 53.72 GB)
Used Dev Size : 52460160 (50.03 GiB 53.72 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 1
Persistence : Superblock is persistent

Update Time : Wed Apr  8 20:33:06 2009
      State : clean, degraded

Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

       UUID : 26558511:fa77c7fd:36792f3c:b777964f (local to host dv9)
     Events : 0.12

Number   Major   Minor   RaidDevice State
   0       8       25        0      active sync   /dev/sdb9
   1       0        0        1      removed

But, I cant’ seem to add the old sda9 drive/partition back to the array. I’ll worry about that later.
Now I’m just trying to get to the data, so I can back it up elsewhere.
But I can’t get past the second layer of encryption (yet, trying to figure out which tech I used for that, dm-crypt maybe? aes?).

/var/log/messages:
Apr 8 20:32:35 dv9 kernel: kjournald starting. Commit interval 5 seconds
Apr 8 20:32:35 dv9 kernel: EXT3 FS on dm-0, internal journal
Apr 8 20:32:35 dv9 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Apr 8 20:32:42 dv9 kernel: JBD: barrier-based sync failed on dm-0 - disabling barriers

This will make it create without having to answer “y” first.
mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/sdb9 missing --force --run

Maybe I can add this to init before boot.crypt-early to try to have it ready, and then maybe it will decrypt properly rather than failing and having to do everything manually?

Will try it (what a kludge).

Well, the kludge did the trick.
I put the command:
mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/sdb9 missing --force --run

Into the /etc/init.d/boot.md file near the end of it’s start declaration.
Rebooted.
It created the degraded md1 array in time for the boot.crypto-early to work and prompt for password.
Then when I logged in as user, it worked.
Phew!
Now using unison to back up to a different laptop. Then will do a fresh wipe and reinstall (the 11.1 has been a disaster due to KDE4 (see my posts on that), I will be doing reinstall with KDE 3.5x instead).
I hope this ordeal helps someone out at least, so they aren’t on their own as I was on this.
:frowning:
Cheers!