raid1, crypt, now can't mount....

[kmleon]

dv9:/ # ls -l /dev/sda9
brw-rw---- 1 root disk 8, 9 Apr 8 18:39 /dev/sda9
dv9:/ # ls -l /dev/sdb9
brw-rw---- 1 root disk 8, 25 Apr 8 04:16 /dev/sdb9

[kmleon]

dv9:/ # mdadm -S /dev/md1
mdadm: stopped /dev/md1
dv9:/ # mdadm -A /dev/md1 /dev/sdb9
mdadm: /dev/md/1 assembled from 1 drive - need all 2 to start it (use --run to insist).
dv9:/ # mdadm /dev/md1 -a /dev/sda9
mdadm: cannot get array info for /dev/md1
dv9:/ # mdadm -A /dev/md1 /dev/sdb9 --run
mdadm: cannot open device /dev/sdb9: Device or resource busy
mdadm: /dev/sdb9 has no superblock - assembly aborted
dv9:/ # mdadm /dev/md1 -a /dev/sda9
mdadm: cannot get array info for /dev/md1

[kmleon]

dv9:/ # mdadm -S /dev/md1
mdadm: stopped /dev/md1
dv9:/ # mdadm --assemble -v /dev/md1 /dev/sdb9
mdadm: looking for devices for /dev/md1
mdadm: /dev/sdb9 is identified as a member of /dev/md/1, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md/1
mdadm: added /dev/sdb9 to /dev/md/1 as 1
mdadm: /dev/md/1 assembled from 1 drive - need all 2 to start it (use --run to insist).


/var/log/messages:
Apr 8 18:50:51 dv9 kernel: md: md1 stopped.
Apr 8 18:50:51 dv9 kernel: md: unbind<sdb9>
Apr 8 18:50:51 dv9 kernel: md: export_rdev(sdb9)
Apr 8 18:51:13 dv9 kernel: md: md1 stopped.
Apr 8 18:51:13 dv9 kernel: md: bind<sdb9>



dv9:/ # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md0 : active raid1 sdb8[0] sda8[1]
31463196 blocks super 1.0 [2/2] [UU]
bitmap: 9/241 pages [36KB], 64KB chunk

unused devices: <none>
dv9:/ # mdadm -A -v /dev/md1 /dev/sdb9 --run
mdadm: looking for devices for /dev/md1
mdadm: /dev/sdb9 is identified as a member of /dev/md/1, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md/1
mdadm: added /dev/sdb9 to /dev/md/1 as 1
mdadm: failed to RUN_ARRAY /dev/md/1: Input/output error

dv9:/ # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md0 : active raid1 sdb8[0] sda8[1]
31463196 blocks super 1.0 [2/2] [UU]
bitmap: 9/241 pages [36KB], 64KB chunk

unused devices: <none>
dv9:/ # mdadm -A -v /dev/md1 /dev/sdb9 --run
mdadm: looking for devices for /dev/md1
mdadm: /dev/sdb9 is identified as a member of /dev/md/1, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md/1
mdadm: added /dev/sdb9 to /dev/md/1 as 1
mdadm: failed to RUN_ARRAY /dev/md/1: Input/output error


Apr 8 18:52:29 dv9 kernel: md: md1 stopped.
Apr 8 18:52:29 dv9 kernel: md: bind<sdb9>
Apr 8 18:52:29 dv9 kernel: md: md1: raid array is not clean -- starting background reconstruction
Apr 8 18:52:29 dv9 kernel: raid1: raid set md1 active with 1 out of 2 mirrors
Apr 8 18:52:29 dv9 kernel: md1: bitmap file is out of date, doing full recovery
Apr 8 18:52:29 dv9 kernel: md1: bitmap initialisation failed: -5
Apr 8 18:52:29 dv9 kernel: md1: failed to create bitmap (-5)



dv9:/ # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md1 : inactive sdb9[1]
52460152 blocks super 1.0

md0 : active raid1 sdb8[0] sda8[1]
31463196 blocks super 1.0 [2/2] [UU]
bitmap: 2/241 pages [8KB], 64KB chunk

unused devices: <none>

[kmleon]

I'm about out of ideas. Anyone at all have ANY suggestions?

[kmleon]

I hope this isn't a fatal (to my old data)mistake. Trying to go ahead and (re)create the array...

v9:/ # mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/sdb9 missing
mdadm: /dev/sdb9 appears to be part of a raid array:
level=raid1 devices=2 ctime=Fri Dec 5 20:22:43 2008
mdadm: size set to 52460160K
Continue creating array? y
mdadm: array /dev/md/1 started.


/var/log/messages:
Apr 8 20:06:25 dv9 kernel: md: md1 stopped.
Apr 8 20:06:25 dv9 kernel: md: unbind<sdb9>
Apr 8 20:06:25 dv9 kernel: md: export_rdev(sdb9)
Apr 8 20:23:37 dv9 kernel: md: bind<sdb9>
Apr 8 20:23:37 dv9 kernel: md: md1: raid array is not clean -- starting background reconstruction
Apr 8 20:23:38 dv9 kernel: raid1: raid set md1 active with 1 out of 2 mirrors


dv9:/var/log # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md1 : active (auto-read-only) raid1 sdb9[0]
52460160 blocks [2/1] [U_]

md0 : active raid1 sdb8[0] sda8[1]
31463196 blocks super 1.0 [2/2] [UU]
bitmap: 4/241 pages [16KB], 64KB chunk

unused devices: <none>



Now what?

[kmleon]

dv9:/ # mdadm -D /dev/md1
/dev/md1:
Version : 0.90
Creation Time : Wed Apr 8 20:23:37 2009
Raid Level : raid1
Array Size : 52460160 (50.03 GiB 53.72 GB)
Used Dev Size : 52460160 (50.03 GiB 53.72 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 1
Persistence : Superblock is persistent

Update Time : Wed Apr 8 20:23:37 2009
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

UUID : 26558511:fa77c7fd:36792f3c:b777964f (local to host dv9)
Events : 0.1

Number Major Minor RaidDevice State
0 8 25 0 active sync /dev/sdb9
1 0 0 1 removed


dv9:/ # /etc/init.d/boot.crypto-early start
Activating crypto devices using /etc/crypttab ...
Please enter passphrase for /dev/md1 (cr_md1):
Please enter passphrase for /dev/md1 (cr_md1): key slot 0 unlocked.
Command successful.


That's an improvement...
but.....


Apr 8 20:26:29 dv9 kernel: ------------[ cut here ]------------
Apr 8 20:26:29 dv9 kernel: WARNING: at kernel/softirq.c:137 local_bh_enable+0x3c/0x92()
Apr 8 20:26:29 dv9 kernel: Modules linked in: sha256_generic aes_x86_64 aes_generic cbc nls_iso8859_1 nls_cp437 vfat fat xt_tcpudp xt_pkttype ipt_LOG xt_limit xt_physdev af_packet binfmt_misc sco bridge stp bnep rfcomm l2cap snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device xt_NOTRACK ipt_REJECT xt_state iptable_raw iptable_filter nf_conntrack_netbios_ns nf_conntrack_ipv4 nf_conntrack ip_tables ip6_tables x_tables cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq fuse ohci_hcd dm_crypt ext2 loop dm_mod arc4 ecb crypto_blkcipher mmc_block iwl3945 snd_hda_intel rfkill snd_pcm iTCO_wdt hci_usb(N) mac80211 uvcvideo snd_timer iTCO_vendor_support sdhci_pci rtc_cmos compat_ioctl32 snd_page_alloc sdhci led_class rtc_core video snd_hwdep btusb snd usb_storage nvidia(PX) option rtc_lib mmc_core joydev ohci1394 output i2c_i801 videodev ricoh_mmc battery ac v4l1_compat pcspkr wmi button usbserial ieee1394 sr_mod intel_agp soundcore cfg80211 bluetooth cdrom i2c_core sg e1000e usbhid hid ff_memless raid
Apr 8 20:26:29 dv9 kernel: 456 async_xor async_memcpy async_tx xor raid0 uhci_hcd ehci_hcd sd_mod crc_t10dif usbcore edd raid1 ext3 mbcache jbd fan ide_pci_generic piix ide_core ata_generic ata_piix thermal processor thermal_sys hwmon ahci libata scsi_mod dock
Apr 8 20:26:29 dv9 kernel: Supported: No
Apr 8 20:26:29 dv9 kernel: Pid: 10227, comm: cryptsetup Tainted: P 2.6.27.19-3.2-default #1
Apr 8 20:26:29 dv9 kernel:
Apr 8 20:26:29 dv9 kernel: Call Trace:
Apr 8 20:26:29 dv9 kernel: [<ffffffff8020da29>] show_trace_log_lvl+0x41/0x58
Apr 8 20:26:29 dv9 kernel: [<ffffffff8049a3da>] dump_stack+0x69/0x6f
Apr 8 20:26:29 dv9 kernel: [<ffffffff8023d562>] warn_on_slowpath+0x51/0x77
Apr 8 20:26:29 dv9 kernel: [<ffffffff802425b3>] local_bh_enable+0x3c/0x92
Apr 8 20:26:29 dv9 kernel: [<ffffffff80433be7>] sk_filter+0x97/0xa0
Apr 8 20:26:29 dv9 kernel: [<ffffffff80441a42>] netlink_broadcast+0x220/0x386
Apr 8 20:26:29 dv9 kernel: [<ffffffff803614cd>] kobject_uevent_env+0x452/0x520
Apr 8 20:26:29 dv9 kernel: [<ffffffff803401d7>] set_disk_ro+0x58/0x8d
Apr 8 20:26:29 dv9 kernel: [<ffffffffa0bde1d1>] __bind+0x11d/0x136 [dm_mod]
Apr 8 20:26:29 dv9 kernel: [<ffffffffa0bde23e>] dm_swap_table+0x54/0x70 [dm_mod]
Apr 8 20:26:29 dv9 kernel: [<ffffffffa0be2766>] do_resume+0xa9/0x122 [dm_mod]
Apr 8 20:26:29 dv9 kernel: [<ffffffffa0be317d>] ctl_ioctl+0x1cb/0x213 [dm_mod]
Apr 8 20:26:29 dv9 kernel: [<ffffffffa0be31d3>] dm_ctl_ioctl+0xe/0x12 [dm_mod]
Apr 8 20:26:29 dv9 kernel: [<ffffffff802bd249>] vfs_ioctl+0x21/0x6c
Apr 8 20:26:29 dv9 kernel: [<ffffffff802bd4b6>] do_vfs_ioctl+0x222/0x231
Apr 8 20:26:29 dv9 kernel: [<ffffffff802bd516>] sys_ioctl+0x51/0x73
Apr 8 20:26:29 dv9 kernel: [<ffffffff8020bfbb>] system_call_fastpath+0x16/0x1b
Apr 8 20:26:29 dv9 kernel: [<00007f9f4c742b77>] 0x7f9f4c742b77
Apr 8 20:26:29 dv9 kernel:
Apr 8 20:26:29 dv9 kernel: ---[ end trace 962f83b48f852457 ]---



hmmm...


But still not mounting enabling the encrypted access properly yet:
dv9:/ # ls /home
<notice that it's empty>

Now doing this:
dv9:/ # mount /dev/mapper/cr_md1 /home
dv9:/ # ls /home
user user.img user.key lost+found


dv9:/ # du -sh /home/*
60K /home/user
44G /home/user.img
4.0K /home/user.key
16K /home/lost+found

Ok. the correct data is in the user.img . Ithink that is the next layer of encyryption I added, but don't know how to access that next.

Anyone?

[kmleon]

Ok, so I _think_the md1 raid1 array is now ok (running in degraded mode):
dv9:/var/log # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md1 : active (auto-read-only) raid1 sdb9[0]
52460160 blocks [2/1] [U_]

md0 : active raid1 sdb8[0] sda8[1]
31463196 blocks super 1.0 [2/2] [UU]
bitmap: 1/241 pages [4KB], 64KB chunk

unused devices: <none>
dv9:/var/log # cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md1 : active raid1 sdb9[0]
52460160 blocks [2/1] [U_]

md0 : active raid1 sdb8[0] sda8[1]
31463196 blocks super 1.0 [2/2] [UU]
bitmap: 2/241 pages [8KB], 64KB chunk

unused devices: <none>
dv9:/var/log # mdadm -D /dev/md1
/dev/md1:
Version : 0.90
Creation Time : Wed Apr 8 20:23:37 2009
Raid Level : raid1
Array Size : 52460160 (50.03 GiB 53.72 GB)
Used Dev Size : 52460160 (50.03 GiB 53.72 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 1
Persistence : Superblock is persistent

Update Time : Wed Apr 8 20:33:06 2009
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

UUID : 26558511:fa77c7fd:36792f3c:b777964f (local to host dv9)
Events : 0.12

Number Major Minor RaidDevice State
0 8 25 0 active sync /dev/sdb9
1 0 0 1 removed





But, I cant' seem to add the old sda9 drive/partition back to the array. I'll worry about that later.
Now I'm just trying to get to the data, so I can back it up elsewhere.
But I can't get past the second layer of encryption (yet, trying to figure out which tech I used for that, dm-crypt maybe? aes?).

/var/log/messages:
Apr 8 20:32:35 dv9 kernel: kjournald starting. Commit interval 5 seconds
Apr 8 20:32:35 dv9 kernel: EXT3 FS on dm-0, internal journal
Apr 8 20:32:35 dv9 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Apr 8 20:32:42 dv9 kernel: JBD: barrier-based sync failed on dm-0 - disabling barriers

[kmleon]

This will make it create without having to answer "y" first.
mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/sdb9 missing --force --run


Maybe I can add this to init before boot.crypt-early to try to have it ready, and then maybe it will decrypt properly rather than failing and having to do everything manually?

Will try it (what a kludge).

[kmleon]

Well, the kludge did the trick.
I put the command:
mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/sdb9 missing --force --run


Into the /etc/init.d/boot.md file near the end of it's start declaration.
Rebooted.
It created the degraded md1 array in time for the boot.crypto-early to work and prompt for password.
Then when I logged in as user, it worked.
Phew!
Now using unison to back up to a different laptop. Then will do a fresh wipe and reinstall (the 11.1 has been a disaster due to KDE4 (see my posts on that), I will be doing reinstall with KDE 3.5x instead).
I hope this ordeal helps someone out at least, so they aren't on their own as I was on this.
:-(
Cheers!