Kernel Panic after openSUSE online upgrade 12.1->12.2 on x86_64 with RAID

Any suggestions to fix my problem are greatly appreciated.

After upgrading following the instructions at https://en.opensuse.org/SDB:System_upgrade, rebooting resulted in a kernel panic. The last bit I could see on the screen was

Waiting for device /dev/md5 to appear: ok
fsck from util-linux-2.20.1
[/sbin/fsck.ext4 (1) -- /] fsck.ext4 -a -C0 /dev/md5
recovering journal
clean, 21583/643386 files, 844544/2520240 blocks
fsck succeeded, Mounting root device read-write
Mounting root /dev/md5
mount -o rw,acl,user_xattr -t ext4 /dev/md5 /root
chroot: failed to run command '/sbin/fsck': no such file or directory
/sbin/init: error while loading shared libraries: libkmod.so.2: cannot open shared object file: no such file or directory
 Kernel panic - not syncing: Attempted to kill init! ]
 Pid: 1, comm: init Not tainted 3.1.10-1.16-desktop #1 ]
 Call Trace: ]
 dump_trace, dump_stack, panic, forget_original_parent, exit_notify, etc ] 

The file /etc/fstab on the system is this:

/dev/sda9	swap	swap	pri=1 0 0 
/dev/sdb9	swap	swap	pri=1 0 0 
/dev/sdc9	swap	swap	pri=1 0 0 
/dev/sdd9	swap	swap	pri=1 0 0 
/dev/sde9	swap	swap	pri=1 0 0 
/dev/sdf9	swap	swap	pri=1 0 0 
/dev/md5	/	ext4	acl,user_xattr 1 1 
/dev/md1	/boot	ext3	acl,user_xattr 1 2 
/dev/md10	/projects	ext4	acl,user_xattr 1 2 
/dev/md8	/tmp	ext4	acl,user_xattr 1 2 
/dev/md6	/usr	ext4	acl,user_xattr 1 2 
/dev/md7	/var	ext4	acl,user_xattr 1 2 
proc	/proc	proc	defaults 0 0 
sysfs	/sys	sysfs	noauto 0 0 
debugfs	/sys/kernel/debug	debugfs	noauto 0 0 
usbfs	/proc/bus/usb	usbfs	noauto 0 0 
devpts	/dev/pts	devpts	mode=0620,gid=5 0 0 
frazer-share.ucsd.edu:/media	/nas1	nfs	defaults 0 0 
X.ucsd.edu:/home	/home	nfs4	rsize=32768,wsize=32768,proto=tcp,intr 0 0 
X.ucsd.edu:/raid	/raid	nfs4	rsize=32768,wsize=32768,proto=tcp,intr 0 0 
X.ucsd.edu:/raid2	/raid2	nfs4	rsize=32768,wsize=32768,proto=tcp,intr 0 0 

I then booted from the openSUSE 12.3 Live DVD.

First, I did “fdisk -l”:

Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9a6f874e


   Device Boot      Start         End      Blocks   Id  System
/dev/sda1              63     1044224      522081   fd  Linux raid autodetect
/dev/sda4         1044225  3907024064  1952989920    5  Extended
/dev/sda5         1044288     5156864     2056288+  fd  Linux raid autodetect
/dev/sda6         5156928    19856339     7349706   fd  Linux raid autodetect
/dev/sda7        19856403    23968979     2056288+  fd  Linux raid autodetect
/dev/sda8        23969043    28081619     2056288+  fd  Linux raid autodetect
/dev/sda9        28081683    39150404     5534361   82  Linux swap / Solaris
/dev/sda10       39150468  3907024064  1933936798+  fd  Linux raid autodetect


Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xcfb381fc


   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63     1044224      522081   fd  Linux raid autodetect
/dev/sdb4         1044225  3907024064  1952989920    5  Extended
/dev/sdb5         1044288     5156864     2056288+  fd  Linux raid autodetect
/dev/sdb6         5156928    19856339     7349706   fd  Linux raid autodetect
/dev/sdb7        19856403    23968979     2056288+  fd  Linux raid autodetect
/dev/sdb8        23969043    28081619     2056288+  fd  Linux raid autodetect
/dev/sdb9        28081683    39150404     5534361   82  Linux swap / Solaris
/dev/sdb10       39150468  3907024064  1933936798+  fd  Linux raid autodetect


Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9aa07bdd


   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1              63     1044224      522081   fd  Linux raid autodetect
/dev/sdc4         1044225  3907024064  1952989920    5  Extended
/dev/sdc5         1044288     5156864     2056288+  fd  Linux raid autodetect
/dev/sdc6         5156928    19856339     7349706   fd  Linux raid autodetect
/dev/sdc7        19856403    23968979     2056288+  fd  Linux raid autodetect
/dev/sdc8        23969043    28081619     2056288+  fd  Linux raid autodetect
/dev/sdc9        28081683    39150404     5534361   82  Linux swap / Solaris
/dev/sdc10       39150468  3907024064  1933936798+  fd  Linux raid autodetect


Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0b7ba768


   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1              63     1044224      522081   fd  Linux raid autodetect
/dev/sdd4         1044225  3907024064  1952989920    5  Extended
/dev/sdd5         1044288     5156864     2056288+  fd  Linux raid autodetect
/dev/sdd6         5156928    19856339     7349706   fd  Linux raid autodetect
/dev/sdd7        19856403    23968979     2056288+  fd  Linux raid autodetect
/dev/sdd8        23969043    28081619     2056288+  fd  Linux raid autodetect
/dev/sdd9        28081683    39150404     5534361   82  Linux swap / Solaris
/dev/sdd10       39150468  3907024064  1933936798+  fd  Linux raid autodetect


Disk /dev/sde: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x23e67f92


   Device Boot      Start         End      Blocks   Id  System
/dev/sde1              63     1044224      522081   fd  Linux raid autodetect
/dev/sde4         1044225  3907024064  1952989920    5  Extended
/dev/sde5         1044288     5156864     2056288+  fd  Linux raid autodetect
/dev/sde6         5156928    19856339     7349706   fd  Linux raid autodetect
/dev/sde7        19856403    23968979     2056288+  fd  Linux raid autodetect
/dev/sde8        23969043    28081619     2056288+  fd  Linux raid autodetect
/dev/sde9        28081683    39150404     5534361   82  Linux swap / Solaris
/dev/sde10       39150468  3907024064  1933936798+  fd  Linux raid autodetect


Disk /dev/sdf: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xb6889aa5


   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1              63     1044224      522081   fd  Linux raid autodetect
/dev/sdf4         1044225  3907024064  1952989920    5  Extended
/dev/sdf5         1044288     5156864     2056288+  fd  Linux raid autodetect
/dev/sdf6         5156928    19856339     7349706   fd  Linux raid autodetect
/dev/sdf7        19856403    23968979     2056288+  fd  Linux raid autodetect
/dev/sdf8        23969043    28081619     2056288+  fd  Linux raid autodetect
/dev/sdf9        28081683    39150404     5534361   82  Linux swap / Solaris
/dev/sdf10       39150468  3907024064  1933936798+  fd  Linux raid autodetect


Disk /dev/md127: 10.5 GB, 10527703040 bytes
2 heads, 4 sectors/track, 2570240 cylinders, total 20561920 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 327680 bytes




Disk /dev/md126: 534 MB, 534511616 bytes
2 heads, 4 sectors/track, 130496 cylinders, total 1043968 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes




Disk /dev/md125: 9901.8 GB, 9901755924480 bytes
2 heads, 4 sectors/track, -1877546416 cylinders, total 19339367040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 327680 bytes




Disk /dev/md124: 10.5 GB, 10527703040 bytes
2 heads, 4 sectors/track, 2570240 cylinders, total 20561920 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 327680 bytes




Disk /dev/md123: 37.6 GB, 37630115840 bytes
2 heads, 4 sectors/track, 9187040 cylinders, total 73496320 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 327680 bytes




Disk /dev/md122: 10.5 GB, 10527703040 bytes
2 heads, 4 sectors/track, 2570240 cylinders, total 20561920 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 327680 bytes

The partition dev names are now different from what is in /etc/fstab. I was able to figure out the partition correspondence:
/dev/md122 is /
/dev/md123 is /usr
/dev/md124 is /var
/dev/md125 is /projects
/dev/md126 is /boot
/dev/md127 is /tmp

I then mounted my disk partitions:

mount /dev/md122 /mnt
mount --bind /proc /mnt/proc
mount --bind /dev /mnt/dev
mount --bind /dev/pts /mnt/dev/pts
mount --bind /sys /mnt/sys
mount /dev/md123 /mnt/usr
mount /dev/md124 /mnt/var
mount /dev/md125 /mnt/projects
mount /dev/md126 /mnt/boot
mount /dev/md127 /mnt/tmp
chroot /mnt /bin/bash

I’m not certain where the problem lies.
I’m aware of the “separate /usr partition problem”. But as far as I can tell, that is supposed to be fixed in the mkinitd version that I have.
After the upgrade the partition names in /etc/fstab are different than what fdisk shows, but I’m not certain if this matters.

I noticed that grub2 is not in boot, even though the package is installed. So I did:

/boot # grub2-install /dev/md122
/usr/sbin/grub2-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub2-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub2-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub2-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub2-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub2-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub2-bios-setup: warning: File system `ext2' doesn't support embedding.
/usr/sbin/grub2-bios-setup: error: embedding is not possible, but this is required for cross-disk install.

Apparently not the right thing to do. Since I’m not familiar with this part of the Linux OS, I decided to stop at this point and ask for help. I don’t want to dig myself deeper.

I would rather not do a complete reinstall, as there are many configuration files, etc. that would take time to reinstate.

If using a LiveCD or LiveUSB with the same openSUSE version finds all of your drives, I would consider doing a clean install but, only mounting /home and not formatting it. You keep all of your personnel settings, allowing the openSUSE installer to handle the new names it find and reinstall all applications and system settings. Its not exactly like an upgrade, but keeping all personnel settings is still a big deal. You pick custom partition for experts, remount all existing partitions at the same place, root for root /, /home for /home, but you only need to format root /, while only mounting /home and any other existing partitions. Its a thought you should consider. Doing a clean install might be the time to go with openSUSE 12.3 as well.

Thank You,

Was it online or offline update?

Waiting for device /dev/md5 to appear: ok 
fsck from util-linux-2.20.1 

12.2 is using util-linux 2.21.2, so you are apprently using old initrd and likely an old kernel. At the very least you need to rebuild initrd - from chroot just call mkinitrd. And do not use Live CD from different version, use 12.2 one.