Hi, Hylke!
These are more details about the problem that affected my machines and perhaps also your machine after the opensuse-13.1
x86_64 update on Feb 2 2015 update, around 19:00 CET.
Analysis:
I. Main problem
/boot/initrd (soft link to /boot/initrd-3.11.10-25-desktop) did not contain
lvm2 feature and lvm configuration.
This was true for those kernel 3.x initrds on both machines, which had had
a modification date of ~ 19:00 Feb 2 2051.
The file difference between an unaffected initrd (cf. remedy below) and the initrds
from the update are:
Linux Rescue 3.11.6-4-default #1 SMP Wed Oct 30 18:04:56 UTC 2013 (e6d4a27) x86_64 x86_64 x86_64 GNU/Linux
-rw-r--r-- 1 root root 33671195 Oct 23 2013 /tmp/boot/initrd-3.7.10-1.1-desktop
usr/lib/udev/collect_lvm
usr/lib/udev/rules.d/11-dm-lvm.rules
boot/61-lvm2.sh
sbin/lvm
var/lock/lvm
config/lvm2.sh
etc/lvm
etc/lvm/lvm.conf
etc/sysconfig/lvm
-rw------- 1 root root 22833897 Feb 2 18:59 /tmp/boot/initrd-3.1.10-1.16-xen
<no LVM...>
-rw------- 1 root root 25089666 Feb 2 19:02 /tmp/boot/initrd-3.11.10-21-desktop
<no LVM...>
-rw------- 1 root root 25089585 Feb 2 19:05 /tmp/boot/initrd-3.11.10-25-desktop
<no LVM...>
The main remedy is to create an initrd for the desired grub boot option
explicitly containing “feature lvm2”, i.e.:
mkinitrd -k vmlinuz-3.11.10-25-desktop -i initrd-3.11.10-25-desktop -d /dev/systemvg/s2rootlv -f lvm2 -B
II. Side problems
To be able to generate a working initrd, mkinitrd needs the original system
to analyze. For that, I had to chroot into that system to prepare the nearly
original situation of the system. I include a recipe and my chroot-script
below.
Some problems occured - part of them is ignorance on my side, part of them
is strange behaviour/layout of the opensuse-13.1 configuration.
II.a mkinitrd changes grub.cfg
The mkinitrd of opensuse changes the grub(2) configuration. This is not
what I expect from an mkinitrd, and I do not like this side effect. But,
ok: there is an option -B that suppresses grub configuration change.
Solution: option -B for mkinitrd.
II.b (chrooted) grub2-mkconfig generates inadequate grub.cfg
On my chroted boxes, I could not get grub2-mkconfig to generate a usable
grub.cfg. Things related to lvm where not present in the generated
grub.cfg, especially strings expressing roots on lvm LVs where missing
completely.
Changing
use_lvmetad = 1
to =0 or back in the chroot-mounted
/etc/lvm/lvm.conf were of no use.
As the grub.cfgs produced by the Feb 2 opensuse update were not corrupt,
and as the grub2-mkconfig on the now-running boxes also works,
this was a problem of my chroot setup.
But the combination with II.a above - mkinitrd changes and corrupts
grub.cfg behind my back - did not improve my repair experience.
Solved by luck and backups of old grub.cfgs and then by the -B option in IIa.
II.c mkinitrd needs /mounts/mp_* chrooted
Still, also with -B and -f lvm2, mkinitrd did not produce a usable initrd.
Besides many messages about failing lvm info use there was a chrooted
/dev/mapper empty except for a control file linking to non-existing
files (chroot-) /mounts/mp_*.
The difference in the created initrds was rather subtle - the
-l
lvm2
feature was there, but three files where missing without
recursively mounted $CHROOTDIR/mounts/* (on the left):
640d639
< boot/11-block.sh
705d703
< config/lvm2.sh
707d704
< config/block.sh
Funny, only a big recursive bind-mount
mount -R
did the
job. I did not look deep enough to see which mounts I was missing when
using mount -B.
Lesson learned: to me, current opensuse maintains a quite complex and
error-prone device and mount structure for lvm devices.
Solution was
mount -R /mounts $CHROOTDIR/mounts
and the same
again for every /mounts/mp_*.
III Solution:
I repaired my second box using the following steps.
III.a copy following script chroot.bash to an usb stick.
#!/bin/bash
# chroot.bash - try to solve a problem after opensuse-13.1 x86_64 update from Feb 2 2015 19:00
# lipi 2015 - reusing chroot.sh of unknown origin. twimc: Thanks!
#
# call: ./chroot.bash <root dev name, e.g. systemvg/rootlv>
#
# NOTE: sorry, no warranty - but it did the trick for me.
# NOTE: the only version of initrd that's re-generated is
# initrd-3.11.10-25-desktop. If your problem is related to another
# initrd (or another problem), this script is of no help.
CHROOTDIR=/mnt/chroot
# Make sure only root can run our script
if $EUID -ne 0 ]; then
echo "This script must be run as root" 1>&2
exit 1
fi
# Make sure that $1 is not empty
if ! -n "$1" ]; then
echo -e " You must give an argument which device you want to chroot in (without leading '/dev/') "
echo -e " Example '$0 systemvg/rootlv' will chroot the device /dev/systemvg/rootlv into /mnt/chroot."
exit 1
fi
# Make sure that directory '/mnt/chroot' exists
if ! -d $CHROOTDIR ]; then
echo Error: No directory $CHROOTDIR existing. Creating it... !! >&2
mkdir -p $CHROOTDIR
fi
if -d $CHROOTDIR ]; then
echo Created directory $CHROOTDIR successfully... >&2
echo Now chrooting into /dev/$1 ... >&2
fi
if -d $CHROOTDIR ]; then
mount -v /dev/$1 $CHROOTDIR
mkdir -v -p $CHROOTDIR/mounts
sysdirs="
/dev
/mounts
/mounts/*
/proc
/sys
"
for x in $(ls -1d $sysdirs | sort) ; do
mount -v -R $x $CHROOTDIR$x
done
FSTAB=""
awk '/^\// && / ]\//' $CHROOTDIR/etc/fstab |
while read source target rest ; do
mount -v $source $CHROOTDIR$target
FSTAB="$FSTAB $target"
done
if grep 'use_lvmetad ]*= ]*1' $CHROOTDIR/etc/lvm/lvm.conf ; then
echo -e " /etc/lvm/lvm.conf: use_lvmetad is not = 1" >&2
else
echo -e " /etc/lvm/lvm.conf: changing use_lvmetad := 0" >&2
sed -i 's/\(use_lvmetad ]*= ]*\)1/\10/' $CHROOTDIR/etc/lvm/lvm.conf
fi
VERSION=3.11.10-25-desktop
echo -e " WORKING SOLELY ON VERSION $VERSION !" >&2
if ! -f $CHROOTDIR/boot/initrd-$VERSION-original ] ; then
echo -e " backup copy of $(ls -l $CHROOTDIR/boot/initrd-$VERSION)" >&2
cp -v $CHROOTDIR/boot/initrd-$VERSION $CHROOTDIR/boot/initrd-$VERSION-original
fi
# here we are on the old box!
echo -e " mkinitrd:" >&2
echo -e " generating initrd -including feature lvm2- on chrooted box; no grub.cfg change." >&2
chroot $CHROOTDIR \
/sbin/mkinitrd -k vmlinuz-$VERSION -i initrd-$VERSION -d /dev/$1 -f lvm2 -B
echo -e " bash shell on chrooted box" >&2
echo -e " if you want to look/repair on your system, do it now." >&2
echo -e " To leave this bash shell, type <enter> <ctrl>-d !" >&2
chroot $CHROOTDIR /bin/bash
echo === left chrooted box.
echo -e " unmounting everything from the chroot directory ..." >&2
umount -v -R $CHROOTDIR
echo -e " deleting chroot directory $CHROOTDIR ..." >&2
rmdir -v $CHROOTDIR
if ! -d $CHROOTDIR ]; then
echo -e "$CHROOTDIR deleted successfully. Good bye !!" >&2
fi
fi
echo -e ' now, "shutdown -h now" and restart without rescue system.' >&2
echo -e ' Good luck.' >&2
III.b rescure system
boot system bricked by Feb 2 opensuse 13.1 update from
opensuse-13.1-x86_64 install dvd into rescue system, log in as root.
III.c usb stick
stick usb stick into bricked box running rescue system.
Search for stick using
fdisk -l
.
If there is to much output, do
fdisk -l | more
and page down
by space, enter…
Look for a section like the one below, showing an /dev/sd* with the
capacity of your stick. Probably has exactly one fat/vfat/ntfs partition.
This section looked like this for me:
...much text...
Disk /dev/sde: 4009 MB, 4009754624 bytes, 7831552 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x1bf0d4df
Device Boot Start End Blocks Id System
/dev/sde1 2048 7831551 3914752 b W95 FAT32
...much text...
So I mounted that stick:
mkdir /tmp/usb
mount /dev/sde1 /tmp/usb
cd /tmp/usb
Do the same for your usb stick partition.
III.d chroot your old system
- root device
Try to remember what your old root partition was. HINT: this is the volume
that did not appear when you tried to boot…
For you, Hylke, the original poster, with the boot failing this way
- Resume device /dev/system/swap not found (ignoring)
- Waiting for device /dev/mapper/system-root to appear:............Could
not find /dev/mapper/system-root.
the root logical volume devive would be /dev/system/root.
- start chroot.bash:
In /tmp/usb, execute ./chroot.bash , for you, Hylke, e.g.
./chroot.bash system/root
IIII. umount stick, reboot
cd /
umount /tmp/usb
Remove usb stick.
shutdown -r now
This should do the trick for you.
Good luck!
lipinger
