On our Xen host we have 3 guests, all running OpenSuse 12.1 (incl. host).
Thursday afternoon I updated guest + host via yast-online-update.
Host + 2 guests came up fine after reboot, one guest did not.
I can’t really read out from error messages where / why it goes wrong.
I get the errors from “xm console server1”, there is nothing in /var/log/messages after it is taken down for reboot.
Disk seems to be mounted correctly.
Error " Failed to open /dev/tty0" should not cause a crash after what I’ve read about it.
sulogin: cannot open console: Cannot allocate memory This sounds bad! But it does continue after that?
However, console is stalled, waited 30 minutes but console was not responding, server not reachable via network.
Complete log at the end of post.
I tried increasing memory for the guest (768 normally, increased to 2048).
I created a new guest, mounted the disk from server1 and ran fsck on the disk, it is clean.
This new guest have now replaced server1, so all our services are running as they shall with all our data in place. I don’t need to get server1 up again, but I must find out why it crashed!
Servers details:
All servers are OpenSuse 12.1 minimal install, text-only.
Xen host is on Dell Poweredge R210 II, 4GB RAM.
This guest hosts a copy of opensuse-repositories where all servers installs and updates from.
It is running apache + php & mysql, all manually compiled to directories under /nih. Splunk installed in /nih/splunk. All these apps works fine on server2, running them from server1-disk without any changes.
I compared /boot/grub/menu.lst and /etc/fstab on server1 with files on the 2 functioning guests, they are identical.
virt0:~ # xm console server1
0.132897] PCI: Fatal: No config space access function found
0.136018] Unable to read sysrq code in control/sysrq
0.181164] i8042: No controller found
0.181709] /home/abuild/rpmbuild/BUILD/kernel-xen-3.1.10/linux-3.1/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
doing fast boot
Creating device nodes with udev
Waiting for device /dev/xvda2 to appear: ok
fsck from util-linux 2.20.1
[/sbin/fsck.ext4 (1) -- /] fsck.ext4 -a -C0 /dev/xvda2
/dev/xvda2: clean, 186957/7757824 files, 24280329/31025408 blocks
fsck succeeded. Mounting root device read-write.
Mounting root /dev/xvda2
mount -o rw,acl,user_xattr -t ext4 /dev/xvda2 /root
2.768203] systemd[1]: [/etc/init.d/mysql.server:18] Failed to add LSB Provides name mysql.service, ignoring: File exists
3.369283] systemd-vconsole-setup[224]: Failed to open /dev/tty0: No such file or directory
Welcome to emergency mode. Use "systemctl default" or ^D to activate default mode.
sulogin: cannot open console: Cannot allocate memory
4.497863] systemd-vconsole-setup[282]: Failed to open /dev/tty0: No such file or directory
Welcome to emergency mode. Use "systemctl default" or ^D to activate default mode.
sulogin: cannot open console: Cannot allocate memory
5.413029] systemd-vconsole-setup[409]: Failed to open /dev/tty0: No such file or directory
5.437942] udevd[324]: 'cdrom_id --lock-media /dev/xvda' [353] terminated by signal 15 (Terminated)
Welcome to emergency mode. Use "systemctl default" or ^D to activate default mode.
sulogin: cannot open console: Cannot allocate memory
5.680932] systemd-vconsole-setup[449]: Failed to open /dev/tty0: No such file or directory
Welcome to emergency mode. Use "systemctl default" or ^D to activate default mode.
sulogin: cannot open console: Cannot allocate memory
5.893750] systemd-vconsole-setup[550]: Failed to open /dev/tty0: No such file or directory
5.932367] udevd[474]: '/etc/sysconfig/network/scripts/ifup-sysctl eth0 -o hotplug' [524] terminated by signal 15 (Terminated)
5.938571] udevd[473]: 'cdrom_id --lock-media /dev/xvdb' [522] terminated by signal 15 (Terminated)
5.939058] udevd[472]: 'cdrom_id --lock-media /dev/xvda' [523] terminated by signal 15 (Terminated)
5.940678] udevd[482]: '/etc/sysconfig/network/scripts/ifup-sysctl lo -o hotplug' [521] terminated by signal 15 (Terminated)
Welcome to emergency mode. Use "systemctl default" or ^D to activate default mode.
sulogin: cannot open console: Cannot allocate memory
6.112744] systemd-vconsole-setup[604]: Failed to open /dev/tty0: No such file or directory
Welcome to emergency mode. Use "systemctl default" or ^D to activate default mode.
sulogin: cannot open console: Cannot allocate memory
udevd[632]: '/sbin/blkid -o udev -p /dev/loop2' [655] terminated by signal 15 (Terminated)
udevd[634]: '/sbin/blkid -o udev -p /dev/loop4' [658] terminated by signal 15 (Terminated)
udevd[633]: '/sbin/blkid -o udev -p /dev/loop3' [657] terminated by signal 15 (Terminated)
Welcome to emergency mode. Use "systemctl default" or ^D to activate default mode.
udevd[637]: '/sbin/blkid -o udev -p /dev/loop7' [661] terminated by signal 15 (Terminated)
udevd[631]: '/sbin/blkid -o udev -p /dev/loop0' [654] terminated by signal 15 (Terminated)
udevd[636]: '/sbin/blkid -o udev -p /dev/loop6' [660] terminated by signal 15 (Terminated)
udevd[635]: '/sbin/blkid -o udev -p /dev/loop5' [659] terminated by signal 15 (Terminated)
sulogin: cannot open console: Cannot allocate memory
udevd[629]: '/sbin/blkid -o udev -p /dev/loop1' [656] terminated by signal 15 (Terminated)
udevd[643]: '/etc/sysconfig/network/scripts/ifup-sysctl lo -o hotplug' [663] terminated by signal 15 (Terminated)
udevd[627]: worker [630] did not accept message -1 (Connection refused), kill it
udevd[627]: worker [642] did not accept message -1 (Connection refused), kill it
udevd[627]: worker [644] did not accept message -1 (Connection refused), kill it
udevd[627]: worker [645] did not accept message -1 (Connection refused), kill it
udevd[627]: worker [646] did not accept message -1 (Connection refused), kill it
udevd[627]: worker [647] did not accept message -1 (Connection refused), kill it
udevd[627]: worker [648] did not accept message -1 (Connection refused), kill it
udevd[627]: worker [649] did not accept message -1 (Connection refused), kill it
udevd[627]: worker [650] did not accept message -1 (Connection refused), kill it
udevd[627]: worker [652] did not accept message -1 (Connection refused), kill it
16.337773] systemd-vconsole-setup[703]: Failed to open /dev/tty0: No such file or directory
Welcome to emergency mode. Use "systemctl default" or ^D to activate default mode.
sulogin: cannot open console: Cannot allocate memory
16.532622] systemd-vconsole-setup[755]: Failed to open /dev/tty0: No such file or directory
Welcome to emergency mode. Use "systemctl default" or ^D to activate default mode.
sulogin: cannot open console: Cannot allocate memory
16.782542] systemd-vconsole-setup[845]: Failed to open /dev/tty0: No such file or directory
16.804566] udevd[778]: '/etc/sysconfig/network/scripts/ifup-sysctl eth0 -o hotplug' [811] terminated by signal 15 (Terminated)
16.804932] udevd[776]: 'cdrom_id --lock-media /dev/xvda' [809] terminated by signal 15 (Terminated)
Welcome to emergency mode. Use "systemctl default" or ^D to activate default mode.
sulogin: cannot open console: Cannot allocate memory
16.977014] systemd-vconsole-setup[911]: Failed to open /dev/tty0: No such file or directory
Welcome to emergency mode. Use "systemctl default" or ^D to activate default mode.
sulogin: cannot open console: Cannot allocate memory
17.185885] systemd-vconsole-setup[996]: Failed to open /dev/tty0: No such file or directory
17.211828] udevd[940]: '/etc/sysconfig/network/scripts/ifup-sysctl lo -o hotplug' [974] terminated by signal 15 (Terminated)
17.215456] udevd[930]: 'cdrom_id --lock-media /dev/xvda' [976] terminated by signal 15 (Terminated)
17.218388] udevd[932]: '/etc/sysconfig/network/scripts/ifup-sysctl eth0 -o hotplug' [977] terminated by signal 15 (Terminated)
Welcome to emergency mode. Use "systemctl default" or ^D to activate default mode.
sulogin: cannot open console: Cannot allocate memory
17.360633] systemd-vconsole-setup[1065]: Failed to open /dev/tty0: No such file or directory
udevd[1085]: 'cdrom_id --lock-media /dev/xvda' [1127] terminated by signal 15 (Terminated)
udevd[1087]: '/etc/sysconfig/network/scripts/ifup-sysctl eth0 -o hotplug' [1126] terminated by signal 15 (Terminated)
udevd[1086]: 'cdrom_id --lock-media /dev/xvdb' [1125] terminated by signal 15 (Terminated)
udevd[1095]: '/etc/sysconfig/network/scripts/ifup-sysctl lo -o hotplug' [1124] terminated by signal 15 (Terminated)