vserver upgrade from 42.1 to 42.2 FAILED, vps boot only to "emergency mode"

Hi all.

after a first upgrade of my Leap 42.1 box at home, with some problems but all solvable in some hours, I decided to upgrade also my small private vserver.

  • followed the SDB:Distribution Upgrade

  • 42.1 was up to date, all patches etc.

  • disabled all existing 42.1 repos

  • added the basic four OSS and non-OSS OpenSuse repos

  • plus the 42.2 server:php:applications repo

  • the zypper dup itself run pretty fast and produced NO errors.

BUT after reboot the vserver boots only to emergency mode

I can login, but CANNOT run the suggested “journalctl -xb” as within the VNC session at that time I can’t type a “-” (dash).

It’s a nightmare, and any ideas would be highly appreciated!

Thanks in advance,
Michael

If this is really a vps, you have to identify the vps technology…
Although generally most can manage an upgrade without a problem, there are a few which might require special steps.

TSU

I’m a step further. Good luck remembered that for this vserver a “rescue system” is available. Gentoo, as far as I’ve seen. Booted the rescue system, and now have access to all (virtual) disks of my “original” vserver. Mounted all native partitions and all (LVM) logical volumnes.

I now have access to the file system in an environment where I can type and use all characters :slight_smile:

Browsing through /var/log/messages, I can see that these reboots to the emergency system haven’t written anything to /var/log/messages.

Can anyone please tell me where to find the boot log for the boot into “emergency mode”?

The hosting company makes a great secret out of it, no official answer. Only thing clear is that’s all KVM based. as I’ve access now to the file system, where can i find out, if possible?

Also maybe worth mentioning: During boot, early stage, a line is shown pretty long:
" ***} A start job is running for dev-sdb2.device (#s / 1min 30s)"

As far as i remember sdb2 is the swap partition.

I was finally able after the hosting company changed the keyboard layout of the (KVM) VNC session to get the journalctl log for the vps: http://pastebin.com/1qLnw2pg

As I’m not very familiar with these logs = i can’t find anything except a mount issue: could someone please be so kind and take a look into this logfile and give me some hints where to look for?

Thanks in advance,
Michael

found a “workaround”.

  • mount the logical volume “/dev/vg_vserver/lv_mail” to somewhere else, temporarily.
  • move all content from file system on /dev/vg_vserver/lv_mail to /var/mail
  • commented out fstab entry for “/dev/vg_vserver/lv_mail”

*** NOT *** sophisticating as I want to know the underlying root cause and my workaround was only possible because of (still) plenty of disk space on all relevant partitions.

Yeah,
One of “those” cloud services.

IMO anyone who deploys in a cloud service should clearly define and understand what can and can’t be done.
And, the service has a duty to either provide you with the tools to do your own troubleshooting or they should be willing to do it themselves.

A big hint for these kinds of services is the source of the virtual machine…
They might provide one for convenience, but find out what it would take for you to upload and deploy your own machine to their service… If it can be done, then you have a certain measure of control. If you can’t or it’s extremely difficult, then walk away.

As for your posted journal, I haven’t had to try to read a file, but if the following works, then you might be able to use journalctl commands. Else, you can try grepping and otherwise parsing for certain types of log entries but even then your results might be uncertain and depend on logging levels (but can be tried. See various journalctl articles on how different error levels are identified).
http://unix.stackexchange.com/questions/199988/howto-inspect-systemd-system-journal-from-another-system

Can’t you be given access to run journalctl directly in your vps? If so, then you can read and parse the journal directly using journalctl commands.

Good luck,
TSU

Agree more or less, more more :slight_smile: At my hosting company, you can’t upload your own machine, would be cool, totally agree. But they provide a very wide range of linux base images, like Fedora, Cent OS, Debian, Ubuntu, and, a pity but rarely available, OpenSuse. Which of course was the reason for me to choose them :slight_smile: All distros in several releases.

I can do inside my vps whatever I want. If not illegal (e.g. spamming). So of course I usually can run journalctl directly on my vps. In theory, reality was, that Emergency Mode means no network, and the only access point left was the VNC based access from the physical machine hosting my vps. This vnc access is funny, you can see all boot menus etc., long before OpenSuse or in fact Grub2 starts. BUT it has had a very strange keyboard layout at first, e.g. no “-” on all keys (tested ALL, believe me ;-)), very bad idea for any flags to be passed to any command. I’ve workaounded this with using the second chance for access, the rescue system. If started, another virtual machine is started, same IPv4, Gentoo in this case, very basic, but you can access = mount all of your virtual disk files. Doing this I’ve created scripts for all commands I later on, again im my “own” vps within Emergency Mod, could finally execute. Painful, but necessary during the night, with no support available (8:00 to 23:00).

Next morning my hosting company’s support immediately reacts, found their mistake with the strange keyboard layout and switched at once.
From then on it was not that much trouble anymore and purely an OpenSuse-related issue.

And that is what I still want to find out:

**Why does a LVM schema which worked totally fine on 42.1 refuses to work on 42.2? ** Leading me into this painful Emergency Mode “experience” :frowning:

I’ve opened an bug for this (https://bugzilla.opensuse.org/show_bug.cgi?id=1011127), let’s wait and see.

Thanks for your help, tsu2!!!