Issue during upgrade from 12.2, connection lost

English Install/Boot/Login
#1

Hello,

I launched an upgrade (https://fr.opensuse.org/SDB:System_upgrade) on a remote server.
I had launched screen before launching the upgrade.

I typed zypper dup --download “in-advance”, in order to download the packages first.

At one point, the screen became frozen. I tried to reconnect with SSH to the server but the connection doesn’t work after password prompt.

Installing: ConsoleKit-0.4.5-19.2.1 …[done]
Installing: yast2-core-3.0.3-1.1 …[done]
Installing: perl-XML-SAX-Base-1.08-7.1.2 …[done]
Installing: perl-Config-Crontab-1.33-7.1.2 …[done]
Installing: check-create-certificate-0.5-14.1.2 …[done]
Installing: yast2-dbus-server-3.0.0-2.1.3 …<100%>-]Write failed: Broken pipe
-----:~ ------$ ssh -p xxx root@xxx.xxx.xxx.xxx
Password:

When I try to connect with SSH in another window (in verbose mode), it hangs on:

debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:
debug3: packet_send2: adding 32 (len 22 padlen 10 extra_pad 64)
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 0
debug3: packet_send2: adding 48 (len 10 padlen 6 extra_pad 64)
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to xxx.xxx.xxx.xxx ([xxx.xxx.xxx.xxx]:xxx).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.

I’m really far away from the server so I cannot do anything physically. I can ask a manual reboot on Monday but do you have any idea to get the control on this server?

Do you have any idea why I cannot connect with SSH?
Is there any chance the upgrade has succeeded?
Do you think I will have some issues after a manual reboot?

Any help will be appreciated.

Thank you in advance,
Max-Henri

#2

On 2014-11-01 23:26, max scc wrote:

> I’m really far away from the server so I cannot do anything physically.
> I can ask a manual reboot on Monday but do you have any idea to get the
> control on this server?

Go there… sorry. No, I don’t know. Besides sending someone that can do
more than reboot.

I don’t believe in remote upgrades of PCs for a reason. They are simply
not ready for this.

If I were to do it, I would connect /the/ serial port of the
machine to another machine. The BIOS, GRUB, and the default
terminal would be set to the serial port. That second machine
should also be able to “reset” this machine.
Machines could be set in pairs.

If at all possible to do it that way…

> Do you have any idea why I cannot connect with SSH?

Guessing, the sshd daemon died. Maybe more.

> Is there any chance the upgrade has succeeded?

No, unless you launched it detached from a controlling terminal.

> Do you think I will have some issues after a manual reboot?

Dunno. Depends on what remains to upgrade, what was already upgraded.
You can only try…

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

#3

Hello Carlos,

Thank you for your answer.

When you say “No, unless you launched it detached from a controlling terminal.”, is “screen” a valid way to detach the upgrade from the controlling terminal?

Thank you,
Max-Henri

#4

On 2014-11-02 21:26, max scc wrote:
>
> Hello Carlos,
>
> Thank you for your answer.
>
> When you say “No, unless you launched it detached from a controlling
> terminal.”, is “screen” a valid way to detach the upgrade from the
> controlling terminal?

AFAIK, yes. Or tmux.
Then, even if ssh dies, the process continues on its own, unless it
stops for a question or some error.

But you need to ssh-in first in order to check what happened…

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

#5

Hello,

Thankfully the reboot fixed the SSH issue.
I am able to connect to the server and the main services are running correctly.

I figured out that I tried an upgrade from 12.2 directly to 13.1. Now, I know it was a mistake.
The version of the release is marked as 13.1 on the server but it is still the old kernel that is running.
The upgrade stopped on : (last lines of the zypper log)

2014-11-01 21:10:36 <1> serveurfichiers(16655) [zypp] PackageProvider.cc(providePackage):112 provided Package from cache (10643)yast2-dbus-server-3.0.0-2.1.3.i586(repo-13.1-oss) at /var/cache/zypp/packages/repo-13.1-oss/suse/i586/yast2-dbus-server-3.0.0-2.1.3.i586.rpm
2014-11-01 21:10:36 <1> serveurfichiers(16655) [zypp] RpmDb.cc(doInstallPackage):1824 RpmDb::installPackage(/var/cache/zypp/packages/repo-13.1-oss/suse/i586/yast2-dbus-server-3.0.0-2.1.3.i586.rpm,0x0000000c)
2014-11-01 21:10:36 <1> serveurfichiers(16655) [zypp++] ExternalProgram.cc(start_program):229 Executing ‘rpm’ ‘–root’ ‘/’ ‘–dbpath’ ‘/var/lib/rpm’ ‘-U’ ‘–percent’ ‘–force’ ‘–nodeps’ ‘–’ ‘/var/cache/zypp/packages/repo-13.1-oss/suse/i586/yast2-dbus-server-3.0.0-2.1.3.i586.rpm’
2014-11-01 21:10:36 <1> serveurfichiers(16655) [zypp++] ExternalProgram.cc(start_program):381 pid 31585 launched

What is the best way to recover from this situation? If there is one…
Try to continue the zypper dup? Try to zypper dup on the 12.3 then on the 13.1?

Thank you,
Max-Henri

#6

If you are doing it remotely, do this in a screen;

rpm --rebuilddb
zypper clean
zypper ref
zypper dup

There is one caveat here; 12.2->13.x includes changes in the network device name. You will no longer have eth0, you’ll have something mystical like ens33 etc. so if you are doing this remotely, on the next boot you will have a network device that is not configured and you won’t be able to connect to it.

But, there is a fix; http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

I suggest adding net.ifnames=0 as a kernel parameter in grub.

#7

On 2014-11-04 15:36, Miuku wrote:
>
> If you are doing it remotely, do this in a screen;

clone the system and try any procedure locally, over local network. Find
the problems and solutions, then do it on the real machine. Far Away
Space Mission style >:-)

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)