Leap 15.2 XEN DomU paravirt Problem Kernel panic

Hello!
mayby i miss something?
I’m unable to install a new paravirt domu on a Leap 15.2 (5.3.18-lp152.57-default) XEN dom0
fullvirt of 15.2 Leap works - but paravirt stops with a
“…Oops: 0000 #1] SMP NOPTI … Kernel panic … Rebooting in 90 sec.”

15.2 New Installation fails and also distro Upgrade from 15.1 stops in boot / boot loop 100% cpu

Before creating a bugreport - should this work? (anybody out there?) or should i wait?
or is xen/paravirt out/depricated ?
(- i found no bug reports -)

Additional infos:
the dom0 machine is a ASUS P11C-M Xeon 12 Core / 32GB / System on 1TB Samsung pro nvme with Leap 15.2
with 3x full (debian 10) and 5x para (SuSE 42.3 - 15.1) domUs -
all machines runs nicely and solid rock -

Installation files comes e.g. from internal apache server via http / iso 15.2 DVD openSUSE-Leap-15.2-DVD-x86_64.iso (Jun 2020)

any suggestions appreciated, and healthiness!!

  1. Make sure your HostOS is fully updated by running the following command with elevated permissions
zypper up
  1. Make sure your Guests are also fully updated, the same command can be run in openSUSE Guests, use whatever is appropriate in your GuestOS. Of course, this will generally mean that the Guest is bootable…

Report back if you still have problems,
Or, you can submit a bug to https://bugzilla.opensuse.org

TSU

thx -
but as i wrote even a fresh installation stops with ‘same’ problem.
Prior starting distribution upgrade: installing all sub-version patches! should standard. :wink:

I’ll check distro upgrade with (likely) 3 different Leap 15.1 machines and some times with fresh installations.

My ‘standard’ upgrade and installation process do not longer work.
Only fullwirt domU’s run.

maybe i wait -

10 min are likely short -

shure - never upgrade bevor update :wink:
the dom0 updated (kernel 04.12.20) and also the domU’s.
There is no newer 15.2 iso like 06.2020 ? (am i right?)

the md5 from http://download.opensuse.org/distribution/leap/15.2/iso/openSUSE-Leap-15.2-DVD-x86_64-Current.iso.mirrorlist
called current is identical to mine -

so i’ll wait.

I’ll try the test again after upgrading the xen server
opensuse: Linux xen5 5.3.18-lp152.63-default #1 SMP Mon Feb 1 17:31:55 UTC 2021

restart: all doum-u’s (7x active, 8x standby) running nice and very fine.

Than clone again one (solid running) 15.1 opensuse:
Linux XXXX 4.12.14-lp151.28.91-default #1 SMP Tue Jan 12 23:05:46 UTC 2021 (e284d31) x86_64 x86_64 x86_64 GNU/Linux

and run distribution upgrade from 15.1 (no more patches)

to 15.2
(changing repos, zypper --gpg-auto-import-keys ref …)

“zypper dup” runs with 1113 packages and - without errors.

  • ended with the standard:
    “Ein Neustart wird benötigt, um sicher zu stellen, dass ihr System von diesen Updates profitiert.”

a reboot fails with a core dump

-> Oops: 0000 #1] SMP NOPTI

Grub bootscreen works (textmode only) testing some ‘Advanced options…’ but
“openSUSE Leap 15.2, with Linux 5.3.18-lp152.63-default (recovery mode)” stops in Kernel panic

starting fallback advanced boot option with grub - “Kernel 4.12.14-lp151.28.83-default” - works

since there are no more 15.1 patches(?), it seems i’ll have no chance to upgrade a
paravirt 15.1 to 15.2?

Probably some new/changed features take problems i can switch off / must change?
or some kernel params?
any suggestions appreceated!

additional informations:

same xen server, another dom-u with opensuse 15.2 (fullvirt) today patched
running and patchable fine.

maybe i miss something?
like ‘xen paravirt’ deprecated? or xen deprecated? or something like?

Thanks!

The information you’re posting is not clear.
Correct the following if not accurate…

Dom0 has been successfully upgraded from 15.1 to 15.2.
You are now attempting to upgrade paravirtualized DumU instances from 15.1 to 15.2 but are unsuccessful.

Did you install xen with libvirt, and did you install using the YaST virtualization installation module, or did you install Xen as packages from the Software Manager or did you do something else?

Nothing relevant to to what you’re doing has likely been deprecated.

TSU

>>Dom0 has been successfully upgraded from 15.1 to 15.2.
ack
even on a plain new 15.2 XEN Server i got the same problem

>>You are now attempting to upgrade paravirtualized DumU instances from 15.1 to 15.2 but are unsuccessful.
ack
it seems the upgrade works without any problems and no errors

  • maybe there is a Kernel/Driver/Xen/virtualisation/Bios/Chip/CPU inconsistence due to new features?

In the last years problems solved by … wating… :wink: or boot params lines like acpi=no disabling barely new CPU Board features.
(so i change from AMD to Intel to prevent power-state hangs)

but the mass of possible permutations makes me mad -

and:

  • Upgrading fullVirt 15.1. DomUs to 15.2 works
  • new installed paravirt 15.2 stops with OOPS

>>Did you install xen with libvirt, and did you install using the YaST …
the XEN DomO Server is a ‘yast-system-conform’ :wink: fully and only yast-installed system, with no additional or not opensuse.org repositories nor additional tools

>>Nothing relevant to to what you’re doing has likely been deprecated.
:wink: a real helpfull information - many thanks!

Your combination of issues is beyond what I can offer suggestions.

I recommend you ask your question on the Virtualization mail list, more Xen users tend to congregate there than in other help channels.
The following is the archive page for the Virtualization mail list, there are subscribe/unsubscribe links in top of the navigation pane to the left.
https://lists.opensuse.org/archive/opensuse-virtual/

Although there have been times in the past when Xen was undergoing change, I’m not aware of any current disruptive changes, so I doubt simply waiting is likely to address issues.

Good Luck,
TSU

i’ll check
thanks!

Seems to be a kind of deadlock -
the opensue Maillist archive stops in march 2020 - I found no informations about paravirt 15.2 upgrade
the update stops on 15.1 and the upgrade procedure from 15.1 to 15.2 have no new patches.
It seems i have no chance to upgrade the pv 15.1 to 15.2 nor 15.3 -

But i made two new Dom-U Installations on the
Dom0 Leap 15.2 5.3.18-lp152.66-default

a 15.3 beta fullwirt -> works

And:
a 15.3 paravirt -
with the same Oops:


    4.091626] systemd[1]: Started Journal Service.
    4.125250] systemd-journald[336]: Received client request to flush runtime journal.
    4.224568] input: PC Speaker as /devices/platform/pcspkr/input/input0
    4.251696] xen_netfront: Initialising Xen virtual ethernet driver
    4.265339] Console: switching to colour frame buffer device 100x37
    4.265856] printk: console [tty0] disabled
    4.265859] printk: console [tty0] enabled
    4.275263] cryptd: max_cpu_qlen set to 1000
    4.311487] AVX2 version of gcm_enc/dec engaged.
    4.311487] AES CTR mode by8 optimization enabled
    4.331505] input: Xen Virtual Keyboard as /devices/virtual/input/input1
    4.331739] input: Xen Virtual Pointer as /devices/virtual/input/input2
    4.579821] BUG: unable to handle page fault for address: ffffc90040243818
    4.579846] #PF: supervisor read access in kernel mode
    4.579861] #PF: error_code(0x0000) - not-present page
    4.579876] PGD 7dbe0067 P4D 7dbe0067 PUD fd34067 PMD 7d5b3067 PTE 0
    4.579895] Oops: 0000 #1] SMP NOPTI
    4.579908] CPU: 1 PID: 401 Comm: systemd-udevd Tainted: G                  N 5.3.18-47-default #1 SLE15-SP3
    4.579936] RIP: e030:pmc_core_probe+0x139/0x390 [intel_pmc_core]

the different to pv 15.2 Oops:
the pv 15.3 beta do not stop the boot process and go on through

The problem is:
due to certificates, encryption, must have https and the standard ‘always patch’ paradicmas there is no longer the
option to let good running systems alive - even if they running stabil and solid as a rock?

everything new? - uff…

maybe there is a kernel grub boot param in 15.2 to let the boot process go throug the Oops (yes…outch)
and then try to patch to 15.3? hopefully the oops will patched than later in 15.3 …
this feels like toothache :wink:

Any suggestions?

!!!Stay healthy!!!

Seems to be a driver memory problem in intel_pmc_core?
after a lot tests with bios switches and permutations, updating Dom0 and DomU
with no succsess i found:

https://github.com/QubesOS/qubes-issues/issues/6052

heureka! and can confirm: a VM with 4096M will not crash.

a VM with 4000M will OOPs

This problem persist in OpenSuse 15.3 rc with 5.3.18-57-default
tested today at 15:00 cest

seems problem still resist in Leap 15.3 Kernel 5.3.18-57-default.
Kernel OOps with less 4G RAM -

current tested with fresh PV at 11:00 cest