Installer cannot see “missing” physical volumes

SamsonovAnton · February 8, 2020, 2:44pm

My current setup successfully underwent each upgrade from Leap 42.1 to Leap 15.0, but is suddenly rejected by Leap 15.1 installer:

The volume group /dev/vgSamLnx is incomplete because some physical volumes are missing.
Incomplete volume groups are not visible in the Partitioner and will be deleted at the
final step, when all the changes are performed in the system.

The strange thing is that those volumes not only appear as fully operational in Leap 15.0, but also seem to be fine when checked manually from within Leap 15.1 installation environment. When googling by the error message text, the only result returned is the installer source code, so I suppose I may have hit some corner case. But my disk configuration is pretty simple:

a pair of identical HDDs,
several identical partitions on both HDDs, with GPT partition table and protective MBR,
several RAID 1 arrays on top of some partitions, implemented by Linux MDRAID,
single LVM volume group on top of those RAID arrays acting as physical volumes,
several LVM logical volumes inside that LVM volume group.

(There is also another HDD in the system containing Windows and an older openSUSE installation on top of its own LVM group, which is not related to the current setup.)

I made several attempts running the offline installer and booting back to the online system, and so on, but could not get the difference in their view of disk devices. The only error I could spot in the installer logs is that GPT backup partition table was missing somehow, but fixing that had no effect on the installer. That else can be done to debug the issue?

This is what I see in the installer environment:

# cat /proc/mdstat

Personalities : [raid1] 
md126 : active raid1 sdc4[1] sdb4[0]
      310383424 blocks super 1.0 [2/2] [UU]
      bitmap: 0/3 pages [0KB], 65536KB chunk

md127 : active raid1 sdc2[1] sdb2[0]
      486536000 blocks super 1.0 [2/2] [UU]
      bitmap: 0/4 pages [0KB], 65536KB chunk

unused devices: <none>

# lsblk

NAME                               MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop0                                7:0    0  70.5M  1 loop  /parts/mp_0000
loop1                                7:1    0  13.8M  1 loop  /parts/mp_0001
loop2                                7:2    0  49.4M  1 loop  /mounts/mp_0000
loop3                                7:3    0  76.3M  1 loop  /mounts/mp_0001
loop4                                7:4    0   4.1M  1 loop  /mounts/mp_0002
loop5                                7:5    0   1.8M  1 loop  /mounts/mp_0003
loop6                                7:6    0    64K  1 loop  /mounts/mp_0004
sda                                  8:0    0 465.8G  0 disk  
├─sda1                               8:1    0  39.2M  0 part  
├─sda2                               8:2    0   752M  0 part  
├─sda3                               8:3    0   240G  0 part  
├─sda4                               8:4    0     1K  0 part  
├─sda5                               8:5    0   510M  0 part  
├─sda6                               8:6    0     4G  0 part  
└─sda7                               8:7    0   216G  0 part  
  ├─vgMain-lvHome                  253:20   0    32G  0 lvm   
  ├─vgMain-lvRoot                  253:21   0    32G  0 lvm   
  ├─vgMain-lvTestDebian7           253:22   0    16G  0 lvm   
  ├─vgMain-lvTestSuse12            253:23   0    16G  0 lvm   
  ├─vgMain-lvTestCentOS6           253:24   0    16G  0 lvm   
  ├─vgMain-lvTestFedora19g         253:25   0    16G  0 lvm   
  ├─vgMain-lvTestFedora19x         253:26   0    16G  0 lvm   
  ├─vgMain-lvTestFreeBSD9          253:27   0    16G  0 lvm   
  ├─vgMain-lvTestWindows51         253:28   0     4G  0 lvm   
  ├─vgMain-lvTestSuse12text        253:29   0     4G  0 lvm   
  ├─vgMain-lvTestFreeBSD10         253:30   0     8G  0 lvm   
  ├─vgMain-lvTestNetBSD6           253:31   0     8G  0 lvm   
  └─vgMain-lvTestNetBSD7sparc      253:32   0     4G  0 lvm   
sdb                                  8:16   0   1.8T  0 disk  
├─sdb1                               8:17   0  1003M  0 part  
├─sdb2                               8:18   0   464G  0 part  
│ └─md127                            9:127  0   464G  0 raid1 
│   ├─vgSamLnx-lvSysRoot           253:0    0    64G  0 lvm   
│   ├─vgSamLnx-lvSysSwap           253:1    0     8G  0 lvm   
│   ├─vgSamLnx-lvSysTemp           253:2    0    32G  0 lvm   
│   ├─vgSamLnx-lvSysWork           253:3    0    32G  0 lvm   
│   ├─vgSamLnx-lvUsrHome           253:4    0   320G  0 lvm   
│   ├─vgSamLnx-lvSysHost           253:5    0     4G  0 lvm   
│   ├─vgSamLnx-lvDevUbuntu         253:6    0    16G  0 lvm   
│   ├─vgSamLnx-lvTmpBench          253:7    0     2G  0 lvm   
│   ├─vgSamLnx-lvDevMCBC           253:8    0     8G  0 lvm   
│   ├─vgSamLnx-lvDevMCBCv3007sparc 253:9    0     8G  0 lvm   
│   ├─vgSamLnx-lvDevMCBCv3002sparc 253:10   0    76G  0 lvm   
│   ├─vgSamLnx-lvDevMCBCv5007x64   253:11   0    16G  0 lvm   
│   ├─vgSamLnx-lvDevAstraSpecial   253:12   0    16G  0 lvm   
│   └─vgSamLnx-lvDevAstraSEv14     253:13   0     8G  0 lvm   
├─sdb3                               8:19   0     2G  0 part  
└─sdb4                               8:20   0   296G  0 part  
  └─md126                            9:126  0   296G  0 raid1 
    ├─vgSamLnx-lvUsrHome           253:4    0   320G  0 lvm   
    ├─vgSamLnx-lvDevScientific64   253:14   0    16G  0 lvm   
    ├─vgSamLnx-lvDevFreeBSD        253:15   0    16G  0 lvm   
    ├─vgSamLnx-lvTestWindows51     253:16   0     4G  0 lvm   
    ├─vgSamLnx-lvTestSambaFS       253:17   0    16G  0 lvm   
    ├─vgSamLnx-lvTestCentOS43      253:18   0    16G  0 lvm   
    └─vgSamLnx-lvTestCentOS49      253:19   0    16G  0 lvm   
sdc                                  8:32   0   1.8T  0 disk  
├─sdc1                               8:33   0  1003M  0 part  
├─sdc2                               8:34   0   464G  0 part  
│ └─md127                            9:127  0   464G  0 raid1 
│   ├─vgSamLnx-lvSysRoot           253:0    0    64G  0 lvm   
│   ├─vgSamLnx-lvSysSwap           253:1    0     8G  0 lvm   
│   ├─vgSamLnx-lvSysTemp           253:2    0    32G  0 lvm   
│   ├─vgSamLnx-lvSysWork           253:3    0    32G  0 lvm   
│   ├─vgSamLnx-lvUsrHome           253:4    0   320G  0 lvm   
│   ├─vgSamLnx-lvSysHost           253:5    0     4G  0 lvm   
│   ├─vgSamLnx-lvDevUbuntu         253:6    0    16G  0 lvm   
│   ├─vgSamLnx-lvTmpBench          253:7    0     2G  0 lvm   
│   ├─vgSamLnx-lvDevMCBC           253:8    0     8G  0 lvm   
│   ├─vgSamLnx-lvDevMCBCv3007sparc 253:9    0     8G  0 lvm   
│   ├─vgSamLnx-lvDevMCBCv3002sparc 253:10   0    76G  0 lvm   
│   ├─vgSamLnx-lvDevMCBCv5007x64   253:11   0    16G  0 lvm   
│   ├─vgSamLnx-lvDevAstraSpecial   253:12   0    16G  0 lvm   
│   └─vgSamLnx-lvDevAstraSEv14     253:13   0     8G  0 lvm   
├─sdc3                               8:35   0     2G  0 part  
└─sdc4                               8:36   0   296G  0 part  
  └─md126                            9:126  0   296G  0 raid1 
    ├─vgSamLnx-lvUsrHome           253:4    0   320G  0 lvm   
    ├─vgSamLnx-lvDevScientific64   253:14   0    16G  0 lvm   
    ├─vgSamLnx-lvDevFreeBSD        253:15   0    16G  0 lvm   
    ├─vgSamLnx-lvTestWindows51     253:16   0     4G  0 lvm   
    ├─vgSamLnx-lvTestSambaFS       253:17   0    16G  0 lvm   
    ├─vgSamLnx-lvTestCentOS43      253:18   0    16G  0 lvm   
    └─vgSamLnx-lvTestCentOS49      253:19   0    16G  0 lvm   
sdd                                  8:48   1  28.9G  0 disk  
├─sdd1                               8:49   1   3.8M  0 part  
├─sdd2                               8:50   1   3.8G  0 part  
└─sdd3                               8:51   1    16G  0 part  /mnt
sr0                                 11:0    1  1024M  0 rom

# pvs

  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  PV         VG       Fmt  Attr PSize   PFree 
  /dev/md126 vgSamLnx lvm2 a--  463.99g     0 
  /dev/md127 vgSamLnx lvm2 a--  296.00g 65.99g
  /dev/sda7  vgMain   lvm2 a--  216.00g 28.00g

# pvscan

  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  PV /dev/md126   VG vgSamLnx        lvm2 [463.99 GiB / 0    free]
  PV /dev/md127   VG vgSamLnx        lvm2 [296.00 GiB / 65.99 GiB free]
  PV /dev/sda7    VG vgMain          lvm2 [216.00 GiB / 28.00 GiB free]
  Total: 3 [975.99 GiB] / in use: 3 [975.99 GiB] / in no VG: 0 [0   ]

# pvdisplay

  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  --- Physical volume ---
  PV Name               /dev/md126
  VG Name               vgSamLnx
  PV Size               464.00 GiB / not usable 4.81 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              118782
  Free PE               0
  Allocated PE          118782
  PV UUID               L592qS-LLAM-T2cX-fhp6-t0FN-whnr-a9OkWW
   
  --- Physical volume ---
  PV Name               /dev/md127
  VG Name               vgSamLnx
  PV Size               296.00 GiB / not usable 4.81 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              75776
  Free PE               16894
  Allocated PE          58882
  PV UUID               8E55fv-0Vh6-ealr-6GVR-8k3m-YmUk-zBM8T1
   
  --- Physical volume ---
  PV Name               /dev/sda7
  VG Name               vgMain
  PV Size               216.00 GiB / not usable 4.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              55295
  Free PE               7168
  Allocated PE          48127
  PV UUID               KhNxP6-HBOg-eICK-rmlj-pNWH-r6Gr-3a3xuQ

See also:

libstorage-ng device graph (XML)
other logs archived:
[LIST]
/var/log contents (except irrelevant)
libstorage-ng log and device graph excerpts from /var/log/YaST2/y2log
lvm logs (pvs, pvscan, pvdisplay, vgs, vgscan, vgdisplay, lvs, lvscan, lvdisplay, vgchange, lvchange)
blkid, dmesg, mount

[/LIST]
PS. After reading recent discussions here on the forum I became aware that the system could be upgraded online. Although I had never tried such method, I would definitely prefer that. However I am afraid that under current circumstances I may end up with non-functioning system if Leap 15.1 treats the disks in the same way as its offline installer does.

tsu2 · February 10, 2020, 5:17pm

I’m not sure what I’m looking at,
But I located in your /var/log the lines

2020-02-08 04:47:06 <1> install(4340) [libstorage] DmRaidImpl.cc:103 activate_dm_raids2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:69 constructor SystemCmd("/sbin/dmraid --activate yes --no_partitions")
2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:188 SystemCmd Executing:"/sbin/dmraid --activate yes --no_partitions"
2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:189 timestamp [94.756683], 2020-02-08 09:47:06 GMT, 2020-02-08 04:47:06 EST
2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:663 Adding Line 1 "/bin/sh: /sbin/dmraid: No such file or directory"
2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:629 pid:5480 added lines:1 stderr:true
2020-02-08 04:47:06 <3> install(4340) [libstorage] SystemCmd.cc:495 THROW:	 Command not found: "/sbin/dmraid --activate yes --no_partitions"
2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:419 stopwatch 0.005174s for "/sbin/dmraid --activate yes --no_partitions"
2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:439 system() Returns:127
2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:681 stderr:/bin/sh: /sbin/dmraid: No such file or directory

Which was then followed by scanning for LVM volumes…

Someone will know why your system intentionally mounted empty RAID volumes… and then failed because it couldn’t find the dmraid command(I don’t)

Short of figuring out what the problem is (maybe it’s worth submitting a bug report just to see if anyone is interested), I’d expect that a backup, rebuild including but to 15.1 and then a restore of at least your data should be your best approach.

Is why I strongly favor virtualization so I can make instant copies and execute multiple tries if something goes wrong…
In fact, if virtualization is something you’d consider moving towards, I’d recommend taking this opportunity to do a P2V migration, and then try to do your upgrade virtually. But, I’d recommend re-installing your Guest as a single disk and mirror only your physical disks… And make multiple copies of your Guests as backups.

TSU

SamsonovAnton · March 1, 2020, 11:36am

Thanks for spotting that. However I did not care about DMRAID because I am using MDRAID on that machine. Nevertheless I tried running the 15.1 installer on another machine which does use DMRAID (Intel ICH10R firmware-based RAID for multi-booting Windows and Linux), and had no trouble discovering my volumes (only that they are plain partitions rather than LVM logical volumes). So it looks like this error does not affect anything, provided that DMRAID is automatically initialized elsewhere before libstorage probes take place.

I strongly favor virtualization so I can make instant copies and execute multiple tries if something goes wrong.
Virtual machines need a host to run on, and the system in question is that host, which needs to be updated. Those numerous logical volumes you see in my logs — they all are raw storage acting as virtual HDDs for guests.

In fact, I used virtualization, too, for reproducing the issue in a clean environment. After confirming the bug with similar configuration having 1 volume group on top of 2 arrays I further simplified it down to 1 vg on 1 mdraid and then also tried each other possible scenario: LVM only and no MDRAID, MDRAID only and no LVM, — the only troublesome combination was LVM on top of MDRAID. Therefore I submitted bug report 1164926, which quickly turned out to be a duplicate of bug 1136641 (I did not find that because of differences in classification and spelling) — an issue known immediately after the 15.1 release, attributed to “a problem not in the installer, but a buggy release of the LVM tools that, unfortunately, slipped into 15.1” — a much older issue with missing lvmetad described in bug report 1099329. The last one was dated a month after 15.0 release, so it might be pure luck that we did not hit that during 15.0 upgrade, because it was resolved just several months ago and will only be fully integrated in 15.2 release.

Worth to note, regarding my thoughts on using online upgrade, is the following comment from bug report 1136641:

Trying to narrow down the problem I decided to try a inline upgrade of my current system running 15.0. After reboot LVM complained, see below, rendering the system not bootable. Thanks to btrfs and snapper I could roll back.
So if I tried that, I would render my system non-functional, too; only that I am not using BTRFS, so rolling back would not be so easy. (Of course, trying that in a virtual machine before applying to the physical host would save from such a disaster.)

jmrm1 · August 13, 2020, 4:50pm

I had the same problem with an attempted 15.0 to 15.1 upgrade on a very similar system setup and previous version history. In the end, I risked updating directly from 15.0 to 15.2 and this seems to have worked just fine but might not be so safe on a system with more packages installed - mine is just acting as a host for virtual machines. It does however seem to prove the theory that the problem is a bug in the 15.1 installer not some misconfiguration of the 15.0 system that’s being upgraded.