Results 1 to 3 of 3

Thread: Installer cannot see “missing” physical volumes

  1. #1
    Join Date
    Jan 2012
    Location
    Russia :: Moscow
    Posts
    47

    Default Installer cannot see “missing” physical volumes

    My current setup successfully underwent each upgrade from Leap 42.1 to Leap 15.0, but is suddenly rejected by Leap 15.1 installer:

    The volume group /dev/vgSamLnx is incomplete because some physical volumes are missing.
    Incomplete volume groups are not visible in the Partitioner and will be deleted at the
    final step, when all the changes are performed in the system.
    The strange thing is that those volumes not only appear as fully operational in Leap 15.0, but also seem to be fine when checked manually from within Leap 15.1 installation environment. When googling by the error message text, the only result returned is the installer source code, so I suppose I may have hit some corner case. But my disk configuration is pretty simple:

    1. a pair of identical HDDs,
    2. several identical partitions on both HDDs, with GPT partition table and protective MBR,
    3. several RAID 1 arrays on top of some partitions, implemented by Linux MDRAID,
    4. single LVM volume group on top of those RAID arrays acting as physical volumes,
    5. several LVM logical volumes inside that LVM volume group.

    (There is also another HDD in the system containing Windows and an older openSUSE installation on top of its own LVM group, which is not related to the current setup.)

    I made several attempts running the offline installer and booting back to the online system, and so on, but could not get the difference in their view of disk devices. The only error I could spot in the installer logs is that GPT backup partition table was missing somehow, but fixing that had no effect on the installer. That else can be done to debug the issue?

    This is what I see in the installer environment:
    Code:
    # cat /proc/mdstat
    
    Personalities : [raid1] 
    md126 : active raid1 sdc4[1] sdb4[0]
          310383424 blocks super 1.0 [2/2] [UU]
          bitmap: 0/3 pages [0KB], 65536KB chunk
    
    md127 : active raid1 sdc2[1] sdb2[0]
          486536000 blocks super 1.0 [2/2] [UU]
          bitmap: 0/4 pages [0KB], 65536KB chunk
    
    unused devices: <none>
    Code:
    # lsblk
    
    NAME                               MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
    loop0                                7:0    0  70.5M  1 loop  /parts/mp_0000
    loop1                                7:1    0  13.8M  1 loop  /parts/mp_0001
    loop2                                7:2    0  49.4M  1 loop  /mounts/mp_0000
    loop3                                7:3    0  76.3M  1 loop  /mounts/mp_0001
    loop4                                7:4    0   4.1M  1 loop  /mounts/mp_0002
    loop5                                7:5    0   1.8M  1 loop  /mounts/mp_0003
    loop6                                7:6    0    64K  1 loop  /mounts/mp_0004
    sda                                  8:0    0 465.8G  0 disk  
    ├─sda1                               8:1    0  39.2M  0 part  
    ├─sda2                               8:2    0   752M  0 part  
    ├─sda3                               8:3    0   240G  0 part  
    ├─sda4                               8:4    0     1K  0 part  
    ├─sda5                               8:5    0   510M  0 part  
    ├─sda6                               8:6    0     4G  0 part  
    └─sda7                               8:7    0   216G  0 part  
      ├─vgMain-lvHome                  253:20   0    32G  0 lvm   
      ├─vgMain-lvRoot                  253:21   0    32G  0 lvm   
      ├─vgMain-lvTestDebian7           253:22   0    16G  0 lvm   
      ├─vgMain-lvTestSuse12            253:23   0    16G  0 lvm   
      ├─vgMain-lvTestCentOS6           253:24   0    16G  0 lvm   
      ├─vgMain-lvTestFedora19g         253:25   0    16G  0 lvm   
      ├─vgMain-lvTestFedora19x         253:26   0    16G  0 lvm   
      ├─vgMain-lvTestFreeBSD9          253:27   0    16G  0 lvm   
      ├─vgMain-lvTestWindows51         253:28   0     4G  0 lvm   
      ├─vgMain-lvTestSuse12text        253:29   0     4G  0 lvm   
      ├─vgMain-lvTestFreeBSD10         253:30   0     8G  0 lvm   
      ├─vgMain-lvTestNetBSD6           253:31   0     8G  0 lvm   
      └─vgMain-lvTestNetBSD7sparc      253:32   0     4G  0 lvm   
    sdb                                  8:16   0   1.8T  0 disk  
    ├─sdb1                               8:17   0  1003M  0 part  
    ├─sdb2                               8:18   0   464G  0 part  
    │ └─md127                            9:127  0   464G  0 raid1 
    │   ├─vgSamLnx-lvSysRoot           253:0    0    64G  0 lvm   
    │   ├─vgSamLnx-lvSysSwap           253:1    0     8G  0 lvm   
    │   ├─vgSamLnx-lvSysTemp           253:2    0    32G  0 lvm   
    │   ├─vgSamLnx-lvSysWork           253:3    0    32G  0 lvm   
    │   ├─vgSamLnx-lvUsrHome           253:4    0   320G  0 lvm   
    │   ├─vgSamLnx-lvSysHost           253:5    0     4G  0 lvm   
    │   ├─vgSamLnx-lvDevUbuntu         253:6    0    16G  0 lvm   
    │   ├─vgSamLnx-lvTmpBench          253:7    0     2G  0 lvm   
    │   ├─vgSamLnx-lvDevMCBC           253:8    0     8G  0 lvm   
    │   ├─vgSamLnx-lvDevMCBCv3007sparc 253:9    0     8G  0 lvm   
    │   ├─vgSamLnx-lvDevMCBCv3002sparc 253:10   0    76G  0 lvm   
    │   ├─vgSamLnx-lvDevMCBCv5007x64   253:11   0    16G  0 lvm   
    │   ├─vgSamLnx-lvDevAstraSpecial   253:12   0    16G  0 lvm   
    │   └─vgSamLnx-lvDevAstraSEv14     253:13   0     8G  0 lvm   
    ├─sdb3                               8:19   0     2G  0 part  
    └─sdb4                               8:20   0   296G  0 part  
      └─md126                            9:126  0   296G  0 raid1 
        ├─vgSamLnx-lvUsrHome           253:4    0   320G  0 lvm   
        ├─vgSamLnx-lvDevScientific64   253:14   0    16G  0 lvm   
        ├─vgSamLnx-lvDevFreeBSD        253:15   0    16G  0 lvm   
        ├─vgSamLnx-lvTestWindows51     253:16   0     4G  0 lvm   
        ├─vgSamLnx-lvTestSambaFS       253:17   0    16G  0 lvm   
        ├─vgSamLnx-lvTestCentOS43      253:18   0    16G  0 lvm   
        └─vgSamLnx-lvTestCentOS49      253:19   0    16G  0 lvm   
    sdc                                  8:32   0   1.8T  0 disk  
    ├─sdc1                               8:33   0  1003M  0 part  
    ├─sdc2                               8:34   0   464G  0 part  
    │ └─md127                            9:127  0   464G  0 raid1 
    │   ├─vgSamLnx-lvSysRoot           253:0    0    64G  0 lvm   
    │   ├─vgSamLnx-lvSysSwap           253:1    0     8G  0 lvm   
    │   ├─vgSamLnx-lvSysTemp           253:2    0    32G  0 lvm   
    │   ├─vgSamLnx-lvSysWork           253:3    0    32G  0 lvm   
    │   ├─vgSamLnx-lvUsrHome           253:4    0   320G  0 lvm   
    │   ├─vgSamLnx-lvSysHost           253:5    0     4G  0 lvm   
    │   ├─vgSamLnx-lvDevUbuntu         253:6    0    16G  0 lvm   
    │   ├─vgSamLnx-lvTmpBench          253:7    0     2G  0 lvm   
    │   ├─vgSamLnx-lvDevMCBC           253:8    0     8G  0 lvm   
    │   ├─vgSamLnx-lvDevMCBCv3007sparc 253:9    0     8G  0 lvm   
    │   ├─vgSamLnx-lvDevMCBCv3002sparc 253:10   0    76G  0 lvm   
    │   ├─vgSamLnx-lvDevMCBCv5007x64   253:11   0    16G  0 lvm   
    │   ├─vgSamLnx-lvDevAstraSpecial   253:12   0    16G  0 lvm   
    │   └─vgSamLnx-lvDevAstraSEv14     253:13   0     8G  0 lvm   
    ├─sdc3                               8:35   0     2G  0 part  
    └─sdc4                               8:36   0   296G  0 part  
      └─md126                            9:126  0   296G  0 raid1 
        ├─vgSamLnx-lvUsrHome           253:4    0   320G  0 lvm   
        ├─vgSamLnx-lvDevScientific64   253:14   0    16G  0 lvm   
        ├─vgSamLnx-lvDevFreeBSD        253:15   0    16G  0 lvm   
        ├─vgSamLnx-lvTestWindows51     253:16   0     4G  0 lvm   
        ├─vgSamLnx-lvTestSambaFS       253:17   0    16G  0 lvm   
        ├─vgSamLnx-lvTestCentOS43      253:18   0    16G  0 lvm   
        └─vgSamLnx-lvTestCentOS49      253:19   0    16G  0 lvm   
    sdd                                  8:48   1  28.9G  0 disk  
    ├─sdd1                               8:49   1   3.8M  0 part  
    ├─sdd2                               8:50   1   3.8G  0 part  
    └─sdd3                               8:51   1    16G  0 part  /mnt
    sr0                                 11:0    1  1024M  0 rom
    Code:
    # pvs
    
      WARNING: Failed to connect to lvmetad. Falling back to device scanning.
      PV         VG       Fmt  Attr PSize   PFree 
      /dev/md126 vgSamLnx lvm2 a--  463.99g     0 
      /dev/md127 vgSamLnx lvm2 a--  296.00g 65.99g
      /dev/sda7  vgMain   lvm2 a--  216.00g 28.00g
    
    # pvscan
    
      WARNING: Failed to connect to lvmetad. Falling back to device scanning.
      PV /dev/md126   VG vgSamLnx        lvm2 [463.99 GiB / 0    free]
      PV /dev/md127   VG vgSamLnx        lvm2 [296.00 GiB / 65.99 GiB free]
      PV /dev/sda7    VG vgMain          lvm2 [216.00 GiB / 28.00 GiB free]
      Total: 3 [975.99 GiB] / in use: 3 [975.99 GiB] / in no VG: 0 [0   ]
    
    # pvdisplay
    
      WARNING: Failed to connect to lvmetad. Falling back to device scanning.
      --- Physical volume ---
      PV Name               /dev/md126
      VG Name               vgSamLnx
      PV Size               464.00 GiB / not usable 4.81 MiB
      Allocatable           yes (but full)
      PE Size               4.00 MiB
      Total PE              118782
      Free PE               0
      Allocated PE          118782
      PV UUID               L592qS-LLAM-T2cX-fhp6-t0FN-whnr-a9OkWW
       
      --- Physical volume ---
      PV Name               /dev/md127
      VG Name               vgSamLnx
      PV Size               296.00 GiB / not usable 4.81 MiB
      Allocatable           yes 
      PE Size               4.00 MiB
      Total PE              75776
      Free PE               16894
      Allocated PE          58882
      PV UUID               8E55fv-0Vh6-ealr-6GVR-8k3m-YmUk-zBM8T1
       
      --- Physical volume ---
      PV Name               /dev/sda7
      VG Name               vgMain
      PV Size               216.00 GiB / not usable 4.00 MiB
      Allocatable           yes 
      PE Size               4.00 MiB
      Total PE              55295
      Free PE               7168
      Allocated PE          48127
      PV UUID               KhNxP6-HBOg-eICK-rmlj-pNWH-r6Gr-3a3xuQ
    See also:
    • libstorage-ng device graph (XML)
    • other logs archived:
      • /var/log contents (except irrelevant)
      • libstorage-ng log and device graph excerpts from /var/log/YaST2/y2log
      • lvm logs (pvs, pvscan, pvdisplay, vgs, vgscan, vgdisplay, lvs, lvscan, lvdisplay, vgchange, lvchange)
      • blkid, dmesg, mount

    PS. After reading recent discussions here on the forum I became aware that the system could be upgraded online. Although I had never tried such method, I would definitely prefer that. However I am afraid that under current circumstances I may end up with non-functioning system if Leap 15.1 treats the disks in the same way as its offline installer does.

  2. #2
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    12,140
    Blog Entries
    2

    Default Re: Installer cannot see “missing” physical volumes

    I'm not sure what I'm looking at,
    But I located in your /var/log the lines
    Code:
    2020-02-08 04:47:06 <1> install(4340) [libstorage] DmRaidImpl.cc:103 activate_dm_raids2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:69 constructor SystemCmd("/sbin/dmraid --activate yes --no_partitions")
    2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:188 SystemCmd Executing:"/sbin/dmraid --activate yes --no_partitions"
    2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:189 timestamp [94.756683], 2020-02-08 09:47:06 GMT, 2020-02-08 04:47:06 EST
    2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:663 Adding Line 1 "/bin/sh: /sbin/dmraid: No such file or directory"
    2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:629 pid:5480 added lines:1 stderr:true
    2020-02-08 04:47:06 <3> install(4340) [libstorage] SystemCmd.cc:495 THROW:	 Command not found: "/sbin/dmraid --activate yes --no_partitions"
    2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:419 stopwatch 0.005174s for "/sbin/dmraid --activate yes --no_partitions"
    2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:439 system() Returns:127
    2020-02-08 04:47:06 <1> install(4340) [libstorage] SystemCmd.cc:681 stderr:/bin/sh: /sbin/dmraid: No such file or directory
    Which was then followed by scanning for LVM volumes...

    Someone will know why your system intentionally mounted empty RAID volumes... and then failed because it couldn't find the dmraid command(I don't)

    Short of figuring out what the problem is (maybe it's worth submitting a bug report just to see if anyone is interested), I'd expect that a backup, rebuild including but to 15.1 and then a restore of at least your data should be your best approach.

    Is why I strongly favor virtualization so I can make instant copies and execute multiple tries if something goes wrong...
    In fact, if virtualization is something you'd consider moving towards, I'd recommend taking this opportunity to do a P2V migration, and then try to do your upgrade virtually. But, I'd recommend re-installing your Guest as a single disk and mirror only your physical disks... And make multiple copies of your Guests as backups.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  3. #3
    Join Date
    Jan 2012
    Location
    Russia :: Moscow
    Posts
    47

    Default Re: Installer cannot see “missing” physical volumes

    Quote Originally Posted by tsu2 View Post
    failed because it couldn't find the dmraid command
    Thanks for spotting that. However I did not care about DMRAID because I am using MDRAID on that machine. Nevertheless I tried running the 15.1 installer on another machine which does use DMRAID (Intel ICH10R firmware-based RAID for multi-booting Windows and Linux), and had no trouble discovering my volumes (only that they are plain partitions rather than LVM logical volumes). So it looks like this error does not affect anything, provided that DMRAID is automatically initialized elsewhere before libstorage probes take place.

    I strongly favor virtualization so I can make instant copies and execute multiple tries if something goes wrong.
    Virtual machines need a host to run on, and the system in question is that host, which needs to be updated. :-) Those numerous logical volumes you see in my logs — they all are raw storage acting as virtual HDDs for guests.

    In fact, I used virtualization, too, for reproducing the issue in a clean environment. After confirming the bug with similar configuration having 1 volume group on top of 2 arrays I further simplified it down to 1 vg on 1 mdraid and then also tried each other possible scenario: LVM only and no MDRAID, MDRAID only and no LVM, — the only troublesome combination was LVM on top of MDRAID. Therefore I submitted bug report 1164926, which quickly turned out to be a duplicate of bug 1136641 (I did not find that because of differences in classification and spelling) — an issue known immediately after the 15.1 release, attributed to “a problem not in the installer, but a buggy release of the LVM tools that, unfortunately, slipped into 15.1” — a much older issue with missing lvmetad described in bug report 1099329. The last one was dated a month after 15.0 release, so it might be pure luck that we did not hit that during 15.0 upgrade, because it was resolved just several months ago and will only be fully integrated in 15.2 release.

    Worth to note, regarding my thoughts on using online upgrade, is the following comment from bug report 1136641:
    Trying to narrow down the problem I decided to try a inline upgrade of my current system running 15.0. After reboot LVM complained, see below, rendering the system not bootable. Thanks to btrfs and snapper I could roll back.
    So if I tried that, I would render my system non-functional, too; only that I am not using BTRFS, so rolling back would not be so easy. (Of course, trying that in a virtual machine before applying to the physical host would save from such a disaster.)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •