Upgrade from Leap 15.6 to 16.0 encounters usrmerge failure

English Install/Boot/Login
1

Situation: computer with Leap 15.6, repositories newly installed (openSUSE-repoes-Leap, zypper --releasever=15.6 ref executed, zypper path --with-update and finally opensuse-migration-tool. Opted for Leap 16.0, Retrieving all packages went fine (998 of 998 retreived) and the the following error:

Retrieving: mariadb-bench-11.8.5-160000.3.1.x86_64.rpm ......................................................................................................................................[done (650.2 KiB/s)]

Checking for file conflicts: ..............................................................................................................................................................................[done]
Make a copy of `/bin'.
Merge the copy with `/usr/bin'.
cp: cannot overwrite directory '/usr/bin.usrmerge/old' with non-directory
UsrMerge conversion failed, cleaning up
!!! ATTENTION: Do NOT proceed if you see this message during
!!! distribution upgrade. Chances are high that your system might
!!! break beyond repair if you do.
error: lua script failed: [string "%prein(filesystem-84.87-160000.2.2.x86_64)"]:22: exit
error: filesystem-84.87-160000.2.2.x86_64: install failed
error: filesystem-15.0-11.8.1.x86_64: erase skipped
(   1/1111) Installing: filesystem-84.87-160000.2.2.x86_64 ...............................................................................................................................................[error]
Installation of filesystem-84.87-160000.2.2.x86_64 failed:
Error: Subprocess failed. Error: RPM failed: Command exited with status 1.
Abort, retry, ignore? [a/r/i] (a):

Choosing abort followed by reboot has as result that I need to remove and reinstall openSUSE-repos-Leap to be able to do zypper ref.

What can I do to make th eupgrade to Leap 16.0 successful?

2

Why not use the dedicated migration tool?

3

I used the dedicated migration tool : opensues-migration-tool.

4

OK, sorry, I overlooked that part.

Before you started the migration, did you do a final zypper dup on the existing Leap 15.6?

5

No, I did not execute zypper dup, I executed zypper up followed by a reboot (which was required because of a kernel update)>.

6

I just downloaded filesystem-84.87-160000.2.2.x86_64.rpm from the download repo to extract the pre-/post-install scripts so we can have a look at them:

% rpm -qp --scripts filesystem-84.87-160000.2.2.x86_64.rpm >filesystem-scripts

pretrans scriptlet (using <lua>):
os.remove ("/usr/include/X11")
os.remove ("/usr/lib/X11")
if not posix.readlink("/var/run") then
   os.rename("/var/run","/var/run.rpmsave.tmpx")
end
if not posix.readlink("/var/lock") then
   os.rename("/var/lock","/var/lock.rpmsave.tmpx")
end
if not posix.stat("/var/run") then
  posix.symlink("/run","/var/run")
end
if not posix.stat("/var/lock") then
  posix.symlink("/run/lock","/var/lock")
end
if posix.stat("/var/run.rpmsave.tmpx") then
  os.execute("mv /var/run.rpmsave.tmpx/* /var/run")
  os.remove("/var/run.rpmsave.tmpx")
end
if posix.stat("/var/lock.rpmsave.tmpx") then
  os.execute("mv /var/lock.rpmsave.tmpx/* /var/lock")
  os.remove("/var/lock.rpmsave.tmpx")
end

local ghosts = {
["/boot"] = 0555,
["/home"] = 0555,
["/mnt"] = 0555,
["/opt"] = 0555,
["/proc"] = 0555,
["/srv"] = 0555,
["/run/lock"] = 0755,
["/sys"] = 0555,
["/usr/local"] = 0755,
["/usr/local/bin"] = 0755,
["/usr/local/include"] = 0755,
["/usr/local/lib"] = 0755,
["/usr/local/libexec"] = 0755,
["/usr/local/man"] = 0755,
["/usr/local/man/man1"] = 0755,
["/usr/local/man/man2"] = 0755,
["/usr/local/man/man3"] = 0755,
["/usr/local/man/man4"] = 0755,
["/usr/local/man/man5"] = 0755,
["/usr/local/man/man6"] = 0755,
["/usr/local/man/man7"] = 0755,
["/usr/local/man/man8"] = 0755,
["/usr/local/man/man9"] = 0755,
["/usr/local/man/mann"] = 0755,
["/usr/local/sbin"] = 0755,
["/usr/local/share"] = 0755,
["/usr/local/src"] = 0755,
["/usr/local/lib64"] = 0755,
}
function mkdir_p(path)
    d = ''
    for p in string.gmatch(path, "([^/]+)") do
	d = d.."/"..p
	posix.mkdir(d)
    end
end
for i in pairs(ghosts) do
  mkdir_p(i)
  posix.chmod(i, ghosts[i])
end

local ghost_links = {
["/var/run"] = "../run",
["/var/lock"] = "../run/lock",
}
for i in pairs(ghost_links) do
  mkdir_p(i:match(".*/"))
  posix.symlink(ghost_links[i], i)
end

preinstall scriptlet (using <lua>):
needmigrate = false
local dirs = {"/bin",
  "/sbin",
  "/lib64",
  "/lib" }
for i in pairs(dirs) do
  local t = posix.stat(dirs[i], "type")
  if t == nil then
    posix.symlink("usr"..dirs[i], dirs[i])
  elseif t == "directory" then
    needmigrate = true
  end
end
if needmigrate then
  if posix.getenv("ZYPP_SINGLE_RPMTRANS") == "1" then
    print("Warning: UsrMerge executed in single transcation mode")
    if not posix.stat("/usr/lib/rpm/lua/usrmerge.lua") then
      error("ERROR: compat-usrmerge file triggers not installed.\n!!! This will go horribly wrong. You need a rescue system now !!!")
    end
    rpm.define("_filesystem_need_posttrans_convertfs 1")
  else
    assert(os.execute("/usr/libexec/convertfs"))
  end
end

posttrans scriptlet (using <lua>):
if rpm.expand("%{?_filesystem_need_posttrans_convertfs}") == "1" then
  assert(os.execute("/usr/libexec/convertfs"))
end
7

I’m sorry, but I can’t quite understand what it all means, I can say that the error messages 'and ‘cannot overwrite directory […] with non-directory’ and ‘UsrMerge conversion failed, cleaning up’ do seem to be present in the code you downloaded.

8

I am trying to figure out what exactly is going on there.

My first suspect was a missing lua interpreter, but apparently this is now part of the rpm command itself.

Then I wanted to have a look at /usr/lib/rpm/lua/usrmerge.lua that is part of the pre-install script, but I don’t have it on my Slowroll. But I found the compat-usrmerge-tools package which might be involved here.

9

In your error message, there is this:

error: lua script failed: [string "%prein(filesystem-84.87-160000.2.2.x86_64)"]:22: exit

Line 22 of that preinstall script would be this:

    assert(os.execute("/usr/libexec/convertfs"))

And we can find that message

Merge the copy with `/usr/bin’

there.

% cat /usr/libexec/convertfs

#!/bin/bash
# based on code from dracut convertfs.sh

ROOT=

is_usrmerged() {
    local r=1
    for dir in bin sbin lib lib64; do
	[ -d "$ROOT/$dir" ] || continue
	[ -L "$ROOT/$dir" ] || return 1
	r=0
    done
    return "$r"
}

# check if there's anything to do
is_usrmerged && exit 0

# the package is installed with AutoReq off, so no guarantee that
# coreutils actually works
if ! { cp --help && find --help; } > /dev/null; then
	echo "tools not functional, exit"
	exit 1
fi

# clean up after ourselves no matter how we die.
cleanup() {
    echo "UsrMerge conversion failed, cleaning up"
    for dir in bin sbin lib lib64; do
	rm -rf -- "$ROOT/usr/${dir}.usrmerge"
    done
    echo "!!! ATTENTION: Do NOT proceed if you see this message during"
    echo "!!! distribution upgrade. Chances are high that your system might"
    echo "!!! break beyond repair if you do."
}

trap 'ret=$?; [[ $ret -ne 0 ]] && cleanup;exit $ret;' EXIT
trap 'exit 1;' SIGINT

set -e

if [ "$(stat -f -c %T "${ROOT:-/}")" = "overlayfs" ]; then
	echo "UsrMerge conversion does not work on overlayfs"
	exit 1
fi

CP_HARDLINK="-l"
while read dev mp other; do
    [ "$mp" = "$ROOT/usr" ] && CP_HARDLINK=""
    for dir in bin sbin lib lib64; do
	[ -d "$ROOT/$dir" ] || continue
	if [ "${mp#$ROOT/$dir}" != "$mp" ] || [ "${mp#$ROOT/usr/$dir}" != "$mp" ]; then
	    echo "Please unmount $mp before the conversion"
	    exit 1
	fi
    done
done < /proc/mounts

# merge / and /usr in new dir in /usr
for dir in bin sbin lib lib64; do
    rm -rf -- "$ROOT/usr/${dir}.usrmerge"
    [[ -L "$ROOT/$dir" ]] && continue
    [[ -d "$ROOT/$dir" ]] || continue
    echo "Make a copy of \`$ROOT/$dir'."
    [[ -d "$ROOT/$dir" ]] \
        && cp -ax $CP_HARDLINK "$ROOT/$dir" "$ROOT/usr/${dir}.usrmerge"
    # cp can't handle copying a dir over non-directories. So move
    # those away in advance. Happened with /lib/udev existing as
    # link on older distros
    while read d; do
	f="$ROOT/usr/$dir.usrmerge/$d"
	if test -L "$f" -o \( -e "$f" -a ! -d "$f" \); then
	    echo "Warning: /$dir/$d conflicts with directory /usr/$dir/$d and will be removed"
	    rm -rf "$f.usrmerge~"
	    mv "$f" "$f.usrmerge~"
	fi
    done < <(find "$ROOT/usr/$dir" -xdev -type d -printf "%P\n" )
    echo "Merge the copy with \`$ROOT/usr/$dir'."
    [[ -d "$ROOT/usr/${dir}.usrmerge" ]] \
        || mkdir -p "$ROOT/usr/${dir}.usrmerge"
    cp -axT -l --backup --suffix=.usrmerge~ "$ROOT/usr/$dir" "$ROOT/usr/${dir}.usrmerge"
    echo "Clean up duplicates in \`$ROOT/usr/$dir'."
    # delete all symlinks that have been backed up. /usr versions
    # override / ones
    find "$ROOT/usr/${dir}.usrmerge" -xdev -type l -name '*.usrmerge~' -delete || :
    # in rare cases the symlinks may point from /usr to /, so remove
    # the link in that case
    while read file; do
	o=${file%%.usrmerge~}
	[ -L "$o" ] && mv -f "$file" "$o"
    done < <(find "$ROOT/usr/${dir}.usrmerge" \
        -xdev -name '*.usrmerge~' -type f)
done
# switch over merged dirs in /usr
for dir in bin sbin lib lib64; do
    if [ -d "$ROOT/usr/${dir}.usrmerge" ]; then
	echo "Switch to new \`$ROOT/usr/$dir'."
	/usr/libexec/xmv "$ROOT/usr/${dir}.usrmerge" "$ROOT/usr/$dir"
    fi
done

# replace dirs in / with links to /usr
for dir in bin sbin lib lib64; do
    if [ ! -L "$ROOT/$dir" -a -d "$ROOT/$dir" ]; then
	  echo "Create \`$ROOT/$dir' symlink."
	  rm --one-file-system -rf "$ROOT/${dir}.usrmerge" || :
	  ln -s usr/$dir "$ROOT/${dir}.usrmerge"
	  /usr/libexec/xmv "$ROOT/$dir" "$ROOT/${dir}.usrmerge"
    fi
done

echo "Clean up backup files."
# everything seems to work; cleanup
for dir in bin sbin lib lib64; do
    for pfx in usr/ /; do
	 # if we get killed in the middle of "rm -rf", ensure not to leave
	 # an incomplete directory, which is moved back by cleanup()
	d="$ROOT/${pfx}${dir}.usrmerge"
	if [ -d "$d" ]; then
	    echo "$d ..."
	    mv "$d" "$d~"
	    rm --one-file-system -rf "$d~"
	fi
    done
done

# XXX: confirm this is needed
for dir in lib lib64; do
    [[ -d "$ROOT/$dir" ]] || continue
    for lib in "$ROOT"/usr/${dir}/lib*.so*.usrmerge~; do
        [[ -f $lib ]] || continue
        mv -v $lib ${lib/.so/_so}
    done
done

set +e

echo "Run ldconfig."
ldconfig -r "$ROOT" || :
10

Please check if you already have a directory /usr/bin.usrmerge/old or /usr/bin.usrmerge.

If you do (which I suspect), please move it out of the way and try again.

11

I do not have a /usr/bin.usrmerge directory. Note: I aborted the migration to be certain ta have a stable and working o/s.

12

Then I fear we’ll have to wait for another pair of eyes to have a look at those scripts. Right now, I am fresh out of ideas.

13

Thank you for your effort!

1 Like
14

I /think/ I found the cause: prior to the actual migration I ran opensuse-migration-tool --dry-run to check there are no problems. It seems that the setting DRYRUN stays set a true, so that when opensuse-migration-tool is called, not everything is upgraded/installed.
My suspision arose when on another computer (I’m upgrading a bunch of them) I noticed a message like ‘dry-run completed, feel free to continue’ and got (different) problems with a non-working version 16.0.
I then reset a computer to version 15.6 and ran opensuse-migration-tool after rebooting an dthat one seems to have upgraded flawlessly.
I will try to upgrade the computers that presented problems now and if they succeed, I will file a bug report on the migration tool (or its documentation).

15

No luck on the machine with the usrmerge failure. That error persists. After Abort, the migration tool asks for a choice of selinux/apparmor/etc. and when cancelling that, I get the message about dry-run (which has not been executed since reboot).
The o/s is 15.6, but the repos are invalid (e.g. Leap_16 in a repo-path). So I had to restore the repos to version 15.6 by reinstalling them.

Conclusion: this topic is still open and awaiting help.

16

That is strange because there should not be literal version strings, but $releasever in the config.

BTW, it is always more useful to show such things with posting code. We of course believe you, but misunderstandings may arise from “story telling” where posting commands with their output will leave no doubts.

17

I agree, so I copied it from a computer which did upgrade to leap 16.0:

zypper lr -d
# | Alias                       | Name                      | Enabled | GPG Check | Refresh | Keep | Priority | Type   | URI                                                                      | Service
--+-----------------------------+---------------------------+---------+-----------+---------+------+----------+--------+--------------------------------------------------------------------------+---------
1 | openSUSE:repo-non-oss       | repo-non-oss (16.0)       | No      | ----      | ----    | -    |   99     | rpm-md | http://cdn.opensuse.org/distribution/leap/16.0/repo/non-oss/x86_64       | openSUSE
2 | openSUSE:repo-non-oss-debug | repo-non-oss-debug (16.0) | No      | ----      | ----    | -    |   99     | N/A    | http://cdn.opensuse.org/debug/distribution/leap/16.0/repo/non-oss/x86_64 | openSUSE
3 | openSUSE:repo-openh264      | repo-openh264 (16.0)      | Yes     | (r ) Yes  | Yes     | -    |   99     | rpm-md | http://codecs.opensuse.org/openh264/openSUSE_Leap_16                     | openSUSE
4 | openSUSE:repo-oss           | repo-oss (16.0)           | Yes     | (r ) Yes  | Yes     | -    |   99     | rpm-md | http://cdn.opensuse.org/distribution/leap/16.0/repo/oss/x86_64           | openSUSE
5 | openSUSE:repo-oss-debug     | repo-oss-debug (16.0)     | No      | ----      | ----    | -    |   99     | N/A    | http://cdn.opensuse.org/debug/distribution/leap/16.0/repo/oss/x86_64     | openSUSE
6 | openSUSE:repo-oss-source    | repo-oss-source (16.0)    | No      | ----      | ----    | -    |   99     | N/A    | http://cdn.opensuse.org/source/distribution/leap/16.0/repo/oss           | openSUSE
7 | trepo                       | trepo                     | No      | ----      | ----    | -    |   99     | rpm-md | http://download.opensuse.org/update/leap/15.6/oss                        |

The culprit is in repo openSUSE:repo-openh264.

18

And what makes you think this is wrong? This is the correct patch for the openh264 repo. Open the URL in a browser and see yourself.

19

I think it is wrong on a computer where the upgrade from 15.6 to 16.0 has been aborted, especially as --releasever was set to 15.6. Unfortunately, I omitted to copy the repo listing on that computer, which is now in version 15.6 with newly installed repos.

20

Your first post shows, that you seem to mix the classic upgrade method via zypper --releasever=15.6 and the opensuse-migration-tool. You don’t do that or else you have side effects. When you are using the opensuse-migration-tool, no manual version settings like you seem to have done are required.

So you have to decide if you want to use the old update method OR the opensuse-migration-tool.
Unfortunately the wiki is down atm. There is the System Upgrade SDB which explains the upgrade methods.