Software Management: The dist-upgrade way

You have always been told that you should only use dist-upgrade in some specific cases (e.g. upgrade from openSUSE 11.4 to 12.1 or using Tumbleweed). But why? Well, it’s because it will do more changes than an “update”. More changes means more possibilities to break something. But if you have said to ZYpp what do you want to allow to change and what you don’t want to change the process is 100% safe. And those extra changes over a normal “update”… well, if they are not bad they must be good. Here I will try to show you how to tell ZYpp what do you want to allow to change and what you don’t want to change.

Let’s look at what the man page of zypper has to say about dist-upgrade to understand what we are going to win over a normal “update”:

This command applies the state of (specified) repositories onto the system; upgrades (or even downgrades) installed packages to versions found in repositories, removes packages that are no longer in the repositories and pose a dependency problem for the upgrade, handles package splits and renames, etc

That’s it, “applies the state of (specified) repositories onto the system”. What there is in a repository right now is what the packagers wanted to be. It’s the latest in the packaging development. What packagers wanted users to have. If what you have installed isn’t the same there is in the repository you are not really updated! The guy that created the package you have installed right now thinks you should change it for the one available now in the repository.

  • “even downgrades”
    Because the Open Build Service is a complex beast that can reference packages from a project into another project sometimes you can find that a repository has a package with a lower version than the one you installed from that same repository one week ago. A downgrade? Why I would want to downgrade?? Because that package with a lowest version (probably just a lower “release”, packages have the form <name>-<version>-<release>, e.g. zypper-1.6.21-5.14.1) is newer than the package you have!! Even if the version-release pair is lower and zypper calls it “downgrade”, it is in fact an upgrade!
    Sure, with the basic main + non-free + updates repos you will never find a downgrade (well, sometimes it happened with the updates repo, but that’s a bug), but you want to use Packman and perhaps even the games repo.

  • “removes packages that are no longer in the repositories and pose a dependency problem for the upgrade, handles package splits and renames”
    This is something that will never happen in the basic main + non-free + updates repos. An “update” is faster than a dist-upgrade ignoring these things. But if you want to use any other repo be sure packages will be removed, splitted and renamed from time to time. And yes, here I include Packman. You probably want to update your packages even if the packager changed the name for whatever reason, true?

OK. Now that we understand why we want to use dist-upgrade let’s see how to avoid breaking anything.
dist-upgrade ignores the version and release of a package, ignores the vendor, changes to a renamed or splitted package… So, what stops it? Priorities. Let’s see what the zypper man page has to say about priorities:

Priority of 1 is the highest, the higher the number the lower the priority. Default priority is 99. Packages from repositories with higher priority will be preferred even in case there is a higher installable version available in the repository with a lower priority.

“dist-upgrade” logic is really simple:
a) It will take care of any weird thing the packager did (rename, split, anything). You don’t need to care.
b) If there are multiple versions of the same package (even if with different names) in different repositories it will select the one from the repository with the highest priority (lowest number)
c) If there are multiple versions of the same package (even if with different names) in different repositories with the same priority it will select the one with the highest version-release.

This “inverted” order was selected to be compatible with YUM, the Fedora package manager, that implemented priorities before. Yum priorities must be between 1 and 99. zypper has not a maximum value, but let’s use the same range. Here is my repo configuration:


$ sudo zypper lr -P
#  | Alias               | Name                          | Enabled | Refresh | Priority
---+---------------------+-------------------------------+---------+---------+---------
 2 | local               | Local                         | Yes     | Yes     |    1    
 4 | multimedia_games    | Games with multimedia support | Yes     | Yes     |    3    
 5 | multimedia_kde      | KDE Multimedia                | Yes     | Yes     |    3    
12 | reddwarf_multimedia | RedDwarf's Multimedia         | Yes     | Yes     |    3    
13 | reddwarf_nonfree    | Non Free parts of RedDwarf    | Yes     | Yes     |    4    
11 | reddwarf            | RedDwarf                      | Yes     | Yes     |    5    
10 | packman             | Packman                       | Yes     | Yes     |   10    
 8 | opensuse_tools      | openSUSE.org Tools            | Yes     | Yes     |   25    
14 | update              | Updates                       | Yes     | Yes     |   49    
 9 | oss                 | OSS                           | Yes     | No      |   50    
 7 | non-oss             | Non OSS                       | Yes     | No      |   51    
 1 | games               | Games                         | Yes     | Yes     |   75    
 3 | multimedia_apps     | Multimedia Apps               | Yes     | Yes     |   75    
 6 | multimedia_libs     | Multimedia Libraries          | Yes     | Yes     |   75

As you can see I set the main repo with priority 50, in the middle of the range. Now I can add repositories and decide if I prefer its packages over the default ones or not.
What priority should I give to the updates repository? Well, in theory it should only include packages already available in the oss repository but with a higher version. So it should be enough to give it a priority of 50 (never below 50, or by ‘b’ updated will never be installed). So the question is: if there is a bug and an update is released with a lower release number than in the main repo, do you want to install it? It was released, so you want it. But the release number is buggy, perhaps has other problems. I decided that in such a case I want it to be installed anyway, so I give it a priority of 49.
What priority should I give to the non-oss repository? Again, in theory, there should never be any package in oss and non-oss at the same type. But if such thing happened, I decided I prefer the open source version. So, I give non-oss a priority of 51.

Now we have Packman. What priority it should have?

  • Over 49?
    In such a case if a package is available in both oss and packman the oss version will be substituted by the Packman version. No matter the version or release number.
  • Under 51?
    In such a case if a package is available in both oss and packman the packman version will never be installed.
  • 50?
    In such a case if a package is available in both oss and packman the package with the highest version-release will be selected.

Since we can add exceptions (explained later) you could decide to use a priority under 51 and then cherry pick the specific packages you want from Packman.
You never will want a priority of 50 because it would make the package selection random. It would depend on the release number, and since the release numbers from Packman and OSS are not coordinated anything could happen.
My decision is to always prefer the Packman version. So I use a priority of 10 (also preferring it over other OBS repos).

Let’s look at multimedia:apps for a repository with a lower than 51 priority. Why I did that? Well, I wanted the packages fluid-soundfont-gm and fluid-soundfont-gs from it. Only that. What would hape happened if I would have used a priority of 50? Again, randomness. And with a priority between 1 and 48? Well… this: “22 packages to upgrade, 1 to downgrade, 4 new, 4 to remove, 23 to change vendor”. It’s not something bad, but is not what I wanted. I just wanted to have fluid-soundfont-gm and fluid-soundfont-gs and to receive any updates for those packages that could appear in the future.

The last thing to keep is mind is something about exceptions I said before. Perhaps in general I want all the packages from Packman over OSS… Well, all but that single package I want to keep from OSS. How can I do that? With the locks. Here is my locks file.

$ cat /etc/zypp/locks
# Use the projectM version from multimedia:apps
type: package
match_type: regex
case_sensitive: on
solvable_name: ^(lib)?projectM
repo: packman
install_status: not-installed

# I don't want the development version of GStreamer
type: package
match_type: regex
case_sensitive: on
solvable_name: ^(gstreamer|libgst|typelib-1_0-Gst)
repo: multimedia_libs
install_status: not-installed
version: > 0.11

# Shouldn't be in Packman. No idea how to avoid them.
type: package
match_type: regex
case_sensitive: on
solvable_name: ^(dragonplayer|juk|kdemultimedia4|kio_audiocd|kmix|kscd|libkcddb4|libkcompactdisc4)
repo: packman
install_status: not-installed

# I don't think they should be in Packman. Not my call.
type: package
match_type: regex
case_sensitive: on
solvable_name: ^(libmms|(lib)?(wavpack|portaudio)|python-distribute|xine-ui|ripit|libqwt[0-9]|qwt|linphone|licq|google-perftools|libetpan|python-(pyparsing|Sphinx|xdg)|perl-(AnyEvent|Class-Factory-Util|Class-Singleton|Config-Tiny|DateTime|DateTime-Format-Builder|DateTime-Format-Strptime|DateTime-Locale|Devel-Cycle|Digest-CRC|File-Find-Rule|JSON-XS|libintl-perl|Number-Compare|PadWalker|Switch|Test-Fatal|Test-Memory-Cycle|Test-Simple|Text-Glob|Try-Tiny)(-debug(info|source))?$)
repo: packman
install_status: not-installed

You can create them with zypper, but I prefer to create it manually to be able to add those comments that explain they cause each lock is there.
I decided I prefer to use the projectM package from multimedia:apps instead of from Packman but in general I prefer Packman packages over the ones from multimedia:apps. What I did? Put Packman with a higher priority than multimedia:apps, but add an exception for projectM.

My configuration is not the one everybody will want. You can use a different one. The point is that with priorities and locks you can decide exactly what you want dist-upgrade to do. And once you do, you have a tool more powerful than a simple “upgrade”.

And just today I find an example of a downgrade in Packman

$ osc -A pm buildhist Essentials libquicktime openSUSE_12.1 x86_64
time                  srcmd5                              rev   vers-rel.bcnt
2011-10-24 20:33:01   87fd8de269069f7c03352447b044342f      7    1.2.3-58.1
2011-10-27 17:05:43   87fd8de269069f7c03352447b044342f      7    1.2.3-58.2
2011-10-30 04:39:04   87fd8de269069f7c03352447b044342f      7    1.2.3-58.3
2011-11-21 05:05:33   87fd8de269069f7c03352447b044342f      7    1.2.3-58.4
2011-11-23 01:09:44   87fd8de269069f7c03352447b044342f      7    1.2.3-58.5
2011-12-06 17:29:48   374c2d8d2c04590bdad37a294430bf9b      7    1.2.3-59.1
2011-12-27 20:45:43   374c2d8d2c04590bdad37a294430bf9b      7    1.2.3-59.2
2012-02-28 12:01:37   0a99cf478f42b1d282ff28d1814550f5      7    1.2.3-60.1
2012-02-28 15:09:00   0a99cf478f42b1d282ff28d1814550f5      7    1.2.3-60.2
2012-03-01 12:41:21   0a99cf478f42b1d282ff28d1814550f5      7    1.2.3-62.1
2012-03-02 15:50:42   fe3f6f5ceaba46430a813de4ba4b2c85      7    1.2.3-63.1
2012-03-13 13:23:01   fe3f6f5ceaba46430a813de4ba4b2c85      7    1.2.3-65.1
2012-03-26 12:14:31   fe3f6f5ceaba46430a813de4ba4b2c85      7    1.2.3-65.2
2012-03-28 00:28:48   fe3f6f5ceaba46430a813de4ba4b2c85      7    1.2.3-65.3
2012-03-30 01:07:45   4cfd645dbff677a7390003346b3bd351      7    1.2.4-66.1
2012-03-30 13:46:56   4cfd645dbff677a7390003346b3bd351      7    1.2.4-66.2
2012-04-12 10:33:38   4cfd645dbff677a7390003346b3bd351      7    1.2.4-66.3
2012-04-15 00:00:37   4cfd645dbff677a7390003346b3bd351      7    1.2.4-66.4
2012-04-18 13:03:12   4cfd645dbff677a7390003346b3bd351      7    1.2.4-54.1

The latest version of libquicktime0 is 1.2.4-54.1. But if you had 1.2.4-66.4 installed a “zypper up” will not install the latest version, a “zypper dup” does. A single day has been enough to show the problem is real.

Changing topic, notice that the dangers of having lots of repos basically disappear once you set your main repos with priority 50. You can have 10000 repos, with the default priority of 99, but not a single package from any of them will ever substitute the official packages. Those repos that you added without thinking will only be used to install extra software.

RedDwarf, this is really great information and your efforts are very much appreciated. I would like to add these definitions taken from the man page for our users to read:

zypper dist-upgrade (dup) [options]

Perform a distribution upgrade. This command applies the state of (specified) repositories onto the system; upgrades (or even downgrades) installed packages to versions found in repositories, removes packages that are no longer in the repositories and pose a dependency problem for the upgrade, handles package splits and renames, etc.

If no repositories are specified via --from or --repo options, zypper will do the upgrade with all defined repositories. This can be a problem if the system contains conflicting repositories, like repositories for two different distribution releases. This often happens if one forgets to remove older release repository after adding a new one, say openSUSE 11.1 and openSUSE 11.2.

To avoid the above trouble, you can specify the repositories from which to do the upgrade using the --from or --repo options. The difference between these two is that when --repo is used, zypper acts as if it knew only the specified repositories, while with --from zypper can eventually use also the rest of enabled repositories to satisfy package dependencies.

zypper update (up) [options] [packagename] …

Update installed packages with newer versions, where possible. This command will not update packages which would require change of package vendor unless the vendor is specified in /etc/zypp/vendors.d, or which would require manual resolution of problems with dependencies. Such non-installable updates will then be listed in separate section of the summary as “The following package updates will NOT be installed:”.

To update individual packages, specify one or more package names. You can use the ‘*’ and ‘?’ wildcard characters in the package names to specify multiple packages matching the pattern.

Great work RedDwarf and thanks so much for your help …

Thank You,

Thanks alot. This is much more informative than what can be found in the official documentation.

Best regards,
Greg

Thanks, RedDwarf. Your post is yet another example of zypper’s (mostly unused) power. Don’t know how I’m going to use this, but I will.

Following RedDwarf’s guide I configured my own priorities :

opensuse:~ # zypper lr -P
#  | Alias                         | Name                            | Enabled | Refresh | Priority
---+-------------------------------+---------------------------------+---------+---------+---------
 4 | kde_48                        | kde 48                          | Yes     | Yes     |    9    
 7 | packman.inode.at-suse         | Packman Repository              | Yes     | Yes     |   10    
 5 | libdvdcss repository          | libdvdcss repository            | Yes     | Yes     |   11    
 3 | emulators                     | emulators                       | Yes     | Yes     |   38    
 2 | download.opensuse.org-mozilla | openSUSE BuildService - Mozilla | Yes     | Yes     |   39    
11 | standard                      | kernel stable                   | Yes     | Yes     |   48    
10 | repo-11.4-update              | openSUSE-11.4 Updates           | Yes     | Yes     |   49    
 9 | repo-11.4-oss                 | openSUSE-11.4 OSS               | Yes     | No      |   50    
 8 | repo-11.4-non-oss             | openSUSE-11.4 Non-OSS           | Yes     | No      |   51    
 6 | openSUSE_11.4                 | network                         | Yes     | Yes     |   60    
 1 | contrib                       | contrib                         | Yes     | Yes     |   61

I’m using zypper dup exclusively now (previously used zupper up). Works very nice so far. I haven’t configured any custom locks as I think I don’t need them at the moment.

Best regards,
Greg