Repo-oss repository gpg key failure

Trying to do an update, but …
Looking for gpg keys in repository repo-oss.
gpgkey=http://cdn.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.key
Retrieving repository ‘repo-oss’ metadata ------------------------------------------------------------------------------[|]
Warning: Digest verification failed for file ‘393b28216cc5bcb72ad100b437c7ec3577403989c5f5b5f3e00d71ce1f81e3d7c602edbca9594e68a99a63b8a04379503c6cbc69eaafbd7a690c5c5ac7572605-appdata.xml.gz’
[/var/tmp/AP_0xV8eOQp/repodata/393b28216cc5bcb72ad100b437c7ec3577403989c5f5b5f3e00d71ce1f81e3d7c602edbca9594e68a99a63b8a04379503c6cbc69eaafbd7a690c5c5ac7572605-appdata.xml.gz]

expected 393b28216cc5bcb72ad100b437c7ec3577403989c5f5b5f3e00d71ce1f81e3d7c602edbca9594e68a99a63b8a04379503c6cbc69eaafbd7a690c5c5ac7572605
but got efb594f4bd6fb0ed23a06a3a7c0fe9fa8e0b86e63b3e6c6de709df03a3e64579ae733c854e73865de3a688a8599f30bcb22d3a9acfb4b7e742d23bad8e9e1dd6

This was fine Thursday but started yesterday

1 Like

Usually this occurs when a mirror is not synced or in the mid of syncing. Simply wait some time and retry.

Digging deeper, the question perhaps is if the http://cdn.opensuse.org should be used at all? They seem to duplicate the http://download.opensuse.org ‘Main…’ ones and are missing on one of the servers but the ‘Main…’ ones are missing on the desktop machine. All were upgraded using the same USB stick originally, but from earlier openSUSE installs.
Just what should be my base set of repositories today?

Both are load balancers/redirectors. The cdn repos are managed by a service which are provided by the openSUSE-repos-xxxx packages.
The services are good for beginners as they are not easily destroyable.

Both load balancers/redirectors serve the same purpose. Decide for one…doesn’t matter.

Been 24 hours now and that was why I did not ask yesterday … as I said, what set should a standardise on across the three machines? Fairly sure I don’t need both
http://download.opensuse.org/tumbleweed/repo/non-oss/ and /tumbleweed/repo/oss - openSUSE Download which are both listed on the main server and the old server has a much tidier set of four download ones which seem to be fine.

The cdn.o.o repos get added, when the system detects Nvidia hardware. So if you are annoyed by a double set off repos, simply remove the d.o.o ones. It is easy…

There is no “standard” in terms of cdn.o.o or d.o.o. As already explained, both are download redirectors/load balancers. The cdn.o.o ones get added automatically on Nvidia machines to be able to install drivers.

ich@rennsemmel:~> zypper lr -d
#  | Alias                      | Name                                   | Enabled | GPG Check | Refresh | Keep | Priority | Type   | URI                                                                            | Service
---+----------------------------+----------------------------------------+---------+-----------+---------+------+----------+--------+--------------------------------------------------------------------------------+---------
 1 | NVIDIA:repo-non-free       | repo-non-free                          | Ja      | (r ) Ja   | Ja      | -    |   99     | rpm-md | https://download.nvidia.com/opensuse/tumbleweed                                | NVIDIA
 2 | openSUSE:repo-non-oss      | repo-non-oss                           | Ja      | (r ) Ja   | Ja      | -    |   99     | rpm-md | http://cdn.opensuse.org/tumbleweed/repo/non-oss                                | openSUSE
 3 | openSUSE:repo-openh264     | repo-openh264                          | Ja      | (r ) Ja   | Ja      | -    |   99     | rpm-md | https://codecs.opensuse.org/openh264/openSUSE_Tumbleweed                       | openSUSE
 4 | openSUSE:repo-oss          | repo-oss                               | Ja      | (r ) Ja   | Ja      | -    |   99     | rpm-md | http://cdn.opensuse.org/tumbleweed/repo/oss                                    | openSUSE
 5 | openSUSE:repo-oss-debug    | repo-oss-debug                         | Nein    | ----      | ----    | -    |   99     | N/A    | http://cdn.opensuse.org/debug/tumbleweed/repo/oss                              | openSUSE
 6 | openSUSE:repo-oss-source   | repo-oss-source                        | Nein    | ----      | ----    | -    |   99     | N/A    | http://cdn.opensuse.org/source/tumbleweed/repo/oss                             | openSUSE
 7 | openSUSE:update-tumbleweed | update-tumbleweed                      | Ja      | (r ) Ja   | Ja      | -    |   99     | rpm-md | http://cdn.opensuse.org/update/tumbleweed                                      | openSUSE
 8 | repo-debug                 | openSUSE-Tumbleweed-Debug              | Nein    | ----      | ----    | -    |   99     | N/A    | http://download.opensuse.org/debug/tumbleweed/repo/oss/                        | 
 9 | repo-source                | openSUSE-Tumbleweed-Source             | Nein    | ----      | ----    | -    |   99     | N/A    | http://download.opensuse.org/source/tumbleweed/repo/oss/                       | 
ich@rennsemmel:~> 

The above is the “standard” set of repositories, provided by the two packages openSUSE-repos-Tumbleweed-NVIDIA and openSUSE-repos-Tumbleweed.

OK that is making sense now since the old server does not have Nvidia graphics … and having deleting the cdn ones, they just reappear again and screw up the refresh currently. Leave until Monday and see what happens then, but irritating that third party packages have too much control :frowning:

As the cdn.o.o repos are provided by services, you can’t simply remove them. You need to remove the relevant packages which provides the services:
sudo zypper rm openSUSE-repos-Tumbleweed-NVIDIA openSUSE-repos-Tumbleweed

This is nonsense. openSUSE-repos-Tumbleweed-NVIDIA is an official repository definition provided by openSUSE. It is only there to add the Nvidia repos. In the past many new users complained and whined, that with other distributions, the drivers get installed automatically, but not with openSUSE. Now the openSUSE devs decided to ease the driver installation, and it is again not right for some users…
With recent hardware, the open drivers get installed. So where is the issue at all? If you don’t want to use specific hardware, don’t buy it in the first place…

I have a similar problem:

$ sudo zypper up                                                                                                                                                                  
Looking for gpg keys in repository openSUSE-Tumbleweed-Oss.
  gpgkey=http://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.key
Retrieving repository 'openSUSE-Tumbleweed-Oss' metadata -----------------------------------------------------------------------------------------------------------------------------------[|]
Warning: Digest verification failed for file '393b28216cc5bcb72ad100b437c7ec3577403989c5f5b5f3e00d71ce1f81e3d7c602edbca9594e68a99a63b8a04379503c6cbc69eaafbd7a690c5c5ac7572605-appdata.xml.gz'
[/var/tmp/AP_0xZE5B7l/repodata/393b28216cc5bcb72ad100b437c7ec3577403989c5f5b5f3e00d71ce1f81e3d7c602edbca9594e68a99a63b8a04379503c6cbc69eaafbd7a690c5c5ac7572605-appdata.xml.gz]

  expected 393b28216cc5bcb72ad100b437c7ec3577403989c5f5b5f3e00d71ce1f81e3d7c602edbca9594e68a99a63b8a04379503c6cbc69eaafbd7a690c5c5ac7572605
  but got  180d6555f19d17a2c0686db19e4cd13d41417763258b72d7e3cb04e7839596a25080b40f355abee89810324bf2d42e744ddb3773b1e49ecd93def75ca71bc046

Accepting packages with wrong checksums can lead to a corrupted system and in extreme cases even to a system compromise.

However if you made certain that the file with checksum '180d..' is secure, correct
and should be used within this operation, enter the first 4 characters of the checksum
to unblock using this file on your own risk. Empty input will discard the file.

Unblock or discard? [180d/...? shows all options] (discard):

zypper lr -d says I’m not using the CDN subdomain:

5 | repo-oss            | openSUSE-Tumbleweed--> | Yes     | ( p) Yes  | Yes     | -    |   99     | N/A    | http://download.opensuse.org/tumbleweed/repo/oss/                       |

Oddly enough the digest verification seems different every time. I’ve gotten:

  • 96a0473f1d4583b8cac24df55fe9f00799744f17c2cca810c795f8c49e06e1a4efdaf372cf0332fbd7975eff0541509913c9d905dea01b6c911cb93aaae234c9
  • 493b2de65e3991bd254974c8ce1cb98ac6ab2a11e9762c95298d64edb66d0a17d8c58c011a65e6545535266bba2c800905c95add57e586fa38e9531227bf4e89
  • 2f947ea2b45a2051578d64018e5ead58f90fcc54e126c736fce7610ac3d4e06f6da0e507125d2a4ce726794a8f058ace55840f6ce07771a26a42bd8c97a848e1
  • e08fcabcece4bfc9bdbd4a3babc0b03aa6d9d76011eaaeeac68a45a174dbcc525fef4f4fc0e6420efc31c4057b117c3cb2493858c3e01867630b5c9844ee0849

You should use “zypper dup” on TW. This performs an automatic refresh of all repos which have the auto refreh flag set.

But the same answer applies for you:

Usually this occurs when a mirror is not synced or in the mid of syncing. Simply wait some time and retry.

https://forums.opensuse.org/search?q=Warning%3A%20Digest%20verification%20failed%20for%20file%20order%3Alatest

1 Like

Look … I am not complaining … I am JUST trying to tidy things up. The QUESTION was just which set of repo’s I should be using and it is NOW clear that despite the fact that currently the cdn mirror is out of sync, I need to standardise on THOSE … and wait for updates to ripple through the system. I was just a little confused as to why something had broken which I had not seen before, otherwise I would not even have looked! If it’s not broken …
SO I will tidy all the two Nvidia machines to use the cdn links and remove the duplicate ones … but that just leaves the question as to why the third machine is happy with the direct links when it HAS openSUSE-repos-Tumbleweed and openSUSE-repos-Tumbleweed-NVIDIA installed as well :frowning: It’s more than just the openSUSE-repos-Tumbleweed-NVIDIA that is triggering the addition of the cdn mirrors I think but I know what to look out for now!
Since neither server even has a desktop installed, needing graphics to support a text only machine is another irritation but it’s been many years since we had text based graphics cards :slight_smile:
( and it’s zypper dup I am using … )

I’m hitting this as well. And getting lots of different checksums that are different to what @usessuse reports.

If I run sudo find /var/adm/mount -name 393b28216cc5bcb72ad100b437c7ec3577403989c5f5b5f3e00d71ce1f81e3d7c602edbca9594e68a99a63b8a04379503c6cbc69eaafbd7a690c5c5ac7572605-appdata.xml.gz -ls while Zypper is complaining about the mismatch (i.e. before it deletes the files) then I get different file sizes each time. All MUCH smaller than the expected size (somewhere around 17KB rather than 7.5MB). That suggests “broken server that fails at a random point in serving the file” (or “trigger-happy firewall breaks the connection”) rather than a bad sync to me. A bad sync would be consistent, not different each time.

But, interestingly, I get the right checksum if I download the file manually :neutral_face:

I’ve then tried manually downloading each file and copying them over the failed download in /var/before saying “accept bad download” (because I just replaced it with a good one). Unfortunately, while that allows zypper dup to start because it has “fresh” repo data and doesn’t need to refresh, it then obviously fails on some of the package downloads.

Ugh, stupid short edit window.

Swapping cdn.opensuse.org for download.opensuse.org on just the OSS repo gives a warning about it being replaced later, but it does seem to have allowed me to do an update just now. But it’s not a long-term fix, because it keeps getting changed back (as was already mentioned).

dig cdn.opensuse.org gives me dualstack.n.sni.global.fastly.net addresses at the moment. dig download.opensuse.org gives me download1.opensuse.org. So maybe Fastly is broken?

Because zypper downloads it from mirrors while redirectors are configured to return metadata directly, without redirecting to mirrors: Yast checksum update problem - #16 by arvidjaar

All these "me too"s are pretty useless without also showing what mirror returns the bad data. As I have shown in the linked post, you can force zypper to download metadata directly. But of course if metadata is corrupted, there is no guarantee that packages themselves are not corrupted. OTOH zypper is using different engine to download packages, which may be more resilient.

Currently there is nothing wrong with cdn.opensuse.org or download.opensuse.org. Happened to upgrade several Tumbleweed hosts without further ado.

erlangen:~ # host cdn.opensuse.org
cdn.opensuse.org is an alias for dualstack.n.sni.global.fastly.net.
dualstack.n.sni.global.fastly.net has address 146.75.121.91
dualstack.n.sni.global.fastly.net has IPv6 address 2a04:4e42:8e::347

erlangen:~ # host download.opensuse.org
download.opensuse.org is an alias for download1.opensuse.org.
download1.opensuse.org has address 195.135.223.226
download1.opensuse.org has IPv6 address 2a07:de40:b250:131:10:151:131:30
erlangen:~ #

FWIW
I was also having this problem and I managed to resolve it by turning off use_geoip_mirror in /etc/zypp/zypp.conf which nudged it towards a good mirror. I can see if I do a zypper vvv ref that the bad mirror is still failing but it finds another and uses that instead.

1 Like

I had the same issue there.

Can confirm that it’s fixed for me now.

I’m still getting the error. According to the /var/log/zypper.log, it appears to be mirror-services.net that’s having the issue:

[ZYPP_MEDIA_CURL++] MediaCurl2.cc(getFileCopy):190 URL: http://opensuse.mirror-services.net/pub/tumbleweed/repo/oss/repodata/393b28216cc5bcb72ad100b437c7ec3577403989c5f5b5f3e00d71ce1f81e3d7c602edbca9594e68a99a63b8a04379503c6cbc69eaafbd7a690c5c5ac7572605-appdata.xml.gz

(Would be useful if Zypper reported that on the CLI rather than reporting cdn.opensuse.org so that we could do diagnostics on the right domain and not get told that the observations that we’re adding to try and help are “pretty useless me-toos”)

If I open that link in Firefox then it once again downloads completely fine. But if I download it with wget then I get bounced to https://www.google.com/. And curl -i confirms that the header is Location: https://www.google.com. Nothing more. Just the home page.

So the problem is that mirror-service.net has some overly-aggressive defences at the moment and are kicking legitimate traffic from Linux tools to Google instead of serving the real file. And the checksum keeps changing because Google’s home page is always a little different.

1 Like

Yes, I believe it failed in my case as well.

https://bugzilla.opensuse.org/, same user/password as here.

You download what URL exactly?

bor@bor-Latitude-E5450:~/tmp$ curl -LO http://opensuse.mirror-services.net/pub/tumbleweed/repo/oss/repodata/393b28216cc5bcb72ad100b437c7ec3577403989c5f5b5f3e00d71ce1f81e3d7c602edbca9594e68a99a63b8a04379503c6cbc69eaafbd7a690c5c5ac7572605-appdata.xml.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   166  100   166    0     0    359      0 --:--:-- --:--:-- --:--:--   359
100 20391    0 20391    0     0  22493      0 --:--:-- --:--:-- --:--:-- 22493
bor@bor-Latitude-E5450:~/tmp$ sha512sum 393b28216cc5bcb72ad100b437c7ec3577403989c5f5b5f3e00d71ce1f81e3d7c602edbca9594e68a99a63b8a04379503c6cbc69eaafbd7a690c5c5ac7572605-appdata.xml.gz 
7fba19782b950ebb8d176ef77153b72051a550b78194d612f05abafd961f71336388949d22b03f432fa3f7e2ea9c3169a475e87cca5cdcad90164644c55394c1  393b28216cc5bcb72ad100b437c7ec3577403989c5f5b5f3e00d71ce1f81e3d7c602edbca9594e68a99a63b8a04379503c6cbc69eaafbd7a690c5c5ac7572605-appdata.xml.gz
bor@bor-Latitude-E5450:~/tmp$ 

The file is corrupted (likely, incomplete).

Conspiracy theories … the right thing in this case is
a) contact the opensuse.mirror-services.net site admins and inform them about this problem
b) contact the openSUSE mirror admins and inform them about this problem

Ideally, both.

it once again downloads completely fine

You download what URL exactly?

The URL from the log file that’s just before that sentence. The log file that you wanted people to post so that we knew the actual URLs. The URL in the log file that’s implicitly referred to by “that URL” when it’s the only preceding URL in the post.

The file is corrupted (likely, incomplete).

No. It’s not. Rather than guessing and assuming, you could see the bit where I already looked at this and said that requests are being redirected to the Google home page, run file <path> and get HTML document, ASCII text, with very long lines (3226) and then run cat <path> and see that it’s the HTML of Google’s home page.

Conspiracy theories …

When I can load a URL in Firefox and not with curl or wget, and the failure is “get redirected to Google” then that fits a common anti-bot defence. Spot the “bad” user agents (curl, wget, Python Requests, etc that are frequently used in scripting) and avoid wasting bandwidth by redirecting them to someone who can handle the load. I don’t see any other reason why a download mirror would want to redirect you to Google.

Call it a conspiracy theory if you want. I call it observations and prior experience making an informed guess at the most likely explanation. Something is causing CLI tools and libcurl requests to get redirected to Google, while letting Firefox correctly download the exact same URL. And that something is generally anti-bot configurations.

the right thing in this case is
a) contact the opensuse.mirror-services.net site admins and inform them about this problem
b) contact the openSUSE mirror admins and inform them about this problem

Ideally, both.

This is the openSUSE support forums. I assumed this was the appropriate place to report problems that aren’t software-related bugs that go in Bugzilla.

I don’t know how I’m supposed to contact the openSUSE mirror admins, though. There’s nothing about the admins on the Mirrors page or the mirror list. There’s a wiki page for adding a mirror, but that only includes the admin@opensuse.org email and I doubt they’d appreciate everyone contacting them with the problem rather than posting it publicly so that other people can see that they don’t need to report it. There’s also a mirror@lists.opensuse.org mailing list, but I doubt all of the dozens of other mirrors want that clogged up with reports. And there is an IRC channel, but I’m not set up to use IRC.

If you can point me to where openSUSE gives contact details for its mirror admins then I can send a message. But as it stands, I can’t send a message to an address that I don’t know!