Hi,
On my update today with ‘zypper up’ all packages gave the warning “Header V3 RSA/SHA1 Signature, key ID b3fd7e48: NOKEY”.
Do I understand correctly that it did not find the key to verify that the packages are OK? Then why have they been installed?
Thanks and regards
Where did those packages come from? Packman?
You can check it with;
rpm -qa --last | head -n 20
^ Will print the last 20 patches.
ge40L:/home/me/backups # rpm -qa --last | head -n 20
libMagick++-6_Q16-2-6.8.6.9-2.8.1.x86_64 Thu Mar 13 08:58:52 2014
file-5.15-4.10.1.x86_64 Thu Mar 13 08:58:52 2014
libmagic1-5.15-4.10.1.x86_64 Thu Mar 13 08:58:51 2014
ImageMagick-6.8.6.9-2.8.1.x86_64 Thu Mar 13 08:58:51 2014
libMagickWand-6_Q16-1-6.8.6.9-2.8.1.x86_64 Thu Mar 13 08:58:50 2014
libssh4-0.5.5-2.4.1.x86_64 Thu Mar 13 08:58:49 2014
file-magic-5.15-4.10.1.x86_64 Thu Mar 13 08:58:49 2014
libMagickCore-6_Q16-1-6.8.6.9-2.8.1.x86_64 Thu Mar 13 08:58:48 2014
That did not show the repo. The updates are from repo-update.
repo-update is:
13 | repo-update | openSUSE-13.1-Update | Yes | Yes | http://download.opensuse.org/update/13.1/
In Apper these updates seem to be named: openSUSE-2014-206, openSUSE-2014-208 and openSUSE-2014-209
The zypper history shows:
ge40L:/home/me/backups # cat /var/log/zypp/history | tail -n 40
# 2014-03-13 08:58:49 libMagickCore-6_Q16-1-6.8.6.9-2.8.1.x86_64.rpm installed ok
# Additional rpm output:
# warning: /var/cache/zypp/packages/repo-update/x86_64/libMagickCore-6_Q16-1-6.8.6.9-2.8.1.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID b3fd7e48: NOKEY
#
2014-03-13 08:58:49|install|libMagickCore-6_Q16-1|6.8.6.9-2.8.1|x86_64||repo-update|73fc29d149aeeedd25aa185a130f1ec4fc81c7e0d4c3a5a1e34a1f67ba1ac5f8|
# 2014-03-13 08:58:49 file-magic-5.15-4.10.1.x86_64.rpm installed ok
# Additional rpm output:
# warning: /var/cache/zypp/packages/repo-update/x86_64/file-magic-5.15-4.10.1.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID b3fd7e48: NOKEY
#
2014-03-13 08:58:49|install|file-magic|5.15-4.10.1|x86_64||repo-update|0c455e964e70be32f8295d268f7cd876e699904ff48aaf6ee51a6788bb774004|
# 2014-03-13 08:58:50 libssh4-0.5.5-2.4.1.x86_64.rpm installed ok
# Additional rpm output:
# warning: /var/cache/zypp/packages/repo-update/x86_64/libssh4-0.5.5-2.4.1.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID b3fd7e48: NOKEY
#
2014-03-13 08:58:50|install|libssh4|0.5.5-2.4.1|x86_64||repo-update|717b001e8fe087cbb2e76e56ee7e2f75818ef4a892dc811b30e879ee12055a56|
# 2014-03-13 08:58:50 libMagickWand-6_Q16-1-6.8.6.9-2.8.1.x86_64.rpm installed ok
# Additional rpm output:
# warning: /var/cache/zypp/packages/repo-update/x86_64/libMagickWand-6_Q16-1-6.8.6.9-2.8.1.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID b3fd7e48: NOKEY
#
2014-03-13 08:58:51|install|libMagickWand-6_Q16-1|6.8.6.9-2.8.1|x86_64||repo-update|93c9a1590749069013f685d51779e292d81f6ce793fb8f2e0abfdda8a0262fde|
# 2014-03-13 08:58:51 libmagic1-5.15-4.10.1.x86_64.rpm installed ok
# Additional rpm output:
# warning: /var/cache/zypp/packages/repo-update/x86_64/libmagic1-5.15-4.10.1.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID b3fd7e48: NOKEY
#
2014-03-13 08:58:51|install|libmagic1|5.15-4.10.1|x86_64||repo-update|ac7265728505b00bf6aa5f8ade540ecc2ee5446dc7ddb5dd284df3e90987d0a9|
# 2014-03-13 08:58:52 ImageMagick-6.8.6.9-2.8.1.x86_64.rpm installed ok
# Additional rpm output:
# warning: /var/cache/zypp/packages/repo-update/x86_64/ImageMagick-6.8.6.9-2.8.1.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID b3fd7e48: NOKEY
#
2014-03-13 08:58:52|install|ImageMagick|6.8.6.9-2.8.1|x86_64||repo-update|868a2f5c6a7cdbf31a5f4af9945ff1f1fd9ee77e6a8f6e7050d053f535f9e222|
# 2014-03-13 08:58:52 file-5.15-4.10.1.x86_64.rpm installed ok
# Additional rpm output:
# warning: /var/cache/zypp/packages/repo-update/x86_64/file-5.15-4.10.1.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID b3fd7e48: NOKEY
#
2014-03-13 08:58:52|install|file|5.15-4.10.1|x86_64||repo-update|d0e6ffc0442590a14ab99327ad3e15565fd291f89753ed4c3345827d4ba3f05c|
# 2014-03-13 08:58:53 libMagick++-6_Q16-2-6.8.6.9-2.8.1.x86_64.rpm installed ok
# Additional rpm output:
# warning: /var/cache/zypp/packages/repo-update/x86_64/libMagick++-6_Q16-2-6.8.6.9-2.8.1.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID b3fd7e48: NOKEY
#
2014-03-13 08:58:53|install|libMagick++-6_Q16-2|6.8.6.9-2.8.1|x86_64||repo-update|cd59e3561a820c79db6f2f435a6a67c878526b186a6ff9844cda860bbc60d1ce|
Yes, I got the same messages, e.g.
(3/3) Installing: ImageMagick-6.8.6.9-2.8.1 .................................................................................[done]
Additional rpm output:
warning: /var/cache/zypp/packages/repo-update/x86_64/ImageMagick-6.8.6.9-2.8.1.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID b3fd7e48: NOKEY
Searched around in the man pages and on the net, but didn’t really come up with an answer.
I understood there are 2 types of signing keys. One for signing the repo and another for signing the rpm package. I think zypper will prompt (unless overwritten by option) if anything is wrong with repo key. But obviously zypper has no options regarding handling of package signing keys,
Documentation (rpmkeys(8) says that package keys are installed in packages named gpg-pubkeys*. However no such package seems to exist in default OpenSUSE 13.1 repos.
Instead I found package openSUSE-build-key
$ zypper info openSUSE-build-key
...
Information for package openSUSE-build-key:
-------------------------------------------
Repository: openSUSE-13.1-Oss
Name: openSUSE-build-key
Version: 1.0-24.1.2
Arch: noarch
Vendor: openSUSE
Installed: Yes
Status: up-to-date
Installed Size: 4.8 KiB
Summary: The public gpg keys for rpm package signature verification
Description:
This package contains the gpg keys that are used to sign the
openSUSE rpm packages. The keys installed here are not actually
used by anything. rpm/zypper use the keys in the rpm db instead.
Hmm, these keys are not used, weird. So why are they installed then in the first place? So what is in rpm db? There is a rpmdb(8) page but helps as little as rpmkeys(8).
This topic almost certainly requires better documentation, cannot imagine that I missed so much… I have the feeling somebody as done a major rewrite of the related code, but not touched the documentation.
Here is how to check package signing keys (or at least what I just came up with)
Download the package (at least I did not come up with a way to check the key of an installed package)
# zypper install -d quassel
# zypper install -d -f sox
# zypper install -d -f file
(you need the -f flag if the package is already installed, although -d tells not to install)
# rpm -v --checksig /var/cache/zypp/packages/repo-oss/suse/x86_64/sox-14.4.1-2.1.2.x86_64.rpm
/var/cache/zypp/packages/repo-oss/suse/x86_64/sox-14.4.1-2.1.2.x86_64.rpm:
Header V3 RSA/SHA256 Signature, key ID 3dbdc284: OK
Header SHA1 digest: OK (f292dc8719b3febaca1f7b4cf891be874b9dcc87)
V3 RSA/SHA256 Signature, key ID 3dbdc284: OK
MD5 digest: OK (daa7e820e92bafd45e47f6245f5f724e)
# rpm -v --checksig /var/cache/zypp/packages/repo-update/x86_64/quassel-mono-0.9.2-12.1.x86_64.rpm
/var/cache/zypp/packages/repo-update/x86_64/quassel-mono-0.9.2-12.1.x86_64.rpm:
Header V3 RSA/SHA256 Signature, key ID 3dbdc284: OK
Header SHA1 digest: OK (79f2b0e9ce0b1d8b96c0c012a89a2f1b08853e8d)
V3 RSA/SHA256 Signature, key ID 3dbdc284: OK
MD5 digest: OK (dbc3b859556c352a38b23d006d49dabe)
# rpm -v --checksig /var/cache/zypp/packages/repo-update/x86_64/file-5.15-4.10.1.x86_64.rpm
/var/cache/zypp/packages/repo-update/x86_64/file-5.15-4.10.1.x86_64.rpm:
Header V3 RSA/SHA1 Signature, key ID b3fd7e48: NOKEY
Header SHA1 digest: OK (381bca2de6c8935a7045e9f56bd22c1b1d09e9e9)
V3 RSA/SHA1 Signature, key ID b3fd7e48: NOKEY
MD5 digest: OK (9b12ef1d0e1b83191732a74ca6ab1730)
So packages in repo-oos and repo-update used to be signed with key ID 3dbdc284, but now they obviously use a new key.
Hi all,
I have encountered the same messages on six production servers, and I started to worry…
This is what I found (for package file-magic):
Zypper downloads
file-magic-5.14_5.15-4.1.2_4.10.1.x86_64.drpm
This file’s signature is verified successfully, otherwise zypper will not use it. You can manually check its signature with
rpm -v --checksig file-magic-5.14_5.15-4.1.2_4.10.1.x86_64.drpm
This .drpm package is signed by a key with id 0x3dbdc284, the openSUSE key.
file-magic-5.14 is now patched with this .drpm package
The resulting (patched) file is:
file-magic-5.15-4.10.1.x86_64.rpm
You can find this file in /var/cache/zypp/packages/repo-update/x86_64 if you use zypper like this:
zypper up -D --skip-interactive
This .rpm file is now installed, but carries a signature created by an unknown key: 0xb3fd7e48
Unfortunately, this key cannot be found on the MIT key server:
gpg --keyserver pgp.mit.edu --recv-key 0xb3fd7e48
For now I assume that the patch (.drpm) was created against a file signed with key 0xb3fd7e48, so the resulting .rpm package (after patching it) is also signed by that (unknown) key.
I sure hope that this is not reason enough to believe that the servers have been compromised and need to be rebuild from scratch… Keeping my fingers crossed…
Yours
Paul
I think the files on the update servers are still signed with key 0x3dbdc284 (.drpm packages).
After applying the patch, the resulting .rpm package is singed with an unkown key (0xb3fd7e48). I couldn’t quickly find any .rpm package from today on the update servers.
It seems to me, there is something weird going on.
Same problem with openSUSE 12.3 this morning: several delta rpm files produced NOKEY errors:
My update repo url is http://download.opensuse.org/update/12.3/
All the problematic files were claimed to originate from there, but when I searched via http (Browser) in that repo, none of the drpm versions was there.
I sure do not want questionable ssh/file/ImageMagick packages, so I did a “zypper clean -a” and then “zypper ref” to rebuild the caches.
After that I reinstalled all suspicious packages (zypper in -f …), without any error message. All versions are back to official versions (double checked and verified via http://software.opensuse.org/).
Is this a security issue with http://download.opensuse.org/ ?
[QUOTE=kalex77;2630357Is this a security issue with http://download.opensuse.org/ ?[/QUOTE]
No. But apparently something went wrong with those updates. Normally the packages should get re-signed with the update repo’s key, but that didn’t happen in this case. So they are still signed with some internal openSUSE:Maintenance key.
This is investigating this at the moment.
If you understand german, see also here:
http://lists.opensuse.org/opensuse-de/2014-03/msg00164.html
I also downgraded those packages to previous versions. But libmagic1 and file-magic seem to be used almost everywhere. e.g. in zypper. So the command to remove the packages used that stuff. So if this was an attack then we would be f#cked anyway…
But how can it be that packages with wrong or missing signatures can be installed by zypper? I would assume the most basic unit test for such code and the most important one is if packages with wrong or missing signatures are refused. How can it be that zypper installs those. That is just something that must not happen, ever. I very much like suse, but if package management code can end up in production that did not pass such a unit test or if such unit tests do not exist, then I have to switch…
Sorry, that was rude…
If it can only happen if the delta rpm has correct signature then it is not that bad.
See the thread in http://lists.opensuse.org/opensuse-security/2014-03/msg00007.html for more info.
They signed the packages with a wrong key.
Because zypper checks the repo metadata signature. If the repo maintainer has signed the metadata describing the package it is considered trusted, even if the package signing key is not. So it’s the responsibility of the repo maintainer not include bad packages into the repo.
Not really the cleanest design, using 2 levels of signing, but only relying on one and warning on the other. No security issue as far as I see in this moment if you are aware that you trust the repo maintainer, but a source for confusion. And given the fact the users are always the weakest point in all signing, PKI etc, it’s not a good idea to confuse them even further…
I guess historical reasons, because rpm existed before zypper AFAIK.
Well, actually it is rpm that checks the signature, not zypper.
So I guess the behaviour should be the same on other rpm-based distros.
And yes, it is true that rpm only prints a warning if the key is unknown.
But if the signature is wrong/invalid, it will refuse to install the package AFAIK.
If you consider this a bug, you should report it at http://bugzilla.novell.com/ (same username/password as here).
Hi,
thanks for your reports and concerns.
The main integrity checking of our repositories is the gpg signature of repomd.xml (repomd.xml.asc) and the chain of sha256 signatures to all the other XML and .RPM files from that file.
The RPM signature is secondary and currently not checked.
The problem was that the updated RPMs did not get resigned correctly in the release process on March 13th due to a bug in the OBS.
I have removed the erronous updates and will be respooling them in the next days.
Ciao, Marcus (openSUSE Maintenance / Security)
Thanks. Much appreciated.
(It seems that I cannot thank without the additional appreciation, or I get a “message too short” error. The appreciation is sincere, nevertheless).
Thanks Marcus for clearing things up.