Corrupted rpm db - how to fix?

Hi everybody! Today I was investigating a problem that I have on my home pc since a few months.
The problem is that zypper repeatedly asks me to confirm repository signing keys even if I already told it to trust the key always.

I found a lot of threads with similar problems, which led me investigating zypper, gpg and rpm.At one point, I found out that my rpm db is corrupted, because when I enter ‘rpm -qa gpg-pubkey*’ this is what I get:

gpg-pubkey-accaf35c-57d58c01
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-6867f5be-4d77cecd
gpg-pubkey-ed340235-577f6c2e
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-0d988d6e-57839cae
gpg-pubkey-ed340235-577f6c2e
gpg-pubkey-1abd1afb-54176598
gpg-pubkey-c50a5d22-591463bb
gpg-pubkey-accaf35c-57d58c01
gpg-pubkey-f191f0e8-58bafb92
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-0d988d6e-57839cae
gpg-pubkey-0d988d6e-57839cae
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-8898875c-4c0241c4
gpg-pubkey-efb20b23-5a4f4f57
gpg-pubkey-0d988d6e-57839cae
gpg-pubkey-f191f0e8-58bafb92
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-11f63c51-3c7dc11d
gpg-pubkey-c50a5d22-591463bb
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-f191f0e8-58bafb92
gpg-pubkey-be1229cf-5631588c
gpg-pubkey-f191f0e8-58bafb92
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-0d988d6e-57839cae
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-c50a5d22-591463bb
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-f191f0e8-58bafb92
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-d38b4796-570c8cd3
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-ed340235-577f6c2e
gpg-pubkey-83a736cf-514c2089
gpg-pubkey-0d988d6e-57839cae
gpg-pubkey-0d988d6e-57839cae
gpg-pubkey-7fac5991-4615767f
gpg-pubkey-0d988d6e-57839cae
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-c3401e12-587def4d
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-7f8840ce-4bb222d5
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-ed340235-577f6c2e
gpg-pubkey-df7587c3-576a5c23
error: rpmdbNextIterator: skipping h#   64697 
Header V3 RSA/SHA256 Signature, key ID 3dbdc284: BAD
Header SHA1 digest: BAD (Expected bf167126ecaa67d16fee74af17096529278aad8d != cd4a91ad1f0d65d360cce5dacffea553e358b550)
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-f191f0e8-58bafb92
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-f191f0e8-58bafb92
gpg-pubkey-f191f0e8-58bafb92
gpg-pubkey-0d988d6e-57839cae
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-f191f0e8-58bafb92
gpg-pubkey-f191f0e8-58bafb92
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-ed340235-577f6c2e
gpg-pubkey-ed340235-577f6c2e
gpg-pubkey-ed340235-577f6c2e
gpg-pubkey-ed340235-577f6c2e
gpg-pubkey-3dbdc284-53674dd4
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-ed340235-577f6c2e
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-72db573c-4da468cb
gpg-pubkey-f4752699-57163f23
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-0d988d6e-57839cae
gpg-pubkey-0d988d6e-57839cae
gpg-pubkey-accaf35c-57d58c01
gpg-pubkey-efb20b23-5a4f4f57
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-d38b4796-570c8cd3
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-accaf35c-57d58c01
gpg-pubkey-0d988d6e-57839cae
gpg-pubkey-0c1289c0-58c6ad7d
gpg-pubkey-d4168679-59669fb3
gpg-pubkey-df7587c3-576a5c23
gpg-pubkey-ed340235-577f6c2e
gpg-pubkey-df7587c3-576a5c23

Besides the fact that there are a lot of duplicated entries (I don’t know if it’s normal), there is that ugly error that asks for immediate action.
So I took a backup of the rpm db and then issued ‘sudo rpm -vv --rebuilddb’, which gave me this (snipped):

(...)
Header SHA1 digest: OK
D: adding "bison" to Name index.
D: adding 53 entries to Basenames index.
D: adding "Development/Languages/C and C++" to Group index.
D: adding 17 entries to Requirename index.
D: adding 2 entries to Providename index.
D: adding 15 entries to Dirnames index.
D: adding 1 entries to Installtid index.
D: adding 1 entries to Sigmd5 index.
D: adding "71bc5f24969fabf4a56f13f92b3c9f7968f2fe1c" to Sha1header index.
D: adding 1 entries to Recommendname index.
D:  read h#   64697
Header SHA1 digest: BAD (Expected bf167126ecaa67d16fee74af17096529278aad8d != cd4a91ad1f0d65d360cce5dacffea553e358b550)
error: cannot add record originally at 64697
D: closed   db index       /usr/lib/sysimage/rpm/Packages
D: closed   db environment /usr/lib/sysimage/rpm
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Packages
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Enhancename
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Supplementname
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Suggestname
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Recommendname
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Transfiletriggername
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Filetriggername
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Sha1header
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Sigmd5
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Installtid
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Dirnames
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Triggername
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Obsoletename
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Conflictname
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Providename
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Requirename
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Group
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Basenames
D: closed   db index       /usr/lib/sysimage/rpmrebuilddb.21797/Name
D: closed   db environment /usr/lib/sysimage/rpmrebuilddb.21797
warning: failed to rebuild database: original database remains in place

So, I don’t know if this is the root cause of my problems with zypper, but it is nevertheless something that has to be dealt with.
Given the fact that rpm itself hasn’t been able to fix the situation, what possibilities do I have to fix the problem?

Thank you in advance.
Cris

I tried dumping and reloading the Packages db by doing :

mv Packages Packages-BACKUP
db_dump Packages-BAKUP | db_load Packages

in /var/lib/rpm, but the new file (which is a bit smaller) still produces the error.

I would do that
Start Yast> repository> gpg keys> find the error key delete it> restart Yast> should ask you whether to accept the new key.

Hi enziosavio!

Thank you for your suggestion. Unfortunately Yast only shows me a handful of keys (6 IIRC, I’m not on the faulty box right now) and it shows no error.
The same happens if I issue this at the cli:
rpm -q gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE} %{SUMMARY}

It will show just 5 or 6 keys with no error.

I’m lost :frowning:

Anybody else has any idea of how to fix this, short of reinstalling?

Thank you in advance
Cris

http://www.hosting.universalsite.org/image-schermatada20180420090247-29B2_5AD991FF.png](http://www.hosting.universalsite.org/share-schermatada20180420090247-29B2_5AD991FF.html)

Sorry but you have the fingerprint that gives you error, identify it between the keys and delete it, close and restart yast, which will ask you to accept the new Repo footprint

id 3dbdc284

you have the id and the key looks for matches

You should be able to delete them with something like this:

rpm -e --allmatches gpg-pubkey-df7587c3-576a5c23

But, do you actually have any problem because of the “duplicated” keys?
If not, I see no reason to reinstall either…

And I do have duplicates here on my system as well, although everything works fine AFAICT. :wink:

If the rpm database is corrupted, you could try to run “rpm --rebuilddb” to hopefully fix it.
“rpm --reinitdb” will create a new, empty, one that would also “fix” the duplicated keys, but it would also lose track of what packages are installed of course.

PS: In my case the key for KDE:Extra is installed multiple (about 60+) times.
That’s probably related to the bug that caused libzypp to “forget” the repo key (triggered by PackageKit) and asking to accept it again and again. (which happened especially with that particular repo for me)

I just had a look at libzypp’s changelog (on Leap 42.3) and found this that sounds related:

* Tue Jan 16 2018 ma@suse.de
- Fix deleted keys not removed from rpmdb too (bsc#1075449)
- version 16.17.9 (0)

I.e. apparently zypper/libzypp did not remove keys from the rpm database until recently due to a bug.

Hi wolfi323!

Well, not exactly.
I am investigating a problem where zypper insists telling me that it received a new repository or package signing key, asking me to validate it each time, even though I have told it to always trust the key a zillion times.
While investigating on this problem, I noticed the error and the duplication. I don’t know if the duplication is malign, it just stroke me as something weird. And since I also have a db corruption problem, I thought the two were related.

Unfortunately, as you can read in my OP, that command failed in fixing my rpm db :(.

More in the next reply…

Cris

Hi wolfi323!

Hey, this is exactly what is happening to me!!
But, despite repeated googling I failed to find a related bug!! Do you have the number? Is it fixed now?
The strange thing is that I only have it on one PC, the other one seems immune.

I am still seeing the problem, though. Had it just five minutes ago when I ran ‘zypper dup’… I had to trust (again) the key for 4 repos.

Cris

I tried this command with the key identified by the error message (gpg-pubkey-3dbdc284-53674dd4).
I can verify that the key is not present anymore, but I still have the error when I run ‘rpm -qa gpg-pubkey*’.

Cris

I’d need to search for the bug report, but that probably wouldn’t help you. It was about PackageKit causing repo keys to get invalidated when it looked for updates (the bug was actually in libzypp), which made it necessary to accept them again and again.

It’s fixed since nearly a year though, and I haven’t had the problem a single time since then. Also, there were a couple of forum threads about it back then, and new ones ceased to appear since then either.

Yeah, I did not see that error message “hidden” in the rest of the output before.

Well, your database is indeed corrupted.
If “rpm --rebuilddb” doesn’t fix it, the only option I see is start fresh with a clean one (rpm --reinitdb) and reinstall all packages I’m afraid.

The rpm database is a standard Berkely DB though, maybe there tools/ways to fix it but I cannot help there.

You won’t see the error fixed yet on your system, Wolfi said he found the issue possibly addressed on 42.3 and you’re on TW.

BTW -
I found it curious that your rpm database is so complex with so many indexes…
I checked on one of my own TW which is nearly a default install and it only has 2 indexes (Packages and Names) and 2 gpg-pubkeys(I haven’t verified, but I’m guessing that means that more than one TW repo may be signed with the same pubkey).
I guess your long list of indexes are due to your many OBS repos which possibly have packages under development with frequent changes…

TSU

Actually the issue was in 42.2, it was fixed already when 42.3 was released.

And of course it is fixed on TW too, since about a year as I wrote.

I found it curious that your rpm database is so complex with so many indexes…
I checked on one of my own TW which is nearly a default install and it only has 2 indexes (Packages and Names)

???

From my 42.3 system:

$ ls /var/lib/rpm

alternatives  Dirnames    Name          Providename  Sha1header
Basenames     Group       Obsoletename  Pubkeys      Sigmd5
Conflictname  Installtid  Packages      Requirename  Triggername

And that’s the norm.
Everything except Packages can be recreated automatically (from Packages), and that’s exactly what “rpm --rebuilddb” does.
In the OP’s case, it seems that Packages itself is corrupt though.

and 2 gpg-pubkeys(I haven’t verified, but I’m guessing that means that more than one TW repo may be signed with the same pubkey).
I guess your long list of indexes are due to your many OBS repos which possibly have packages under development with frequent changes…

The number of indexes has nothing to do with the number of repos.

The number of gpg-keys of course does, because usually each repo has its own key.

I had a similar problem once, and I solved adding the YaST-Head repo and updated all Yast to that Repo

Hm, I don’t see how this would magically fix a corrupted rpm database though… :\

It would of course help if you encounter some bug that’s already fixed, but the fix is not in the standard repos yet.
That’s not really the case here though AFAICT.

I’m ashamed to say I had two Firefox tabs open on the Forum … but it was an answer I wanted to give here
https://forums.opensuse.org/showthread.php/530717-Yast-quot-Software-Management-quot-crashing/page3

It’s probably no longer relevant anyway, but I stumbled over it today, so here it is:
http://bugzilla.opensuse.org/show_bug.cgi?id=1036659
:wink:

Hi wolfi323.

Thank you wolfi323!
I’m still busy trying to fix my rpm db… I had a few good suggestions, if I succeed in fixing it I’ll post the solution here.

Cris