12.1 Repo Data is incorrect

dwestf · January 22, 2012, 4:03pm

DenverD

libqt4-4.7.4-19.4.1.x86_64.rpm has a date of 9 Dec 2011. Its checksum matches the repodata file of 9 Jan 2012 that is still on my secondary server (Colossus). The libqt4 that I was getting from openSUSE and its mirrors matches the checksum that is in the 9 Jan repodata file.

Thanks to Carlos and aria2c we can get a version of he file that checksum matches what is in the repodata file data 20 Jan. Aria2c does not preserve creation dates so I have no idea what the build date is, all I can say that it is not the same file that was put into the update repo on 9 Dec. Both files pass rpm --checksig, which says that it comes from openSUSE.

Why do we have a new libqt4 without a new version number. If it got patched and the version was not changed then existing 12.1 computers will not get the update. New installs will have a good chance of failing because they get new repodata but old files because the mirrors are not syncing.

I did send an email to ftpadmin mirror and admin at openSUSE. ftpadmin failed because it does not exist and mirror kicked back saying that it was for a mailing list. I have not got a kickback from admin so I hope it is in a queue waiting to be read.

Dave W

DenverD · January 22, 2012, 4:40pm

On 01/22/2012 04:06 PM, dwestf wrote:
>
>
> libqt4-4.7.4-19.4.1.x86_64.rpm has a date of 9 Dec 2011. Its checksum
> matches the repodata file of 9 Jan 2012 that is still on my secondary
> server (Colossus). The libqt4 that I was getting from openSUSE and its
> mirrors matches the checksum that is in the 9 Jan repodata file.

either i can’t read or i’m just too dense to understand but: what i read
in your statement above is that all the lib4-4.7.4-19.4.1.x86_64.rpm
files you have locally or can now download (no matter their date or
their location) all have the same checksum…

and in every case their checksum matchs the checksum listed in the
repodata file–is that correct?

–
DD
openSUSE®, the “German Engineered Automobiles” of operating systems!

robin_listas · January 22, 2012, 5:48pm

On 2012-01-22 16:40, DenverD wrote:
> On 01/22/2012 04:06 PM, dwestf wrote:

> either i can’t read or i’m just too dense to understand but: what i read in
> your statement above is that all the lib4-4.7.4-19.4.1.x86_64.rpm files you
> have locally or can now download (no matter their date or their location)
> all have the same checksum…

Except the one downloaded from http://mirror.cst.temple.edu/, it has the
same name but incorrect checksum (see the aria2c download text several
posts back, Date: Sun, 22 Jan 2012 00:16:03 GMT).

What I find hard to believe is that it has the correct signature. If that
is true it is a big bug somewhere in the chain of events.

And it is true…


> cer@Telcontar:~/tmp/aa> sha256sum libqt4-4.7.4-19.4.1.x86_64.edu.rpm libqt4-4.7.4-19.4.1.x86_64.rpm
> c7a3129be10ea272359b1e9468eb0293c7b1569d59b6ca01931428c15933b44d  libqt4-4.7.4-19.4.1.x86_64.edu.rpm
> 746850567a5f53353f0deebc44ae0b7ed04528570f0a1c87bfee68345e77298c  libqt4-4.7.4-19.4.1.x86_64.rpm
> cer@Telcontar:~/tmp/aa> rpm --checksig libqt4-4.7.4-19.4.1.x86_64.edu.rpm libqt4-4.7.4-19.4.1.x86_64.rpm
> libqt4-4.7.4-19.4.1.x86_64.edu.rpm: rsa sha1 (md5) pgp md5 OK
> libqt4-4.7.4-19.4.1.x86_64.rpm: rsa sha1 (md5) pgp md5 OK
> cer@Telcontar:~/tmp/aa>


> -rw-r--r--  1 cer users 3648318 Dec  9 10:06 libqt4-4.7.4-19.4.1.x86_64.edu.rpm
> -rw-r--r--  1 cer users 3648318 Jan 22 00:34 libqt4-4.7.4-19.4.1.x86_64.rpm

And in this case, it is understandable that this site does not update their
rpms because they have the same name (and release) as those upstream, there
is nothing to update. Unless they also check the file timestamp upstream.
The “…edu.rpm” above is donwloaded via wget (timestamp preserved) while
the other one is downloaded via aria2c, which puts a local timestamp.
Meaning I don’t have the real timestamp for it. Ok, I’ll download it again
from a different mirror (Spain) with wget. And…:


> -rw-r--r--  1 cer users 3648318 Jan 22 00:34 libqt4-4.7.4-19.4.1.x86_64.aria2.rpm
> -rw-r--r--  1 cer users 3648318 Dec  9 10:06 libqt4-4.7.4-19.4.1.x86_64.edu.rpm
> -rw-r--r--  1 cer users 3648318 Dec  9 10:06 libqt4-4.7.4-19.4.1.x86_64.rpm

Same date, Dec 9.

But looking inside with mc the build dates are:

libqt4-4.7.4-19.4.1.x86_64.rpm

Signature : RSA/8, Fri Dec 9 10:06:20 2011, Key ID b88b2fd43dbdc284
Build Date: Tue Dec 6 15:07:42 2011

libqt4-4.7.4-19.4.1.x86_64.edu.rpm

Signature : RSA/8, Fri Dec 9 10:06:11 2011, Key ID b88b2fd43dbdc284
Build Date: Tue Dec 6 15:07:42 2011

Ie, the same! But they are different files with different checksums.

Dave, you can also create a bugzilla (Component Download Infrastructure).
This is probably read by more people. If you post the number here I can add
info to the report corroborating your findings.

But maybe nothing will be done till Monday or later.

–
Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

dwestf · January 22, 2012, 6:48pm

Thanks for all the help Carlos.

Here is the bug report. I did a man on aria2c if you add -R it will get timestaps.

-rw-r–r-- 1 dbw users 3648318 Dec 9 04:06 libqt4-4.7.4-19.4.1.x86_64.rpm

https://bugzilla.novell.com/show_bug.cgi?id=742772

robin_listas · January 22, 2012, 10:03pm

On 2012-01-22 18:56, dwestf wrote:
>
> Thanks for all the help Carlos.
>
> Here is the bug report. I did a man on aria2c if you add -R it will get
> timestaps.

Ah, interesting that.

> https://bugzilla.novell.com/show_bug.cgi?id=742772

Thanks. Added to it and corrected the component.

–
Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

horstneumannnbcon · January 23, 2012, 9:57am

I think we have a similar issue here.

I installed openSuSE 12.1 (64) this weekend but YOU failed when updating kdebase4 package dolphine
(target version 4.7.2-4.4.1) because of a checksum error.
I currently use the default repro configuration.

Any idea ??

dwestf · January 23, 2012, 1:08pm

horstneumannnbcon

As you read in my first post, I keep a local copy of repositories. When I went to update a new 12.1 64bit computer, the update that worked the previous week on a new build failed with Digest Errors. Below is the list of files that I had Digest Errors on.

In trying to see what was going on I just picked one file to look at, libqt4. The first thing that I saw was that the checksum for libqt4 changed in the repodata files. They are the *.xml.gz files in Index of /update/12.1/repodata directory. They tell the update what files are in the repo and what there checksums are.

It appears that, at least in the case of libqt4, the rpm was built and was signed twice by the openSUSE team making two copies of the RPM. And at some point the second copy was put in to update repo. Then there is some triger that rebuilds the repodata files. We do not know why two RPM were built and intorduced to the repo at different times.

What makes this a problem is the way mirrors sync thems, to save cpu time most mirrors look for file name changes. In the case of libqt4 the two files have the same name, same date, very close time, and I think the same size. The only for mirrors to sync with this problem is to checksum checks, this takes a lot of CPU time at both ends.

To fix this the openSUSE team will have to decide which rpm to keep, and then get the mirrors in sync. There are a few way I can think of to do this.

One, rebuild the RPM’s with a forced version number change.

Two, delete the RPM’s for a day, then put back in after the mirrors also delete the RPM’s

Three, contact the mirror admin’s and have force checksum during sync update. Since it maybe unknown how may files are out of sync this maybe the best option. These 30 files are just from a base 64bit kde install, there maybe more RPM’s outof sync.

Dave W

device-mapper
krb5
libfreetype6
libqt4
libxml2-python
pam-modules
splashy
systemd
lvm2
ft2demos
libqt4-sql
splashy-branding-openSUSE
systemd-sysvinit
postfix
libqt4-x11
libqt4-sql-sqlite
libqt4-sql-mysql
libqt4-qt3support
polkit-kde-agent-1
okular
libkonq5
konsole
keditbookmarks
plasmoid-folderview
kfind
kdepasswd
kdebase4-libkonq
konqueror-plugins
dolphin
konqueror

dwestf · January 23, 2012, 4:44pm

I forced my local repo to update with aria2c.


for i in `ls *rpm`;do echo $i;rm -rf $i;aria2c -R --http-proxy=avgw:8080 http://download.opensuse.org/update/12.1/x86_64/$i;done

Had to add in the rm, otherwise aria2c said file was already downloaded.

All files except dolphin were updated, dolphin still reported digest error.

Still have a checksum mismatch between RPM and repdata for dolphin. The checksum in the metalink data matches repo file from 9 Jan.


zcat ../repodata/32ca040bf5837a12126d34067cb0ee196b091b76cf1ca1aaa9a0c2642c1fd1d7-filelists.xml.gz|grep dolphin|grep x86 
<package pkgid="4a73848e62be00c5a04fdfbdeaebe7e7decdb125173de754a65675a6af284d25" name="dolphin" arch="x86_64">

cat dolphin-4.7.2-4.4.1.x86_64.rpm.meta4
<?xml version="1.0" encoding="UTF-8"?>
<metalink xmlns="urn:ietf:params:xml:ns:metalink">
  <generator>MirrorBrain/2.15.0</generator>
  <origin dynamic="true">http://download.opensuse.org/update/12.1/x86_64/dolphin-4.7.2-4.4.1.x86_64.rpm.meta4</origin>
  <published>2012-01-23T14:26:17Z</published>
  <publisher>
    <name>openSUSE</name>
    <url>http://download.opensuse.org</url>
  </publisher>

  <file name="dolphin-4.7.2-4.4.1.x86_64.rpm">
    <size>909042</size>

    <!-- <mtime>1323960742</mtime> -->

    <!-- internal id: 58154689 -->
    <hash type="md5">401d48f1b15a00a4eb9fcfec1059b794</hash>
    <hash type="sha-1">ebe92c250fbe5da365d5cc95cf0632cf0c739523</hash>
    <hash type="sha-256">903b3243f0e3012e5dc9dd01c26b6f618fcb808d65394df97e6724002ce70e4e</hash>
    <pieces length="262144" type="sha-1">
      <hash>75494cf4696f4363cd8aa2521cd1ba5648a11790</hash>
      <hash>eb659ed7bb323e4fbe28e4fb634510bce35715e5</hash>
      <hash>2614cc83b1aefd46d3495fa949d09cf7dfa4d37d</hash>
      <hash>b1cdd409f346f85d8c14eeb6601660df5b5d986a</hash>
    </pieces>


    <!-- Found 116 mirrors: 0 in the same network prefix, 0 in the same autonomous system,
         17 handling this country, 4 in the same region, 75 elsewhere -->

    <!-- Mirrors in the same network (160.107.0.0/16): -->

    <!-- Mirrors in the same AS (687): -->

    <!-- Mirrors which handle this country (US): -->
    <url location="us" priority="1">http://mirror.cst.temple.edu/opensuse/update/12.1/x86_64/dolphin-4.7.2-4.4.1.x86_64.rpm</url>
     <url location="us" priority="2">http://opensuse.mirror.netriplex.com/update/12.1/x86_64/dolphin-4.7.2-4.4.1.x86_64.rpm</url>
  </file>
</metalink>

Had to shorten metelink past in order to post.

All the other files checksum matched current repodata files.

Dave W

DenverD · January 23, 2012, 4:59pm

On 01/23/2012 04:46 PM, dwestf wrote:
> All files except dolphin were updated, dolphin still reported digest
> error.
>
> Still have a checksum mismatch between RPM and repdata for dolphin.
> The checksum in the metalink data matches repo file from 9 Jan.

this is all very interesting information…but as mentioned earlier
putting it here does nothing toward a fix…

because, the folks who have the capability to make the fix do not read
here…

it would be much better if you update the place(s) where they do read…

–
DD
openSUSE®, the “German Engineered Automobiles” of operating systems!

dwestf · January 23, 2012, 7:46pm

I compared my current mirror with my backup copy from 9 Jan. Out of the 680 RPM’s that are in the 9 Jan mirror, 9 do not exist in the current mirror because they have new versions. 122 RPM’s exist in both mirrors but have been changed since 9 Jan. I posted my results to the bug report. I am trying to figure a way to compare all of the RPM’s in the current mirror with the checksums in the repodata files.


pwd /colossusRaid/Distributions/openSUSE/12.1/Update/x86_64

for i in  `ls *.rpm`;do diff -q $i
/atlas/Distributions/openSUSE/12.1/Update/x86_64/$i 2>/dev/null;done|grep differ|awk '{print $2}'

I was wrong in my previous post, the files did not update they just did not give an error. Because dolphin failed the whole update failed. To get them to update I went in to Software Management and set Protect on dolphin. Then I went back into Update and selected do not patch dolphin. All of the RPM’s updated. That is the work around to get your system to update, protect the RPM and select do not patch when asked about conflict.

Dave W

robin_listas · January 23, 2012, 10:53pm

On 2012-01-23 16:59, DenverD wrote:
> it would be much better if you update the place(s) where they do read…

The issue was reported yesterday, still no response

–
Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

dwestf · January 24, 2012, 9:20pm

Andreas added this to the bug report.

Andreas Jaeger 2012-01-24 16:28:08 UTC
This is beeing worked on - after the next build service publish of the update
repo, everything should be fine again.

Dave W

DenverD · January 25, 2012, 10:07am

On 01/24/2012 09:26 PM, dwestf wrote:
> everything should be fine again.

thanks for posting the update…

and THANKS for the find…

and BIG THANKS!! for your persistence in getting it reported (even if i
was not smart enough to see the problem!)…

and, thanks to you ‘they’ have now removed the frustrating bread crumbs
to “ftpadmin” [failed because it does not exist] and “mirror” [kicked
back saying that it was for a mailing list]

you get my Done Good Award for the month of January!
(and, Carlos scores an Assist!)

–
DD
openSUSE®, the “German Engineered Automobiles” of operating systems!

dwestf · January 26, 2012, 1:28pm

Ok,

Saw a lot up updates in the 12.1 x86 come down yesterday when I synced my mirror. Was still seeing around six RPM’s that did not match repodata, even after pulling those files with aria2c their cheksum did not match repodata. The RPM’s checksum did match the meta4 files.

After my mirror synced this morning, only two RPM’s did not match repodata, but did after they were pulled down with aria2c.

So it looks like all the correct 12.1 x86 RPM’s are in the master mirror and are slowly being pushed out.

Command used to compare RPM’s with repodata.


for i in `ls *.rpm`;do printf "%-80s" "$i" ;zcat ../repodata/*-filelists.xml.gz|grep -c `sha256sum $i|awk '{print $1}'` ;done|grep "0$"

Command used to compare RPM’s with meta4 data.


for i in `ls *.rpm`;do printf "%-80s" "$i" ;grep -c `sha256sum $i|awk '{print $1}'` $i.meta4;done

Command used to force download of existing RPM with meta4 data.


for i in `ls *rpm`;do echo $i;rm -rf $i;aria2c -R --http-proxy=avgw:8080 http://download.opensuse.org/update/12.1/x86_64/$i;done

Dave W

dwestf · February 3, 2012, 1:55pm

Problem is back. The Update repo was fine at the start of the week, but I now show 194 files have changed since Monday. Using anl.gov as a mirror I show four files that do not match repodata. Marcus want to blame the mirrors for being out of sync instead of openSUSE for changing files that should change until tere is a new version number.

Dave

From the bug report:

Marcus Rückert 2012-02-03 10:44:14 UTC
again … i can not control other mirrors. if those mirrors would resync from
rsync.opensuse.org or stage.opensuse.org they should pull new files.

David Westfall 2012-02-03 12:48:16 UTC
You are right, you do not control the other mirrors, but you do control the
master copy. This is what backups are for. You can restore a file that has
changed, for what ever reason, to its original copy.

I have been in contact with the mirror admin at anl.gov, and I will send him a
copy of your last post, and this one. And I will let him know that you think
that it is his responsibility to fix your problem. He has been very helpful
and must be tired of me asking him to check his sync.

I am sure that the other mirror admins don’t want to be told that they need to
increase their work load, to fix a problem caused by openSUSE, by ensuring that
all RPM’s in their mirror match repodata.

This has been going on for a couple of weeks now, and no one has said why these
copies are being created and why they replacing the original RPM in the Update
repo. No RPM should change in a repo until there is a version number change.

This problem will not be resolved until the openSUSE team finds out why
multiple copies of RPS’s are being created and pushed into to the update repo
at different times.

DenverD · February 3, 2012, 4:54pm

On 02/03/2012 01:56 PM, dwestf wrote:
> This problem will not be resolved until the openSUSE team finds out why
> multiple copies of RPS’s are being created and pushed into to the
> update repo at different times.

wow, this sounds bad…

–
DD
Read what Distro Watch writes: http://tinyurl.com/SUSEonDW

dwestf · February 16, 2012, 4:25pm

The problem has spread. When doing a upgrade from 11.3 32bit to 12.1 32bit I got digest errors in noarch. I did a full sync and check and found errors in i586, i686, noarch and x86_64. Below is my post the bug report. If use the Default Online repos you should be fine. If you use a remote mirror you may have problems.

David Westfall 2012-02-16 15:15:09 UTC
Still have not heard from anl.gov. Just sent out another message, also sent
one to temple.edu. I also asked if they monitor the mirror mailing list or IRC
as listed on openSUSE:Mirror infrastructure - openSUSE .

Found when I did a zypper dup on a 11.3 32bit to 12.1 that there were digest
errors in noarch. Did a sync to anl.gov. and found that the problem has spread
to i586 i686 and noarch. The following command gives a count for each
directory. Remove |wc -l to get list of files.

12.1/Update # for i in i586 i686 noarch x86_64;do for j in ls $i/*.rpm;do
printf “%-80s” “$j” ;cat repodata/*-filelists.xml|grep -c sha256sum $j|awk '{print $1}' ;done|grep “0$”|wc -l;done
223
14
9
4

Why are these RPS’s getting two signed copies?

How does the second copy get in to repo?

Where has the second copy been stored before it goes into the repo? In some
cases it was month before the second copy went in to the repo.

Dave W

DenverD · February 16, 2012, 7:04pm

On 02/16/2012 04:26 PM, dwestf wrote:
> The problem has spread.

to me, this is worrisome …

my look just now at the bug
<https://bugzilla.novell.com/show_bug.cgi?id=742772> i get the feeling
that the openSUSE folks are saying the problem is at the mirrors, and
not with anything they control…

but, wasn’t it a giant disk crash which started the problem of mirrors
not matching the master server’s date/time stamps?

i guess that means anytime in the future the ‘master server’ falls over
and it is brought back up someone has go though this all again–am i right?

and who should do that? the first guy who notices the problem, or the
folks with the broken (not redundant enough) hardware…

–
DD
Read what Distro Watch writes: http://tinyurl.com/SUSEonDW

dwestf · February 16, 2012, 9:03pm

Sorry if this is to long.

So far no one has said why these RPS’s have different checksums. It can’t be from a restore, thats the point of checksums to ensure the file is an exact match. I have not heard about a crash.

The problem is that two different versions exist. They have the same name, size, creation date, and very close creation time. Which is why most mirrors will be out of sync. They dont sync based on checksum, to much overhead.

If the openSUSE team has a mailing list and a IRC channel, are they telling the mirror admins to force syncs. Or do they expect the end user to contact all of the mirror admins and tell them what is going on.

I have been trying to figure out how these files are being created. I know that they are using the openSUSE Build Service to create the RPM. When you look at both versions of the RPM with “rpm -qip” you see they have the same build Time Stamp but different Time Stamps on the signature. So if it was the UPDATE project running twice or two different projects building the same RPM’s they would have different Build Time Stamps. This means a RPM is built signed place in the repo, then signed again and the second copy is delayed being put into the repo. If the second copy went in to the repo as soon as it was signed we would not have a problem because the mirrors would never see the first copy.

I see two problems causing this sync problem. One the RPM gets two signatures creating two copies of the RPM. Two the second copy is delayed, in some cases up to a month, being put into the mirror. Because of the delay the mirrors all pull down the first copy, and ignore the second. If the mirrors are syncing around every 4 hours (see openSUSE:Mirror infrastructure - openSUSE ), where are the second copies sitting. Their creation is with in minuets of the first copy.

If all of the mirrors did a full checksum sync today, the problem would be back next week unless the openSUSE team stopped the creation of the second copies.

Maybe other Forum Members need to add to the bug report, to show the openSUSE team that this very important issue.

Dave W.

DenverD · February 16, 2012, 9:55pm

On 02/16/2012 09:06 PM, dwestf wrote:
> Maybe other Forum Members need to add to the bug report, to show the
> openSUSE team that this very important issue.

i agree, but alas i am unqualified to make a comment there (as i’m still
unsure of what is going on)…but, i think the initial cause was a
complete and sudden loss of disk arrays at the openSUSE build service
about a month ago (and i think that is what Andreas Jaeger also thought
when he replied to the bug on 2012-01-24 16:28:08 UTC with “This is
beeing worked on - after the next build service publish of the update
repo, everything should be fine again.”

and, the way i read the bug: Marcus Rueckert said at 2012-02-02 12:23:55
UTC “This should be fixed now. If you can still see this issue please
reopen the bug.”

but, i don’t see that the bug was ever actually marked resolved, and is
now “need info” and the back and forth between you and Marcus looks like
one of you (and i don’t know which) is not following what the other is
saying, or trying to say…

someone who understands better than i should set one of you on the
right course…

–
DD http://tinyurl.com/SUSEonDW