Can't trust normal rsync of root because zypper dup may not change file dates!

English Install/Boot/Login
1

I have been using rsync to clone ext4 root partitions, something along the lines of the following…

# rsync -ax -HAX --delete --devices --sparse --itemize-changes / /mnt/altroot/

It seemed to work quite well, or at least I though it did, but in the last few weeks, it started to miss files because some zypper dup updated files had the same size and date. I user `rsync --checksum`` to investigate further:

rsync -ax -HAX --delete --dry-run --checksum --devices --sparse --itemize-changes / /mnt/altroot/

More than 100 files were different. I picked a text file for closer examination…

# ls -l /usr/share/man/man1/spamc.1.gz /mnt/altroot/usr/share/man/man1/spamc.1.gz 
-r--r--r-- 1 root root 6254 Jan 21 02:29 /mnt/altroot/usr/share/man/man1/spamc.1.gz
-r--r--r-- 1 root root 6254 Jan 21 02:29 /usr/share/man/man1/spamc.1.gz

# rpm -q -f /usr/share/man/man1/spamc.1.gz
spamassassin-spamc-4.0.1-78.3.x86_64
# rpm -V spamassassin-spamc-4.0.1-78.3.x86_64

# chroot /mnt/altroot/
# rpm -q -f /usr/share/man/man1/spamc.1.gz
spamassassin-spamc-4.0.1-78.3.x86_64
kosmos1.gentoo.co.nz:/ # rpm -V spamassassin-spamc-4.0.1-78.3.x86_64
..5......    /usr/bin/spamc
..5......  d /usr/share/man/man1/spamc.1.gz

In the above, the rpm -V listed no issues for /, but incorrect md5sums for two spamc files on /mnt/altroot.

From the above, it seemed that some files were updated by the zypper dup without altering their dates and sizes. I took a look and confirmed it:

# zcat /usr/share/man/man1/spamc.1.gz > root-spamc.1
# zcat /mnt/altroot/usr/share/man/man1/spamc.1.gz > altroot-spamc.1
# diff root-spamc.1altroot-spamc.1
58c58
< .TH SPAMC 1 2025-01-20 "perl v5.40.1" "User Contributed Perl Documentation"
---
> .TH SPAMC 1 2025-01-20 "perl v5.40.0" "User Contributed Perl Documentation"

I then checked the mount options, they were rw,noatime, so that doesn’t explain it.

It seems zypper dup or the build process may update files without changing their dates .

For safety I need to pass --checksum to rsync if I’m cloning root partitions. (Fortunately, they’re on SSD and NVME.)

2

I don’t have a solution … this is just an opinion post. (I know, some folks are thinking, “not again”).

We only backup /home … we don’t care about the / filesystem. If it goes catastrophic, way easier to install from scratch. If we were to ever want to back it up, would use an imaging solution. And yea, I suspect some folks have “involved” / (filesystem) setups.

We do strictly use rsync for /home backups. Fast and efficient and dependable.

Your research suggests the nature of zypper runs have [unexpected] hidden side effects. Maybe someone will have a solution… or a bug report is in order ??

3

@myswtest Likewise, but I’m not even interested in home… the important stuff I have on /data and use soflinks in home. That all gets rsync over ssh to another system. Then I have scripts to run as root and user to recreate everything. The good thing is I can use the scripts with combustion as well for MicroOS installs if required :wink:

2 Likes
4

I do also use rsync for home, for both online and offline backups. Occasionally I throw in a --checksum and come back in a couple of days. it’s never resulted in any surprises.

It’s possible zypper or the build-system has always done this. I only noticed because of multiple resource to rollback in recent times due to Plasma-6 and Nvidia-drivers (Plasma-5+X11 was a rock, Plasma-6+X11, not so much). Using two bootable roots is a quick dig-out (my scripts can clone in either direction).

I do also like two bootable roots. It’s a much faster dig out than a reinstall. When anything odd occurs it’s quite handy to have two different vintages of TW immediately available. I will persist with this approach even if I have to wait a few minutes for --checksum to work through.

5

May I kindly ask you to stop using bold so much?

zypper sets time to whatever time is stored in the package metadata. openSUSE is using reproducible builds so it is quite possible that two releases of the same package are using the same time, and of course there is nothing wrong with having the same file sizes in general.

Two different versions of the same package are expected to use different timestamps though. But in your case there is no new version, the package has been rebuilt against new Perl version. Unfortunately, this Perl version also happened to be recorded in the package files.

@bmwiedemann I guess it is something for you :slight_smile:

1 Like
6

The code is supposed to add the rebuild counter to the latest changelog timestamp, so in theory new versions should get new timestamps.

Maybe if you do a vendor-change there could be collisions. Or you use packages from a home: project that does not update changelogs?

Btw: ls only shows minutes, so you would need to use stat to see the difference, but rsync should respect a one-second-diff.

If nothing else helps, rsync -c does.

7

You miss the point. The version of the spamassasin-spamc did not change. But it depends on Perl and the version of Perl changed and spamassasin has been rebuilt against this new version; only rebuild counter changed. It is just that the Perl version is also recorded in the man page of the spamassasin-spamc when it is built.

8

If I understand correctly, spamc source was not changed, but it was rebuilt, and the rebuild resulted in pulling in differences, so the rebuilt packaged rpm is different, event though the original source is the same.

That would explain the other 100+ differences I encountered, which included executable and .so files. For example, if C macro in a dependency was to change, a package using that macro might not have changed, but it will be rebuilt and the rpm content may potentially be different, but the rpm timestamps will remain the same.

Given the above, it seems that Rsync backup of a OpenSUSE root without --checksum is unsafe (although it’s fine if the destination starts empty).

9

Look at https://build.opensuse.org/public/build/openSUSE:Factory/standard/x86_64/spamassassin/_log

The build happens at Tue Apr 15 12:23:07 UTC 2025. The SOURCE_DATE_EPOCH_MTIME is set to 1737379794 == Mon Jan 20 13:29:54 UTC 2025. The SOURCE_DATE_EPOCH is set to 1737379791 == Mon Jan 20 13:29:51 UTC 2025. That loosely (time zone unknown) corresponds to

and

10

spamassassin-4.0.1-78.3.src.rpm was produced with a rebuild counter of 3 - and that is included in SOURCE_DATE_EPOCH_MTIME . But maybe that one is not used anymore? Some patches were dropped in the move to rpm-4.20 some months ago.

1 Like
11

Thanks for investigating. That may possibly explain why my cloned root scheme started to have issues, and I discovered I needed to use --checksum. Last year it had been working well without requiring that option.

12

If you still have the original “bad” state, the output of

ls --full-time ...

for both file copies would be useful. So far everything is just guesswork.

13

I’d already corrected the alternate root, but I thought of another way to check. I keep /var/cache/zypp to seed the cache on other desktops. I investigated further by reinstalling the build counter 1, 2 and 3 rpms from the cache.

The build-counter-1 is dated slight newer than 2 and 3. Build-counter-2 and 3 have exactly the same date (and size), which is almost certainly the cause of the issue I encountered.

# rpm --upgrade --force /var/cache/zypp/packages/download.opensuse.org-oss/x86_64/spamassassin-spamc-4.0.1-78.1.x86_64.rpm 
# ls --full-time /usr/share/man/man1/spamc.1.gz 
-r--r--r-- 1 root root 6254 2025-01-21 02:29:52.000000000 +1300 /usr/share/man/man1/spamc.1.gz

# rpm --upgrade --force /var/cache/zypp/packages/download.opensuse.org-oss/x86_64/spamassassin-spamc-4.0.1-78.2.x86_64.rpm 
kosmos1:~ # ls --full-time /usr/share/man/man1/spamc.1.gz 
-r--r--r-- 1 root root 6254 2025-01-21 02:29:51.000000000 +1300 /usr/share/man/man1/spamc.1.gz

# rpm --upgrade --force /var/cache/zypp/packages/download.opensuse.org-oss/x86_64/spamassassin-spamc-4.0.1-78.3.x86_64.rpm 
# ls --full-time /usr/share/man/man1/spamc.1.gz 
-r--r--r-- 1 root root 6254 2025-01-21 02:29:51.000000000 +1300 /usr/share/man/man1/spamc.1.gz

I went on to confirm that the build-counter 2 file contained perl v5.40.0 and build-counter 3 contained perl v5.40.1.

14

That confirms what @bmwiedemann suggested - looks, like support for per-build counter timestamps was lost.

1 Like
15 
Wed Feb 12 13:36:45 CET 2025 - mls@suse.de

...
- drop unused 0001-Add-option-to-set-mtime-of-files-in-rpms.patch
  patch
1 Like
16

Yes, that is the one. I’m proposing to re-add part of it upstream = Add an option to use a different mtime by bmwiedemann · Pull Request #3728 · rpm-software-management/rpm · GitHub

1 Like
17

Trying another approach: Add macros for reproducible-builds by bmwiedemann · Pull Request #88 · openSUSE/rpm-config-SUSE · GitHub

1 Like
18

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.