I’ve been running Tumbleweed for a few weeks now, quite new to rolling release distros other than a brief stint a few years ago with EndeavourOS.
I’ve been running LMDE on my main workstation these last few years with my own poor man’s version of btrfs/snapper/apt config, finally working up the courage to move to Tumbleweed, have it already running on my secondary machine and a VM.
Anyway, while I mostly like zypper I wish the async features would be released soon(er). The Github issue (feature request) for it has been going on for over 8 years at this point but there’s hope on the horizon as the PR for it is almost fully realized pending some major decisions on how to proceed without breaking existing API.
With that in mind, I created a small python program zypperoni to refresh repos and download packages safely in an async manner. Only slow step now in my usual refresh/update workflow is the actual installation part itself which cannot be safely parallelized (or so I think).
I would appreciate any feedback on the project if you have some time to spare:
6700k:~ # zypperoni --debug refresh
2024-02-07 14:04:10,527: INFO: Getting all enabled repos
2024-02-07 14:04:10,557: DEBUG: Enabled repos: ['download.opensuse.org-non-oss', 'download.opensuse.org-oss', 'download.opensuse.org-tumbleweed', 'home_eyecreate_branches_filesystems', 'jalbum', 'packman']
2024-02-07 14:04:10,557: DEBUG: Using selector: EpollSelector
2024-02-07 14:04:10,557: INFO: Refreshing repo [1/6] 'download.opensuse.org-non-oss'
2024-02-07 14:04:10,658: INFO: Refreshing repo [2/6] 'download.opensuse.org-oss'
2024-02-07 14:04:10,760: INFO: Refreshing repo [3/6] 'download.opensuse.org-tumbleweed'
2024-02-07 14:04:10,861: INFO: Refreshing repo [4/6] 'home_eyecreate_branches_filesystems'
2024-02-07 14:04:10,961: INFO: Refreshing repo [5/6] 'jalbum'
2024-02-07 14:04:11,062: INFO: Refreshing repo [6/6] 'packman'
2024-02-07 14:04:12,277: ERROR: Error refreshing repo [1/6] 'download.opensuse.org-non-oss'. zypper exit code: 4
2024-02-07 14:04:12,277: DEBUG: [zypper error]
Problem retrieving files from 'Haupt-Repository (NON-OSS)'.
Download (curl) error for 'https://download.opensuse.org/tumbleweed/repo/non-oss/repodata/repomd.xml':
Error code: Connection failed
Error message: Could not resolve host: download.opensuse.org
Please see the above error message for a hint.
Skipping repository 'Haupt-Repository (NON-OSS)' because of the above error.
Could not refresh the repositories because of errors.
2024-02-07 14:04:12,300: ERROR: Error refreshing repo [3/6] 'download.opensuse.org-tumbleweed'. zypper exit code: 4
2024-02-07 14:04:12,300: DEBUG: [zypper error]
Problem retrieving files from 'Hauptaktualisierungs-Repository'.
Download (curl) error for 'https://download.opensuse.org/update/tumbleweed/repodata/repomd.xml':
Error code: Connection failed
Error message: Could not resolve host: download.opensuse.org
Please see the above error message for a hint.
Skipping repository 'Hauptaktualisierungs-Repository' because of the above error.
Could not refresh the repositories because of errors.
2024-02-07 14:04:12,330: ERROR: Error refreshing repo [4/6] 'home_eyecreate_branches_filesystems'. zypper exit code: 4
2024-02-07 14:04:12,330: DEBUG: [zypper error]
Problem retrieving files from 'Branch project for package btrbk (openSUSE_Tumbleweed)'.
Download (curl) error for 'https://download.opensuse.org/repositories/home:/eyecreate:/branches:/filesystems/openSUSE_Tumbleweed/repodata/repomd.xml':
Error code: Connection failed
Error message: Could not resolve host: download.opensuse.org
Please see the above error message for a hint.
Skipping repository 'Branch project for package btrbk (openSUSE_Tumbleweed)' because of the above error.
Could not refresh the repositories because of errors.
2024-02-07 14:04:12,477: ERROR: Error refreshing repo [2/6] 'download.opensuse.org-oss'. zypper exit code: 4
2024-02-07 14:04:12,477: DEBUG: [zypper error]
Problem retrieving files from 'Haupt-Repository (OSS)'.
Download (curl) error for 'https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml':
Error code: Connection failed
Error message: Could not resolve host: download.opensuse.org
Please see the above error message for a hint.
Skipping repository 'Haupt-Repository (OSS)' because of the above error.
Could not refresh the repositories because of errors.
2024-02-07 14:04:12,649: ERROR: Error refreshing repo [5/6] 'jalbum'. zypper exit code: 4
2024-02-07 14:04:12,649: DEBUG: [zypper error]
Problem retrieving files from 'jalbum'.
Download (curl) error for 'https://jalbum.net/download/software/yumrepo/repodata/repomd.xml':
Error code: Connection failed
Error message: Could not resolve host: jalbum.net
Please see the above error message for a hint.
Skipping repository 'jalbum' because of the above error.
Could not refresh the repositories because of errors.
2024-02-07 14:04:12,891: ERROR: Error refreshing repo [6/6] 'packman'. zypper exit code: 4
2024-02-07 14:04:12,891: DEBUG: [zypper error]
Problem retrieving files from 'packman'.
Download (curl) error for 'http://ftp.fau.de/packman/suse/openSUSE_Tumbleweed/repodata/repomd.xml':
Error code: Connection failed
Error message: Could not resolve host: ftp.fau.de
Please see the above error message for a hint.
Skipping repository 'packman' because of the above error.
Could not refresh the repositories because of errors.
2024-02-07 14:04:13,070: INFO: Cleaning up temp mounts...
2024-02-07 14:04:13,457: INFO: Cleaning up temp directory...
6700k:~ #
zypper refresh readily refreshes:
Repository 'Haupt-Repository (NON-OSS)' is up to date.
Repository 'Haupt-Repository (OSS)' is up to date.
Repository 'Hauptaktualisierungs-Repository' is up to date.
Repository 'Branch project for package btrbk (openSUSE_Tumbleweed)' is up to date.
Repository 'jalbum' is up to date.
Repository 'packman' is up to date.
All repositories have been refreshed.
real 0m0.980s
user 0m0.377s
sys 0m0.106s
6700k:~ #
Cleaning everything:
6700k:~ # zypper clean --all
All repositories have been cleaned up.
6700k:~ # time zypper refresh
Retrieving repository 'Haupt-Repository (NON-OSS)' metadata ..........................................................................................................................................................................[done]
Building repository 'Haupt-Repository (NON-OSS)' cache ...............................................................................................................................................................................[done]
Retrieving repository 'Haupt-Repository (OSS)' metadata ..............................................................................................................................................................................[done]
Building repository 'Haupt-Repository (OSS)' cache ...................................................................................................................................................................................[done]
Retrieving repository 'Hauptaktualisierungs-Repository' metadata .....................................................................................................................................................................[done]
Building repository 'Hauptaktualisierungs-Repository' cache ..........................................................................................................................................................................[done]
Retrieving repository 'Branch project for package btrbk (openSUSE_Tumbleweed)' metadata ..............................................................................................................................................[done]
Building repository 'Branch project for package btrbk (openSUSE_Tumbleweed)' cache ...................................................................................................................................................[done]
Retrieving repository 'jalbum' metadata ..............................................................................................................................................................................................[done]
Building repository 'jalbum' cache ...................................................................................................................................................................................................[done]
Retrieving repository 'packman' metadata .............................................................................................................................................................................................[done]
Building repository 'packman' cache ..................................................................................................................................................................................................[done]
All repositories have been refreshed.
real 0m16.191s
user 0m5.268s
sys 0m1.740s
6700k:~ #
Thanks for the debug output! I suppose you’re not using the default NetworkManager for managing DNS queries. Could you let me know which program is managing the DNS on your system (resolvconf/systemd-resolved, etc.). Usually /etc/resolv.conf should say something useful about this.
With zypperoni I took the safer route of bind mounting the root fs as read-only, with only /var/cache/zypp being read-write, to avoid any catastrophic issues. The problem with this deny-by-default, allow-by-choice approach is it could lead to edge cases like this.
Zypperoni’s async nature seems much better suited for those like myself with high latency to mirrors. Barely 2x performance improvement in your case. On my best run I went from 30m to 5m with 20 parallel jobs downloading around 2000 packages.
Sad part is it still takes my modest machine around 30m to install those 2000 packages.
Pulled and installed new version (you may consider printing the version). Tried zypper download:
6700k:~ # time zypper -n install -dD texstudio
Loading repository data...
Reading installed packages...
Resolving package dependencies...
...
Retrieving: texstudio-4.6.3-1.3.x86_64.rpm ...............................................................................................................................................................................[done (4.8 MiB/s)]
Checking for file conflicts: .........................................................................................................................................................................................................[done]
real 0m53.832s
user 0m10.567s
sys 0m4.786s
6700k:~ #
zypperoni:
6700k:~ # time zypperoni --no-confirm in-download texstudio
2024-02-09 11:50:34,293: INFO: Getting packages and its dependecies to be downloaded for installation
2024-02-09 11:50:35,118: INFO: Number of packages to download: 132
2024-02-09 11:50:35,118: INFO: Total download size: 170.42 MB
2024-02-09 11:50:35,118: INFO: Space usage difference after operation: +563.68 MB
2024-02-09 11:50:35,118: INFO: Downloading package [1/132] 'ghostscript-fonts-std'
...
2024-02-09 11:51:11,932: INFO: Cleaning up temp mounts...
2024-02-09 11:51:12,454: INFO: Cleaning up temp directory...
real 0m38.264s
user 2m3.254s
sys 0m16.276s
6700k:~ #
Even with the improved latency of cdn parallel zypperoni is is still somewhat faster than zypper download.
@pavinjoseph
Hi Pavin,
I wrote another parallel zypper downloader it has a different take unlike zypperoni which makes no configuration changes zkk GitHub - wcomegys/zkk.opensuse
does make configuration changes. I build a folder for a configuration file and a sub folder for parallel repository files. These repository files support tracking for if the URI string is enabled speed number of successful downloads failed download date time of failures. If the aria2 download fails the download program (ZK_downLoader) can pick a new mirror and try again up to the failure number. Each time a new download is started the (zkk) repository file for that download is read and sorted so the fastest mirrors are at the top. I have good documentation so I would appreciate you taking a look. Maybe we can work to create a new zypper standard for parallel downloads and each repository having multiple mirrors with download speeds tracked.
Thanks
William
PS I have multiple routers I find the WIFI router that is more at the front of the Alphabet seems to be better at DNS.
If you want to create “a new zypper standard” you need to bring up your ideas at the relevant places and work together with the zypper developers. This is the user forum.
Most users work with the well developed, tested and approved tools provided by the distribution. Not with untested 3rd party repos…
@hui
Zypper users have been asking for these features for 10 years, and it has seen no action. I use the approved repo mirrors.
You can read this in the documentation, I did not direct the message to you. Maybe you should read the documentation before making things up in your mind.
Cool project, first time seeing one written in Rexx lang.
Currently dealing with a high workload and some minor health issues, will circle back around and check it out in detail if my memory doesn’t entirely fail me.
to refresh repos and download packages safely in an async manner. Only slow step now in my usual refresh/update workflow is the actual installation part itself which cannot be safely parallelized (or so I think).
