Yet another zypper async CLI program (3-4x faster)

Hey everyone,

I’ve been running Tumbleweed for a few weeks now, quite new to rolling release distros other than a brief stint a few years ago with EndeavourOS.

I’ve been running LMDE on my main workstation these last few years with my own poor man’s version of btrfs/snapper/apt config, finally working up the courage to move to Tumbleweed, have it already running on my secondary machine and a VM.

Anyway, while I mostly like zypper I wish the async features would be released soon(er). The Github issue (feature request) for it has been going on for over 8 years at this point but there’s hope on the horizon as the PR for it is almost fully realized pending some major decisions on how to proceed without breaking existing API.

With that in mind, I created a small python program zypperoni :pizza: to refresh repos and download packages safely in an async manner. Only slow step now in my usual refresh/update workflow is the actual installation part itself which cannot be safely parallelized (or so I think).

I would appreciate any feedback on the project if you have some time to spare:

PRs and Issues are welcome :slight_smile:

Pavin.

4 Likes

Looks quite useful, however:

6700k:~ # zypperoni --debug refresh
2024-02-07 14:04:10,527: INFO: Getting all enabled repos
2024-02-07 14:04:10,557: DEBUG: Enabled repos: ['download.opensuse.org-non-oss', 'download.opensuse.org-oss', 'download.opensuse.org-tumbleweed', 'home_eyecreate_branches_filesystems', 'jalbum', 'packman']
2024-02-07 14:04:10,557: DEBUG: Using selector: EpollSelector
2024-02-07 14:04:10,557: INFO: Refreshing repo [1/6] 'download.opensuse.org-non-oss'
2024-02-07 14:04:10,658: INFO: Refreshing repo [2/6] 'download.opensuse.org-oss'
2024-02-07 14:04:10,760: INFO: Refreshing repo [3/6] 'download.opensuse.org-tumbleweed'
2024-02-07 14:04:10,861: INFO: Refreshing repo [4/6] 'home_eyecreate_branches_filesystems'
2024-02-07 14:04:10,961: INFO: Refreshing repo [5/6] 'jalbum'
2024-02-07 14:04:11,062: INFO: Refreshing repo [6/6] 'packman'
2024-02-07 14:04:12,277: ERROR: Error refreshing repo [1/6] 'download.opensuse.org-non-oss'. zypper exit code: 4
2024-02-07 14:04:12,277: DEBUG: [zypper error]
Problem retrieving files from 'Haupt-Repository (NON-OSS)'.
Download (curl) error for 'https://download.opensuse.org/tumbleweed/repo/non-oss/repodata/repomd.xml':
Error code: Connection failed
Error message: Could not resolve host: download.opensuse.org

Please see the above error message for a hint.
Skipping repository 'Haupt-Repository (NON-OSS)' because of the above error.
Could not refresh the repositories because of errors.

2024-02-07 14:04:12,300: ERROR: Error refreshing repo [3/6] 'download.opensuse.org-tumbleweed'. zypper exit code: 4
2024-02-07 14:04:12,300: DEBUG: [zypper error]
Problem retrieving files from 'Hauptaktualisierungs-Repository'.
Download (curl) error for 'https://download.opensuse.org/update/tumbleweed/repodata/repomd.xml':
Error code: Connection failed
Error message: Could not resolve host: download.opensuse.org

Please see the above error message for a hint.
Skipping repository 'Hauptaktualisierungs-Repository' because of the above error.
Could not refresh the repositories because of errors.

2024-02-07 14:04:12,330: ERROR: Error refreshing repo [4/6] 'home_eyecreate_branches_filesystems'. zypper exit code: 4
2024-02-07 14:04:12,330: DEBUG: [zypper error]
Problem retrieving files from 'Branch project for package btrbk (openSUSE_Tumbleweed)'.
Download (curl) error for 'https://download.opensuse.org/repositories/home:/eyecreate:/branches:/filesystems/openSUSE_Tumbleweed/repodata/repomd.xml':
Error code: Connection failed
Error message: Could not resolve host: download.opensuse.org

Please see the above error message for a hint.
Skipping repository 'Branch project for package btrbk (openSUSE_Tumbleweed)' because of the above error.
Could not refresh the repositories because of errors.

2024-02-07 14:04:12,477: ERROR: Error refreshing repo [2/6] 'download.opensuse.org-oss'. zypper exit code: 4
2024-02-07 14:04:12,477: DEBUG: [zypper error]
Problem retrieving files from 'Haupt-Repository (OSS)'.
Download (curl) error for 'https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml':
Error code: Connection failed
Error message: Could not resolve host: download.opensuse.org

Please see the above error message for a hint.
Skipping repository 'Haupt-Repository (OSS)' because of the above error.
Could not refresh the repositories because of errors.

2024-02-07 14:04:12,649: ERROR: Error refreshing repo [5/6] 'jalbum'. zypper exit code: 4
2024-02-07 14:04:12,649: DEBUG: [zypper error]
Problem retrieving files from 'jalbum'.
Download (curl) error for 'https://jalbum.net/download/software/yumrepo/repodata/repomd.xml':
Error code: Connection failed
Error message: Could not resolve host: jalbum.net

Please see the above error message for a hint.
Skipping repository 'jalbum' because of the above error.
Could not refresh the repositories because of errors.

2024-02-07 14:04:12,891: ERROR: Error refreshing repo [6/6] 'packman'. zypper exit code: 4
2024-02-07 14:04:12,891: DEBUG: [zypper error]
Problem retrieving files from 'packman'.
Download (curl) error for 'http://ftp.fau.de/packman/suse/openSUSE_Tumbleweed/repodata/repomd.xml':
Error code: Connection failed
Error message: Could not resolve host: ftp.fau.de

Please see the above error message for a hint.
Skipping repository 'packman' because of the above error.
Could not refresh the repositories because of errors.

2024-02-07 14:04:13,070: INFO: Cleaning up temp mounts...
2024-02-07 14:04:13,457: INFO: Cleaning up temp directory...
6700k:~ # 

zypper refresh readily refreshes:

Repository 'Haupt-Repository (NON-OSS)' is up to date.                                                                                                                                                                                      
Repository 'Haupt-Repository (OSS)' is up to date.                                                                                                                                                                                          
Repository 'Hauptaktualisierungs-Repository' is up to date.                                                                                                                                                                                 
Repository 'Branch project for package btrbk (openSUSE_Tumbleweed)' is up to date.                                                                                                                                                          
Repository 'jalbum' is up to date.                                                                                                                                                                                                          
Repository 'packman' is up to date.                                                                                                                                                                                                         
All repositories have been refreshed.

real    0m0.980s
user    0m0.377s
sys     0m0.106s
6700k:~ # 

Cleaning everything:

6700k:~ # zypper clean --all
All repositories have been cleaned up.
6700k:~ # time zypper refresh
Retrieving repository 'Haupt-Repository (NON-OSS)' metadata ..........................................................................................................................................................................[done]
Building repository 'Haupt-Repository (NON-OSS)' cache ...............................................................................................................................................................................[done]
Retrieving repository 'Haupt-Repository (OSS)' metadata ..............................................................................................................................................................................[done]
Building repository 'Haupt-Repository (OSS)' cache ...................................................................................................................................................................................[done]
Retrieving repository 'Hauptaktualisierungs-Repository' metadata .....................................................................................................................................................................[done]
Building repository 'Hauptaktualisierungs-Repository' cache ..........................................................................................................................................................................[done]
Retrieving repository 'Branch project for package btrbk (openSUSE_Tumbleweed)' metadata ..............................................................................................................................................[done]
Building repository 'Branch project for package btrbk (openSUSE_Tumbleweed)' cache ...................................................................................................................................................[done]
Retrieving repository 'jalbum' metadata ..............................................................................................................................................................................................[done]
Building repository 'jalbum' cache ...................................................................................................................................................................................................[done]
Retrieving repository 'packman' metadata .............................................................................................................................................................................................[done]
Building repository 'packman' cache ..................................................................................................................................................................................................[done]
All repositories have been refreshed.

real    0m16.191s
user    0m5.268s
sys     0m1.740s
6700k:~ # 
1 Like

Thanks for the debug output! I suppose you’re not using the default NetworkManager for managing DNS queries. Could you let me know which program is managing the DNS on your system (resolvconf/systemd-resolved, etc.). Usually /etc/resolv.conf should say something useful about this.

Pavin.

Host 6700k uses NetworkManager defaults:

6700k:~ # cat /etc/NetworkManager/system-connections/FRITZ\!Box\ Karl\ Mistelberger.nmconnection 
[connection]
id=FRITZ!Box Karl Mistelberger
uuid=948baf91-cbcf-4b99-834a-931621e03e66
type=wifi
permissions=user:karl:;
timestamp=1697970093

[wifi]
mode=infrastructure
ssid=FRITZ!Box Karl Mistelberger

[wifi-security]
key-mgmt=wpa-psk
psk=1234567890....

[ipv4]
method=auto

[ipv6]
addr-gen-mode=stable-privacy
method=auto

[proxy]
6700k:~ # 

More:

6700k:~ # curl -I https://download.opensuse.org/tumbleweed/repo/oss/x86_64/0ad-0.0.26-3.2.x86_64.rpm
HTTP/2 302 
date: Wed, 07 Feb 2024 13:51:50 GMT
server: Mojolicious (Perl)
cache-control: public, max-age=300
location: https://mirror.linux-schulserver.de/opensuse/tumbleweed/repo/oss/x86_64/0ad-0.0.26-3.2.x86_64.rpm
vary: Accept,COUNTRY,X-COUNTRY,Fastly-SSL
content-type: application/x-rpm

6700k:~ # 

Thanks, that’s quite a bit more than my basic config enp1s0.nmconnection:

[connection]
id=enp1s0
permissions=
interface-name=enp1s0
type=ethernet
[ipv4]
method=auto
[ipv6]
method=auto

Could you provide the outputs of:

cat /etc/resolv.conf
l /run

With zypperoni I took the safer route of bind mounting the root fs as read-only, with only /var/cache/zypp being read-write, to avoid any catastrophic issues. The problem with this deny-by-default, allow-by-choice approach is it could lead to edge cases like this.

Yep. It’s the same on all hosts, obfuscated addresses:

erlangen:~ # readlink /etc/resolv.conf 
/run/NetworkManager/resolv.conf
erlangen:~ # cat /etc/resolv.conf
# Generated by NetworkManager
search fritz.box
nameserver 192.168.178.1
nameserver fd00:xxxx:xxxx:...
nameserver 2001:xxxx:xxxx:..
erlangen:~ # 
1 Like

Thanks for all your help in debugging. You’re an awesome dude :grinning:
I’ve pushed an update to fix the DNS resolver issue.

Downloaded 2400 packages, 2.2 GiB:

Zypper download 15:14 min:

tw-netinstall:~ # journalctl -b -u dup.service -g 'Started|Consumed'
Feb 07 17:57:13 tw-netinstall systemd[1]: Started Upgrade System.
Feb 07 18:12:27 tw-netinstall systemd[1]: dup.service: Consumed 4min 17.755s CPU time.
tw-netinstall:~ # 

Parallel (10) zypper download 8:16 min:

tw-netinstall:~ # journalctl -b -u dupp.service -g 'Started|Consumed'
Feb 07 18:21:35 tw-netinstall systemd[1]: Started Upgrade System.
Feb 07 18:29:51 tw-netinstall systemd[1]: dupp.service: Consumed 28min 37.724s CPU time.
tw-netinstall:~ # 
1 Like

Zypperoni’s async nature seems much better suited for those like myself with high latency to mirrors. Barely 2x performance improvement in your case. On my best run I went from 30m to 5m with 20 parallel jobs downloading around 2000 packages.

Sad part is it still takes my modest machine around 30m to install those 2000 packages. :smiling_face_with_tear:

Installation: 12:07 min.

tw-netinstall:~ # journalctl -b -1 -u dup.service -g 'Started|Consumed'
Feb 07 18:53:55 tw-netinstall systemd[1]: Started Upgrade System.
Feb 07 19:06:02 tw-netinstall systemd[1]: dup.service: Consumed 8min 34.669s CPU time.
tw-netinstall:~ # 
tw-netinstall:~ # inxi -zSMC
System:
  Kernel: 6.7.2-1-default arch: x86_64 bits: 64
  Console: pty pts/0 Distro: openSUSE Tumbleweed 20240206
Machine:
  Type: Desktop Mobo: ASRock model: Z170 Pro4S serial: <filter> UEFI: American Megatrends v: P7.50
    date: 01/23/2018
CPU:
  Info: quad core model: Intel Core i7-6700K bits: 64 type: MT MCP cache: L2: 1024 KiB
  Speed (MHz): avg: 800 min/max: 800/4200 cores: 1: 800 2: 800 3: 800 4: 800 5: 800 6: 800
    7: 800 8: 800
tw-netinstall:~ # 
1 Like

Checked latency:

erlangen:~ # ping -c 3 cdn.opensuse.org
PING cdn.opensuse.org (2a04:4e42:8e::347) 56 data bytes
64 bytes from 2a04:4e42:8e::347: icmp_seq=1 ttl=60 time=10.7 ms
64 bytes from 2a04:4e42:8e::347: icmp_seq=2 ttl=60 time=11.2 ms
64 bytes from 2a04:4e42:8e::347: icmp_seq=3 ttl=60 time=11.8 ms

--- cdn.opensuse.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 10.693/11.233/11.770/0.439 ms
erlangen:~ # ping -c 3 download.opensuse.org
PING download.opensuse.org (2a07:de40:b250:131:10:151:131:30) 56 data bytes
64 bytes from 2a07:de40:b250:131:10:151:131:30: icmp_seq=1 ttl=58 time=24.3 ms
64 bytes from 2a07:de40:b250:131:10:151:131:30: icmp_seq=2 ttl=58 time=23.9 ms
64 bytes from 2a07:de40:b250:131:10:151:131:30: icmp_seq=3 ttl=58 time=23.8 ms

--- download.opensuse.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 23.773/23.993/24.347/0.252 ms
erlangen:~ # 

Switched to the faster option:

erlangen:~ # repos
#  | Alias                               | Enabled | GPG Check | Refresh | Priority | URI
---+-------------------------------------+---------+-----------+---------+----------+--------------------------------------------------------------------------------------------------
 7 | Packman                             | Yes     | (r ) Yes  | Yes     |   90     | http://ftp.fau.de/packman/suse/openSUSE_Tumbleweed/
23 | non-oss                             | Yes     | (r ) Yes  | Yes     |   99     | https://cdn.opensuse.org/tumbleweed/repo/non-oss/
25 | oss                                 | Yes     | (r ) Yes  | Yes     |   99     | https://cdn.opensuse.org/tumbleweed/repo/oss/
32 | update                              | Yes     | (r ) Yes  | Yes     |   99     | https://cdn.opensuse.org/update/tumbleweed/
 4 | Geo                                 | Yes     | (r ) Yes  | Yes     |  100     | https://cdn.opensuse.org/repositories/Application:/Geo/openSUSE_Tumbleweed/
12 | google-chrome                       | Yes     | (r ) Yes  | Yes     |  100     | https://dl.google.com/linux/chrome/rpm/stable/x86_64
18 | home_eyecreate_branches_filesystems | Yes     | (r ) Yes  | Yes     |  100     | https://cdn.opensuse.org/repositories/home:/eyecreate:/branches:/filesystems/openSUSE_Tumbleweed/
19 | home_kukuk_qmapshack                | Yes     | (r ) Yes  | Yes     |  100     | https://cdn.opensuse.org/repositories/home:/kukuk:/qmapshack/openSUSE_Tumbleweed/
20 | jalbum                              | Yes     | (  ) No   | Yes     |  100     | https://jalbum.net/download/software/yumrepo/
22 | myrepo                              | Yes     | (  ) No   | Yes     |  100     | dir:/home/karl/Downloads/myrepo
erlangen:~ # 

It’s pretty bad for me with 100ms latency to cdn.opensuse.org and 200ms to download.opensuse.org.

Btw, I released zypperoni v0.2 today with it supporting most of the major zypper commands :grinning: but breaking compatibility with zypperoni v0.1.x.

Pulled and installed new version (you may consider printing the version). Tried zypper download:

6700k:~ # time zypper -n install -dD texstudio 
Loading repository data...
Reading installed packages...
Resolving package dependencies...
...
Retrieving: texstudio-4.6.3-1.3.x86_64.rpm ...............................................................................................................................................................................[done (4.8 MiB/s)]

Checking for file conflicts: .........................................................................................................................................................................................................[done]

real    0m53.832s
user    0m10.567s
sys     0m4.786s
6700k:~ # 

zypperoni:

6700k:~ # time zypperoni --no-confirm in-download texstudio 
2024-02-09 11:50:34,293: INFO: Getting packages and its dependecies to be downloaded for installation
2024-02-09 11:50:35,118: INFO: Number of packages to download: 132
2024-02-09 11:50:35,118: INFO: Total download size: 170.42 MB
2024-02-09 11:50:35,118: INFO: Space usage difference after operation: +563.68 MB
2024-02-09 11:50:35,118: INFO: Downloading package [1/132] 'ghostscript-fonts-std'
...
2024-02-09 11:51:11,932: INFO: Cleaning up temp mounts...

2024-02-09 11:51:12,454: INFO: Cleaning up temp directory...

real    0m38.264s
user    2m3.254s
sys     0m16.276s
6700k:~ #

Even with the improved latency of cdn parallel zypperoni is is still somewhat faster than zypper download.

1 Like

Appreciate the feedback, I have added an option to view the version string.

@pavinjoseph
Hi Pavin,
I wrote another parallel zypper downloader it has a different take unlike zypperoni which makes no configuration changes zkk GitHub - wcomegys/zkk.opensuse
does make configuration changes. I build a folder for a configuration file and a sub folder for parallel repository files. These repository files support tracking for if the URI string is enabled speed number of successful downloads failed download date time of failures. If the aria2 download fails the download program (ZK_downLoader) can pick a new mirror and try again up to the failure number. Each time a new download is started the (zkk) repository file for that download is read and sorted so the fastest mirrors are at the top. I have good documentation so I would appreciate you taking a look. Maybe we can work to create a new zypper standard for parallel downloads and each repository having multiple mirrors with download speeds tracked.
Thanks
William
PS I have multiple routers I find the WIFI router that is more at the front of the Alphabet seems to be better at DNS.

If you want to create “a new zypper standard” you need to bring up your ideas at the relevant places and work together with the zypper developers. This is the user forum.

Most users work with the well developed, tested and approved tools provided by the distribution. Not with untested 3rd party repos…

1 Like

@hui
Zypper users have been asking for these features for 10 years, and it has seen no action. I use the approved repo mirrors.
You can read this in the documentation, I did not direct the message to you. Maybe you should read the documentation before making things up in your mind.

I didn’t make up anything. If you want to see your stuff in the official distribution, work together with the devs. Not more, not less.

2 Likes

Fast indeed but doesn’t work in Leap 15.6.

Cool project, first time seeing one written in Rexx lang. :100:
Currently dealing with a high workload and some minor health issues, will circle back around and check it out in detail if my memory doesn’t entirely fail me. :slightly_smiling_face: