Parallel download of rpm packages

[karlmistelberger]

Another thing to consider: If all packages are cached, it can still happen that new ones are added and repos need refreshing. What i've ran into, is that cached packages did not meet the checksums provided from the repo refresh, and would generate errors.

That said, I can see the idea of paralel downloading to the max bandwidth.

And that said, I'm in NL with a very stable 300/30 Mbit internet connection and apparently a couple of good mirrors.

Well, no need for mirrors, as openSUSE resides round the corner:

erlangen:~ # ping -c 1 pontifex.opensuse.org
PING pontifex.opensuse.org(pontifex.opensuse.org (2620:113:80c0:8::13)) 56 data bytes
64 bytes from pontifex.opensuse.org (2620:113:80c0:8::13): icmp_seq=1 ttl=59 time=16.0 ms

--- pontifex.opensuse.org ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 16.002/16.002/16.002/0.000 ms
erlangen:~ #

I am paying for 25/5 Mbit only, but updating works like a charm. Nice to hear mirroring works well 20,000 km away.

[awerlang]

Another thing to consider: If all packages are cached, it can still happen that new ones are added and repos need refreshing. What i've ran into, is that cached packages did not meet the checksums provided from the repo refresh, and would generate errors.

Good catch! I have disabled auto refreshing, but I should either find a way to download new packages (maybe parsing zypper dup output or somehow calling libzypp directly), or follow with a zypper dup giving up on parallel download for those.

Well, no need for mirrors, as openSUSE resides round the corner:

erlangen:~ # ping -c 1 pontifex.opensuse.org
PING pontifex.opensuse.org(pontifex.opensuse.org (2620:113:80c0:8::13)) 56 data bytes
64 bytes from pontifex.opensuse.org (2620:113:80c0:8::13): icmp_seq=1 ttl=59 time=16.0 ms

--- pontifex.opensuse.org ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 16.002/16.002/16.002/0.000 ms
erlangen:~ #

I am paying for 25/5 Mbit only, but updating works like a charm. Nice to hear mirroring works well 20,000 km away.

I can see why you're not worried about that

Consider the server closest to me (756km):

PING opensuse.c3sl.ufpr.br(opensuse.c3sl.ufpr.br (2801:82:80ff:8000::b)) 56 data bytes
64 bytes from opensuse.c3sl.ufpr.br (2801:82:80ff:8000::b): icmp_seq=1 ttl=43 time=579 ms

--- opensuse.c3sl.ufpr.br ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 578.850/578.850/578.850/0.000 ms

Repeat a thousand times... Serial throughtput is 300-1000 kB/s. There's a faster server (1228 km) but unfortunately it only serves Leap packages. I reached out to them but the email returned. Maybe I'll switch to a farther away server.

[awerlang]

So I looked into ftp (no quality servers nearby), curl --parallel (unreliable), curl url-list... (okay), pick local mirror directly (better), and then I settled with aria2. aria2 can take advantage of global mirror infrastructure to split resources into parts. From 20m down to 3:30m. I think I can stop squeezing the last bit of speed by now.

Thanks everyone for your input!