download with rsync all packages and make local repo

On Sun, 28 Nov 2010 09:06:01 GMT, mfaridi
<mfaridi@no-mx.forums.opensuse.org> wrote:

>
>in home I do not have internet connection , but in work I have internet
>connection, in home I install Fedora 14 for my 6 years old daughter and
>she use it for play games like supertux and openarena and …
>So I want install opensuse for my daughter and I want test it .
>for fedora , I download all packages with rsync in work and move them
>to home by USN flash and then I make localrepo in home and install all
>packages , I need , I want do this for OpenSUSE , all of us know DVD ,
>does not has all packages , I need , so I have to download all packages
>and make localrepo in home and install all packages , I need ,
>Can I do this for OpenSuse or not ,
>I want download all packages need by OpenSuse by rsysc and make loacl
>repo,
>How I can do this for Opensuse ???

I just rsync the OSS, non-OSS, update and packman repos whole, about 90 GB
total and do not worry about trying to cut them back to 64-bit only. Much
of the executables are not in binaries but in executables in PHP, Perl.
Python, TCL/Tk, Java, etc (are architecture independent).

Having the whole banana makes it easier to load other new applications as
i wish.

This thread gave me a great idea so I am currently creating a local repo for 11.3 oss, non-oss, Packman and updates. Here is my rsync script:


#!/bin/bash                                                                                   
# rsync mirror.aarnet.edu.au and ftp5.gwdg.de mirror for openSUSE local repo

# 11.3 updates - mirror.aarnet.edu.au
rsync -a -v -v --delete-after --progress --stats mirror.aarnet.edu.au::opensuse/opensuse/update/11.3/ /data/repo/update/11.3/                                                                
                                                                                                                                                                                                
# 11.3 oss and non-oss - mirror.aarnet.edu.au
rsync -a -v -v --delete-after --progress --stats mirror.aarnet.edu.au::opensuse/opensuse/distribution/11.3/repo/ /data/repo/distribution/11.3/repo/

# 11.3 Packman - ftp5.gwdg.de
rsync -a -v -v --delete-after --progress --stats ftp5.gwdg.de::pub/linux/misc/packman/suse/11.3/ /data/repo/packman/suse/11.3/

Then I have apache running and it is pointing to that directory for Document root.
Created a openSUSE 11.3 Virtual Machine on my openSUSE 11.3 KVM host just to this this :slight_smile:

Another way might be to set up Apache as a forward proxy.

mod_proxy - Apache HTTP Server

This can map the URL space of the remote site into your local site, and it will only fetch on demand. You could combine it with a rsync scheme if you want to preseed your cache. However it doesn’t solve the offline access problem if you don’t have the package you want in the cache.

On 2010-12-04 23:06, ah7013 wrote:
>
> This thread gave me a great idea so I am currently creating a local repo
> for 11.3 oss, non-oss, Packman and updates. Here is my rsync script:

Hint: try “rsync.opensuse.org”.


Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

Would you recommend rsync.opensuse.org over mirror.aarnet.edu.au? My reason for using mirror.aarnet.edu.au is that I have found mirror.aarnet.edu.au to be fast (speed I get off that mirror is about the same as my max download speed)

On 2010-12-05 09:36, ah7013 wrote:
>
> robin_listas;2261868 Wrote:

>> Hint: try “rsync.opensuse.org”.

> Would you recommend rsync.opensuse.org over mirror.aarnet.edu.au? My
> reason for using mirror.aarnet.edu.au is that I have found
> mirror.aarnet.edu.au to be fast (speed I get off that mirror is about
> the same as my max download speed)

As a general recommendation, yes. Supossedly you get a mirror appropiate
for your location automatically.


Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

OK thanks for the advice. Once this rsync script is finished (still going, doing packman now then done) I will switch to rsync.opensuse.org. Downloaded 29GB so far :slight_smile:

I added this line to /etc/crontab:

0 23 * * * root /data/rsync-distro 2>&1

where /data/rsync-distro is my script from my above post. Should this work OK?

@ah7013: You did exactly the right thing: you took your local (fast) mirror. As seen from the opensuse server you are about the maximum possible distance away (at least as long as you remain on this planet). And rsync will NOT automatically switch to the nearest mirror. Do not change your winning horse.

OK I will stay on my local mirror then if there is no benefit switching to repo.opensuse.org

On 2010-12-05 22:06, ah7013 wrote:

> OK thanks for the advice. Once this rsync is finished (still going,
> doing packman now then done) I will switch to rsync.opensuse.org.
> Downloaded 29GB so far :slight_smile:

Try, and report back, so that we know if my suggestion was correct or wrong :slight_smile:


Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

On 2010-12-05 22:36, vodoo wrote:
>
> @ah7013: You did exactly the right thing: you took your local (fast)
> mirror. As seen from the opensuse server you are about the maximum
> possible distance away (at least as long as you remain on this planet).
> And rsync will NOT automatically switch to the nearest mirror. Do not
> change your winning horse.

No, rsync does not switch. But rsync.opensuse.org is supposed to be a
redirector, and gives you the IP of a suitable rsync server for your
location, automatically.


Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

But rsync.opensuse.org is supposed to be a redirector, and gives you the IP of a suitable rsync server for your location, automatically.

But - if this is true - why does it limit access to a maximum of 50 concurrent connections? So far I have never seen any redirection taking place.

Yes I saw that as well:


andrew@linux-j388:~/Downloads> rsync rsync.opensuse.org::

This is rsync.opensuse.org, public rsync server of openSUSE.org,
limited to 50 connections.

On 2010-12-06 10:36, vodoo wrote:
>
>> But rsync.opensuse.org is supposed to be a redirector, and gives you the
>> IP of a suitable rsync server for your location, automatically.
>
> But - if this is true - why does it limit access to a maximum of 50
> concurrent connections? So far I have never seen any redirection taking
> place.

I simply don’t know.

I remember time ago people that should know (ie, staff) telling to use that
address instead. I understood that it worked as a redirector, but I have no
proof. So I’m interested in knowing.

We could ask on one of the maillists.


Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

OK Here is an update:

rsync has finally finished and I now have my own repo :slight_smile:
Installed apache and pointed DocumentRoot to /data/repo and everything is working :slight_smile:

One thing I did is comment out the Packman rsync line becasuse from here the Packman mirror is too slow (I am certain its the distance not the mirror itself) and also Packman is HUGE :slight_smile:

But OSS, Non-OSS and Updates are working really well. I kept the old OSS, Non-OSS and Updates repos which point to download.opensuse.org in YaST in case I am going out but they are disabled. So when I go out I will enable these and disable my local mirror in YaST (I called it openSUSE-11.3-Oss-Local, openSUSE-11.3-Non-Oss-Local and openSUSE-11.3-Updates-Local).

Thanks all guys
I was in vacation and after 7 days I come here and I download all packages I need with rsync and all of them right now is 36GB and I want move them to house and Install OpenSuse 11.3 AMD 64 and I want copy all packages and make local reopo on my Hard disk , I do not want install Virtualbox or something else and I want make this local repo with

 zypper 

please guide me How I can do this with

zypper

?
I want copy all packages on home partition and make local repo with zypper

On 2010-12-13 15:36, mfaridi wrote:
> and I download all
> packages I need with rsync

Just a note.

You people should do the initial download as a normal download, ie, without
rsync. When done, then do the rsync to maintain it. Why? Because rsync is
heavy on the servers, few offer that service. If many people start using
it, they might have to consider dropping out.

The polite thing is then to download normally, using hhtp, and keep the
repo with rsync later.


Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

Given you have a fast line, and want to be nice to the server, you can add:

--bwlimit=KBPS          limit I/O bandwidth; KBytes per second

to your command line.