Page 1 of 3 123 LastLast
Results 1 to 10 of 24

Thread: Using rsync to clone http software repositories to a local folder

  1. #1
    Join Date
    Jan 2009
    Location
    Romania, Bucharest
    Posts
    795

    Default Using rsync to clone http software repositories to a local folder

    I'm still on the quest of switching from Windows to OpenSuse, hopefully during this year. I mostly solved my issues and differences with Win, except for a remaining concern. In Windows I keep installers for all programs I'm adding, so in case their website goes down or my internet connection fails I can still install my system and applications. Linux uses software repositories, and it's hard to have the safety that if my internet goes down for a day (or some custom repository disappears temporarily or permanently) I won't lose my software and the ability to install it again. Keeping the individual RPMs is hard due to dependencies, and because I have enough disk space I thought the best way to deal with this is by cloning software repositories to my drive, and keeping them up to date whenever possible.

    The best way I found is rsync, which seems just perfect for what I want. But I can't understand how to configure it so that it clones software repositories from HTTP. I looked a lot on google and asked on IRC (someone posted a wiki page where I didn't find the answer). It explains how to clone from a rsync repository, how to use rsync to synchronize a local folder to another, but not how to download a http / ftp folder structure to a local folder. In the examples I seen, you use something like rsync.opensuse.org:penSuse where the :: doesn't make sense since it's not part of the URL, so it doesn't work with normal paths. Basically, what I'm looking for is a command like this:

    Code:
    rsync -options --delete download.opensuse.org/distribution/12.1/repo/oss/ /home/mircea/repositories/opensuse-12.1-oss/
    Does anyone know the proper command line? Yes, I know there is also a rsync.opensuse.org page, but I also want to use this for repositories that might not have a rsync server, so I just download the whole folder structure instead. Also I know the official repositories are large, but as long as they're not over 100GB I can handle. This is important for me so I can use OpenSuse safely, so please let me know how it's done.
    openSUSE Tumbleweed x64, KDE Framework 5

  2. #2
    Join Date
    Mar 2010
    Location
    Austin - Texas
    Posts
    10,140
    Blog Entries
    48

    Smile Re: Using rsync to clone http software repositories to a local folder

    Quote Originally Posted by MirceaKitsune View Post
    I'm still on the quest of switching from Windows to OpenSuse, hopefully during this year. I mostly solved my issues and differences with Win, except for a remaining concern. In Windows I keep installers for all programs I'm adding, so in case their website goes down or my internet connection fails I can still install my system and applications. Linux uses software repositories, and it's hard to have the safety that if my internet goes down for a day (or some custom repository disappears temporarily or permanently) I won't lose my software and the ability to install it again. Keeping the individual RPMs is hard due to dependencies, and because I have enough disk space I thought the best way to deal with this is by cloning software repositories to my drive, and keeping them up to date whenever possible.

    The best way I found is rsync, which seems just perfect for what I want. But I can't understand how to configure it so that it clones software repositories from HTTP. I looked a lot on google and asked on IRC (someone posted a wiki page where I didn't find the answer). It explains how to clone from a rsync repository, how to use rsync to synchronize a local folder to another, but not how to download a http / ftp folder structure to a local folder. In the examples I seen, you use something like rsync.opensuse.org:penSuse where the :: doesn't make sense since it's not part of the URL, so it doesn't work with normal paths. Basically, what I'm looking for is a command like this:

    Code:
    rsync -options --delete download.opensuse.org/distribution/12.1/repo/oss/ /home/mircea/repositories/opensuse-12.1-oss/
    Does anyone know the proper command line? Yes, I know there is also a rsync.opensuse.org page, but I also want to use this for repositories that might not have a rsync server, so I just download the whole folder structure instead. Also I know the official repositories are large, but as long as they're not over 100GB I can handle. This is important for me so I can use OpenSuse safely, so please let me know how it's done.
    Have a look at my bash script intended to do what you are asking about here:

    S.L.R.C. - SuSE Local Repository Creator - Version 1.20 - Now for Packman & openSUSE 11.4 & 12.1 - Blogs - openSUSE Forums

    Thank You,
    My Blog: https://forums.opensuse.org/blogs/jdmcdaniel3/

    Software efficiency halves every 18 months, thus compensating for Moore's Law

    Its James again from Austin, Texas

  3. #3
    Join Date
    Jan 2009
    Location
    Romania, Bucharest
    Posts
    795

    Default Re: Using rsync to clone http software repositories to a local folder

    Quote Originally Posted by jdmcdaniel3 View Post
    Have a look at my bash script intended to do what you are asking about here:

    S.L.R.C. - SuSE Local Repository Creator - Version 1.20 - Now for Packman & openSUSE 11.4 & 12.1 - Blogs - openSUSE Forums

    Thank You,
    Thanks, that's also very useful. Since I'm still new and don't feel like modding other user's scripts for now (and rsync looks great for this too), I'd still like to know exactly what the rsync command is to clone a simple http or ftp folder structure. If I'm not comfy with that I might try other scripts for this purpose, but I'd like to try this first.
    openSUSE Tumbleweed x64, KDE Framework 5

  4. #4
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    4,654

    Default Re: Using rsync to clone http software repositories to a local folder

    I have not tried it myself, but the first thing I see in your command is
    that you do not use a proper URL
    Code:
    download.opensuse.org/distribution/12.1/repo/oss/
    so you should at least access a http resource by using the http path.
    Code:
    http://download.opensuse.org/distribution/12.1/repo/oss/
    --
    PC: oS 12.1 x86_64 | i7-2600@3.40GHz | 16GB | KDE 4.8.3 | GeForce GT 420
    ThinkPad E320: oS 12.1 x86_64 | i3@2.30GHz | 8GB | KDE 4.8.3 | HD 3000
    eCAFE 800: oS 12.1 i586 | AMD Geode LX 800@500MHz | 512MB | KDE 3.5.10

  5. #5
    Join Date
    Jan 2009
    Location
    Romania, Bucharest
    Posts
    795

    Default Re: Using rsync to clone http software repositories to a local folder

    Quote Originally Posted by martin_helm View Post
    I have not tried it myself, but the first thing I see in your command is
    that you do not use a proper URL
    Code:
    download.opensuse.org/distribution/12.1/repo/oss/
    so you should at least access a http resource by using the http path.
    Code:
    http://download.opensuse.org/distribution/12.1/repo/oss/
    Sorry, that's because I tried it last night and rsync always said http:// is not recognized. So it gave me the impression it adds that on its own. But I already tried both with and without, and neither work.
    openSUSE Tumbleweed x64, KDE Framework 5

  6. #6
    Join Date
    Jun 2008
    Location
    Netherlands
    Posts
    25,368

    Default Re: Using rsync to clone http software repositories to a local folder

    Did you look into HTTrack? I have good experiences in using it to mirror (Parts of) web-sites to my system.
    The web-site is here: HTTrack Website Copier - Free Software Offline Browser (GNU GPL)
    And the opeenSUSE installables are here: software.opensuse.org:

    Also wget might be an option for you (it is allready on your openSUSE):
    Code:
    man wget
    I make my backups over the LAN using rsync/rsyncd, But it needs rsyncd running at the server side and I doubt that softewate opensuse.orgruns such an rsynx server. It is an http server.

    These are different protocols (and they use different ports by default, 80 resp.873).
    Henk van Velden

  7. #7
    Join Date
    Feb 2010
    Location
    Germany
    Posts
    4,654

    Default Re: Using rsync to clone http software repositories to a local folder

    Taking quickly a deeper look, isn't it something like for example
    Code:
    rsync -rltp \
    rsync://ftp5.gwdg.de/pub/opensuse/distribution/12.1/repo/oss \
    ~/repo/
    that seems to work for me (of course choose your favorite mirror with
    rsync support and a local path of your choice).

    --
    PC: oS 12.1 x86_64 | i7-2600@3.40GHz | 16GB | KDE 4.8.3 | GeForce GT 420
    ThinkPad E320: oS 12.1 x86_64 | i3@2.30GHz | 8GB | KDE 4.8.3 | HD 3000
    eCAFE 800: oS 12.1 i586 | AMD Geode LX 800@500MHz | 512MB | KDE 3.5.10

  8. #8
    Join Date
    Jun 2008
    Location
    Netherlands
    Posts
    25,368

    Default Re: Using rsync to clone http software repositories to a local folder

    Quote Originally Posted by MirceaKitsune View Post
    Sorry, that's because I tried it last night and rsync always said http:// is not recognized. So it gave me the impression it adds that on its own. But I already tried both with and without, and neither work.
    To add again to the point where your misunderstanding is:
    rsync does NOT use the HTTP protocol, thus adding http:// will not help (and as you saw it only makes the syntax of the parameter ununderstandable by rsync) and omitting it will not add it as a default like in a HTTP browser.
    Henk van Velden

  9. #9
    Join Date
    Jun 2008
    Location
    Netherlands
    Posts
    25,368

    Default Re: Using rsync to clone http software repositories to a local folder

    Quote Originally Posted by martin_helm View Post
    Taking quickly a deeper look, isn't it something like for example
    Code:
    rsync -rltp \
    rsync://ftp5.gwdg.de/pub/opensuse/distribution/12.1/repo/oss \
    ~/repo/
    that seems to work for me (of course choose your favorite mirror with
    rsync support and a local path of your choice).

    --
    PC: oS 12.1 x86_64 | i7-2600@3.40GHz | 16GB | KDE 4.8.3 | GeForce GT 420
    ThinkPad E320: oS 12.1 x86_64 | i3@2.30GHz | 8GB | KDE 4.8.3 | HD 3000
    eCAFE 800: oS 12.1 i586 | AMD Geode LX 800@500MHz | 512MB | KDE 3.5.10
    They seem to run an rsync deamon then, but I guess tthat they are about the only one of all the mirrors.
    Having said that, GWDG is of course one of the most reliable mirrors of all.
    Henk van Velden

  10. #10
    Join Date
    Jun 2008
    Location
    Netherlands
    Posts
    25,368

    Default Re: Using rsync to clone http software repositories to a local folder

    And @MirceaKitsune, did you study
    Code:
    man rsync
    There is at the very beginning:
    Code:
    Access via remote shell:
      Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
      Push: rsync [OPTION...] SRC... [USER@]HOST:DEST
    
    Access via rsync daemon:
      Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]
            rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
      Push: rsync [OPTION...] SRC... [USER@]HOST::DEST
            rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
    It shows that for every syntax containing a remote system at least one : is somehere in the parameter. Thus simply using a hostname and a path added after a / will be interpreted as a path only (and I asume that the directory softeware.opensuse.org is not on your system).
    And as said above the protocol definition http:// is not mentioned at all, but rsync:// is.
    Henk van Velden

Page 1 of 3 123 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •