Results 1 to 10 of 10

Thread: Is there an application to download a mirror of a repostory web site?

  1. #1
    Join Date
    Jan 2018
    Location
    Annandale, VA
    Posts
    109

    Default Is there an application to download a mirror of a repostory web site?

    I have a desktop with no Internet access. I'd like to be able to download repositories to a USB thumb drive and take them to the desktop. Is there a suitable application in openSUSE, or should I write a script to do it?

  2. #2
    Join Date
    Oct 2008
    Location
    Glasgow, Scotland
    Posts
    1,131

    Default Re: Is there an application to download a mirror of a repostory web site?

    rsync is the tool for creating repository mirrors. I hope you have a big thumb drive.
    ~Thank you for sharing an interesting problem.
    --
    slàinte mhath,
    rayH

  3. #3
    Join Date
    Jan 2018
    Location
    Annandale, VA
    Posts
    109

    Default Re: Is there an application to download a mirror of a repostory web site?

    I have no control over the remote machines. I need to download using the existing servers, which are all HTTP, hence my reference to web.

  4. #4
    Join Date
    Oct 2008
    Location
    Glasgow, Scotland
    Posts
    1,131

    Default Re: Is there an application to download a mirror of a repostory web site?

    You can connect to openSUSE mirors using variously http:, ftp:, and rsync:. Not all mirrors support all protocols.
    I do not know where you are or how you will connect to the openSUSE mirrors, but you can modify the following to suit. This has been crudely cut from my old mirroring scripts without testing.

    Code:
    #! /bin/bash
     
    ## local mirror of openSUSE repositories
    
    fromStem="//anorien.csc.warwick.ac.uk/download.opensuse.org"
    toStem="/repos/opensuse"
    
    # 15.0-Update
    path="update/leap/15.0"
    localPath="$toStem/$path"
    if [ "$localPath/" ] ;then mkdir -p "$localPath/" ;else sudo chown -R $USER $localPath ;fi
    
    rsync -avzPH --delete  --delete-excluded  --exclude "*.drpm"  rsync:$fromStem/$path/*  $localPath/
    
    # 15.0-repo
    path="/distribution/leap/15.0/repo"
    localPath="$toStem/$path"
    if [ "$localPath/" ] ;then mkdir -p "$localPath/" ;else sudo chown -R $USER $localPath ;fi
    
    rsync -avzPH --delete  --delete-excluded  rsync:$fromStem/$path/*  $localPath/
    ~Thank you for sharing an interesting problem.
    --
    slàinte mhath,
    rayH

  5. #5
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    26,525
    Blog Entries
    15

    Default Re: Is there an application to download a mirror of a repostory web site?

    Quote Originally Posted by eng-int View Post
    You can connect to openSUSE mirors using variously http:, ftp:, and rsync:. Not all mirrors support all protocols.
    I do not know where you are or how you will connect to the openSUSE mirrors, but you can modify the following to suit. This has been crudely cut from my old mirroring scripts without testing.

    Code:
    #! /bin/bash
     
    ## local mirror of openSUSE repositories
    
    fromStem="//anorien.csc.warwick.ac.uk/download.opensuse.org"
    toStem="/repos/opensuse"
    
    # 15.0-Update
    path="update/leap/15.0"
    localPath="$toStem/$path"
    if [ "$localPath/" ] ;then mkdir -p "$localPath/" ;else sudo chown -R $USER $localPath ;fi
    
    rsync -avzPH --delete  --delete-excluded  --exclude "*.drpm"  rsync:$fromStem/$path/*  $localPath/
    
    # 15.0-repo
    path="/distribution/leap/15.0/repo"
    localPath="$toStem/$path"
    if [ "$localPath/" ] ;then mkdir -p "$localPath/" ;else sudo chown -R $USER $localPath ;fi
    
    rsync -avzPH --delete  --delete-excluded  rsync:$fromStem/$path/*  $localPath/
    Hi
    You can see the full list here, there is a column with supported protocols;
    https://mirrors.opensuse.org/
    Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
    SUSE SLE, openSUSE Leap/Tumbleweed (x86_64) | GNOME DE
    If you find this post helpful and are logged into the web interface,
    please show your appreciation and click on the star below... Thanks!

  6. #6
    Join Date
    Jan 2018
    Location
    Annandale, VA
    Posts
    109

    Default Re: Is there an application to download a mirror of a repostory web site?

    While I may eventually want to download a mirror of the LEAP 15 repositories, the one's I'm concerned with right now aren't openSUSE, or even Linux. I know that there are HTTP-based mirror applications; what I'm asking is whether any of them are packaged for rpm-based installation on openSUSE.

  7. #7
    Join Date
    Jun 2008
    Location
    Groningen, Netherlands
    Posts
    19,662
    Blog Entries
    14

    Default Re: Is there an application to download a mirror of a repostory web site?

    Quote Originally Posted by shmuelmetz View Post
    While I may eventually want to download a mirror of the LEAP 15 repositories, the one's I'm concerned with right now aren't openSUSE, or even Linux. I know that there are HTTP-based mirror applications; what I'm asking is whether any of them are packaged for rpm-based installation on openSUSE.
    Well, if you would mention those applications, that would help. But: https://software.opensuse.org/search
    ° Appreciate my reply? Click the star and let me know why.

    ° Perfection is not gonna happen. No way.

    https://en.opensuse.org/openSUSE:Board#Members
    http://en.opensuse.org/User:Knurpht
    http://nl.opensuse.org/Gebruiker:Knurpht

  8. #8
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    26,525
    Blog Entries
    15

    Default Re: Is there an application to download a mirror of a repostory web site?

    Hi
    You mean like httrack?
    https://www.httrack.com/

    Code:
     zypper if httrack
    Information for package httrack:
    --------------------------------
    Repository     : openSUSE-Tumbleweed-Oss
    Name           : httrack
                Version        : 3.49.2-1.
    Then there is curl and wget....
    Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
    SUSE SLE, openSUSE Leap/Tumbleweed (x86_64) | GNOME DE
    If you find this post helpful and are logged into the web interface,
    please show your appreciation and click on the star below... Thanks!

  9. #9
    Join Date
    Jan 2018
    Location
    Annandale, VA
    Posts
    109

    Default Re: Is there an application to download a mirror of a repostory web site?

    Quote Originally Posted by malcolmlewis View Post
    Hi
    You mean like httrack?
    https://www.httrack.com/

    Code:
     zypper if httrack
    Information for package httrack:
    --------------------------------
    Repository     : openSUSE-Tumbleweed-Oss
    Name           : httrack
                Version        : 3.49.2-1.
    Then there is curl and wget....
    Thanks; I've installed httrack.

    Aren't curl and wget for individual files? Do they have the ability to recurse into an entire directory tree?

  10. #10
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    26,525
    Blog Entries
    15

    Default Re: Is there an application to download a mirror of a repostory website?

    On Thu 12 Jul 2018 08:06:03 PM CDT, shmuelmetz wrote:

    malcolmlewis;2873343 Wrote:
    > Hi
    > You mean like httrack?
    > https://www.httrack.com/
    >
    > >

    Code:
    --------------------
    > >

    > zypper if httrack
    > Information for package httrack:
    > --------------------------------
    > Repository : openSUSE-Tumbleweed-Oss
    > Name : httrack
    > Version : 3.49.2-1.
    >

    --------------------
    > >

    >
    > Then there is curl and wget....


    Thanks; I've installed httrack.

    Aren't curl and wget for individual files? Do they have the ability to
    recurse into an entire directory tree?


    Hi
    From the man page of wget...
    Code:
    Wget can follow links in HTML, XHTML, and CSS pages, to create local
    versions of remote web sites, fully recreating the directory structure
    of the original site.  This is sometimes referred to as "recursive
    downloading."  While doing that, Wget respects the Robot Exclusion
    Standard (/robots.txt).  Wget can be instructed to convert the links in
    downloaded files to point at the local files, for offline viewing.
    See the recursive section in the man page for the details...

    --
    Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
    SLES 15 | GNOME Shell 3.26.2 | 4.12.14-23-default
    If you find this post helpful and are logged into the web interface,
    please show your appreciation and click on the star below... Thanks!


Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •