Is there an application to download a mirror of a repostory web site?

English Applications
1

I have a desktop with no Internet access. I’d like to be able to download repositories to a USB thumb drive and take them to the desktop. Is there a suitable application in openSUSE, or should I write a script to do it?

2

rsync is the tool for creating repository mirrors. I hope you have a big thumb drive.

3

I have no control over the remote machines. I need to download using the existing servers, which are all HTTP, hence my reference to web.

4

You can connect to openSUSE mirors using variously http:, ftp:, and rsync:. Not all mirrors support all protocols.
I do not know where you are or how you will connect to the openSUSE mirrors, but you can modify the following to suit. This has been crudely cut from my old mirroring scripts without testing.


#! /bin/bash
 
## local mirror of openSUSE repositories

fromStem="//anorien.csc.warwick.ac.uk/download.opensuse.org"
toStem="/repos/opensuse"

# 15.0-Update
path="update/leap/15.0"
localPath="$toStem/$path"
if  "$localPath/" ] ;then mkdir -p "$localPath/" ;else sudo chown -R $USER $localPath ;fi

rsync -avzPH --delete  --delete-excluded  --exclude "*.drpm"  rsync:$fromStem/$path/*  $localPath/

# 15.0-repo
path="/distribution/leap/15.0/repo"
localPath="$toStem/$path"
if  "$localPath/" ] ;then mkdir -p "$localPath/" ;else sudo chown -R $USER $localPath ;fi

rsync -avzPH --delete  --delete-excluded  rsync:$fromStem/$path/*  $localPath/
5

Hi
You can see the full list here, there is a column with supported protocols;
https://mirrors.opensuse.org/

6

While I may eventually want to download a mirror of the LEAP 15 repositories, the one’s I’m concerned with right now aren’t openSUSE, or even Linux. I know that there are HTTP-based mirror applications; what I’m asking is whether any of them are packaged for rpm-based installation on openSUSE.

7

Well, if you would mention those applications, that would help. But: Search

8

Hi
You mean like httrack?
https://www.httrack.com/


 zypper if httrack
Information for package httrack:
--------------------------------
Repository     : openSUSE-Tumbleweed-Oss
Name           : httrack
            Version        : 3.49.2-1.

Then there is curl and wget…

9

Thanks; I’ve installed httrack.

Aren’t curl and wget for individual files? Do they have the ability to recurse into an entire directory tree?

10

On Thu 12 Jul 2018 08:06:03 PM CDT, shmuelmetz wrote:

malcolmlewis;2873343 Wrote:
> Hi
> You mean like httrack?
> https://www.httrack.com/
>
> >
Code:

> >
> zypper if httrack
> Information for package httrack:
> --------------------------------
> Repository : openSUSE-Tumbleweed-Oss
> Name : httrack
> Version : 3.49.2-1.
>

> >
>
> Then there is curl and wget…

Thanks; I’ve installed httrack.

Aren’t curl and wget for individual files? Do they have the ability to
recurse into an entire directory tree?

Hi
From the man page of wget… :wink:


Wget can follow links in HTML, XHTML, and CSS pages, to create local
versions of remote web sites, fully recreating the directory structure
of the original site.  This is sometimes referred to as "recursive
downloading."  While doing that, Wget respects the Robot Exclusion
Standard (/robots.txt).  Wget can be instructed to convert the links in
downloaded files to point at the local files, for offline viewing.

See the recursive section in the man page for the details…

–
Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
SLES 15 | GNOME Shell 3.26.2 | 4.12.14-23-default
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below… Thanks!