Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: updaterepos - skipping repos that are down and allowing multiple urls

  1. #1

    Default updaterepos - skipping repos that are down and allowing multiple urls

    This is a workaround to the US pacman mirror unavailability. It does refresh all enabled repos while skipping the ones that are down after a timeout (3 seconds by default). It also replaces unreachable URLS with the next available one, allowing the use of multiple URLs for repos and finally restores the default URL (the first one defined in the alias) when it is back. You can add other repos by setting a variable with the same name as their alias. This example includes only packman. This script might need some testing (meaning you're using it at your own risk.)

    Code:
    #! /bin/bash
    #
    #: Title       : updaterepos
    #: Date Created: Fri May 13 10:51:42 PDT 2011
    #: Last Edit   : Fri May 13 12:52:04 PDT 2011
    #: Author      : please_try_again 
    #: Version     : 1.0
    #: Description : Refreshes all enabled repos, while skipping repos that are down after a defined timeout
    #                Replaces repos that are down by the next available repo defined for their alias - if any  
    #                Replaces repos that are up   by the first repo defined for their alias if it is back online 
    
    # You can define the timeout
    TimeOut=3
    
    # check if we are root
    [ $UID -eq 0 ] || exec echo "You have to run this script as root or sudo"
    
    # use lsb_release to get openSUSE version
    which lsb_release >/dev/null 2>&1 || zypper -n in lsb-release
    
    suseRel=$(lsb_release -rs)
    
    # You can define multiples urls for each repo's alias, as in the packman example below.
    
    packman="http://packman.unixheads.com/suse/openSUSE_$suseRel/ \
    http://ftp.uni-erlangen.de/pub/mirrors/packman/suse/openSUSE_$suseRel/ \
    http://ftp.gwdg.de/pub/linux/misc/packman/suse/openSUSE_$suseRel/ \
    http://ftp.halifax.rwth-aachen.de/packman/suse/openSUSE_$suseRel/ \
    http://mirror.geht-schon.de/packman.links2linux.de/suse/openSUSE_$suseRel/ \
    http://packman.jacobs-university.de/suse/openSUSE_$suseRel/ \
    http://packman.inode.at/suse/openSUSE_$suseRel/"
    
    eval $(zypper lr -u | tr -d " " | awk -F "|" '/^[1-9]/{ E=tolower($4) ; printf "ralias[%s]=%s;rurl[%s]=%s;enable[%s]=%s;", $1, $2, $1, $6, $1, E}')
    
    i=0
    while [ $i -le ${#ralias[*]} ] ; do
    	let i++
    	uri=${rurl[$i]} ; alias=${ralias[$i]}
    	[ "${uri#http*}" != "$uri" -o "${uri#ftp*}" != "$uri" ] || continue	
    	[ "${enable[$i]}" == "yes" ] || continue
    	rfile=/etc/zypp/repos.d/$alias.repo	 
    	[ -f $rfile ] || continue
    	curl --connect-timeout $TimeOut $uri > /dev/null 2>&1 && { 
    	if [ "${!alias}" ] ; then
    		for mirror in ${!alias} ; do
    			if [ "$uri" != "$mirror" ] ; then
    				curl --connect-timeout $TimeOut $mirror > /dev/null 2>&1 && sed -i "s|$uri|$mirror|" $rfile
    			fi
    			break
    		done
    	fi
    	} || {
    		for mirror in ${!alias} ; do
    			if [ "$uri" != "$mirror" ] ; then
    				curl --connect-timeout $TimeOut $mirror > /dev/null 2>&1 && { 
    					sed -i "s|$uri|$mirror|" $rfile
    					break
    				}
    			fi
    		done
    	} 
    	zypper refresh -r "$alias"	
    done
    Last edited by hcvv; 14-May-2011 at 01:00. Reason: Corrected typo in the title

  2. #2

    Default Re: updaterepos - skipping repos that are down and allowing multiple urls

    Comment pasted from that thread: Packman broken?

    Quote Originally Posted by robin_listas View Post
    What I don't see is how you change the URL for zypper :-?

    I mean, zypper must think it is using packman with the same URL as always,
    but in fact you change it somehow. At least, I think you do, but I didn't
    see how.
    This is how I change the URL:
    Code:
    sed -i "s|$uri|$mirror|" $rfile
    Where $rfile is the file /etc/zypp/repos.d/$alias.repo. I do it by "sediting" the repo file because I wanted to change only the URL and not any other property of the repo, like 'autorefresh', 'enabled' (although the script skips disabled repos earlier) or 'keeppackages'. Finally zypper refreshes the repo with the new URL. For me it's a simple workaround, as I didn't want to worry about the packman US mirror being available or not.

    Quote Originally Posted by robin_listas View Post
    If this works, you have a good idea to propose "them" :-)
    IMHO "they" would just need to replace an URL with an array of URLs.

  3. #3
    Join Date
    Feb 2009
    Location
    Spain
    Posts
    25,547

    Default Re: updaterepos - skipping repos that are down and allowing multipleurls

    On 2011-05-14 16:06, please try again wrote:
    >
    > _Comment_pasted_from_that_thread:_'Packman_broken?'
    > (http://tinyurl.com/5rgy669)_
    >
    > robin_listas;2339581 Wrote:
    >>
    >> What I don't see is how you change the URL for zypper :-?
    >>
    >> I mean, zypper must think it is using packman with the same URL as
    >> always,
    >> but in fact you change it somehow. At least, I think you do, but I
    >> didn't
    >> see how.
    >>

    >
    > This is how I change the URL:
    >
    > Code:
    > --------------------
    > sed -i "s|$uri|$mirror|" $rfile
    > --------------------
    >
    > Where $rfile is the file /etc/zypp/repos.d/$alias.repo. I do it by
    > "sediting" the repo file because I wanted to change only the URL and not
    > any other property of the repo, like 'autorefresh', 'enabled' (although
    > the script skips disabled repos earlier) or 'keeppackages'.


    Yes, that's was the trick I knew you must be doing somehow.

    Mmm... weird. That directory is not named after the 'alias' zypper uses,
    but after the name. When you list the output of "zypper lr --details", each
    repo has both a name and an alias. The alias is used for the directories
    names in "/var/cache/zypp/packages/". But looking at "/etc/zypp/repos.d/"
    it is using instead the names.

    So zypper is not consistent. For some filenames it uses "alias", for other
    it uses "names". Interesting!

    > Finally
    > zypper refreshes the repo with the new URL. For me it's a simple
    > workaround, as I didn't want to worry about the packman US mirror being
    > available or not.


    It is a good solution.


    > robin_listas;2339581 Wrote:
    >>
    >> If this works, you have a good idea to propose "them" :-)
    >>

    >
    > IMHO "they" would just need to replace an URL with an array of URLs.


    They went another route.

    They use a redirector, that when you ask for a package it can give back a
    metalink info file, which includes a list of mirrors for that package. The
    downloader can use that metalink file to download from several mirrors at
    once, and to bypass bad mirrors.

    That's the theory. And packman is not included in the redirector.

    --
    Cheers / Saludos,

    Carlos E. R.
    (from 11.2 x86_64 "Emerald" at Telcontar)

  4. #4

    Default Re: updaterepos - skipping repos that are down and allowing multipleurls

    Quote Originally Posted by robin_listas View Post
    Mmm... weird. That directory is not named after the 'alias' zypper uses,
    but after the name. When you list the output of "zypper lr --details", each
    repo has both a name and an alias. The alias is used for the directories
    names in "/var/cache/zypp/packages/". But looking at "/etc/zypp/repos.d/"
    it is using instead the names.

    So zypper is not consistent. For some filenames it uses "alias", for other
    it uses "names". Interesting!
    The alias seems to be used in both cases for the download directory and repo file names. However if you edit the repo file afterwards as I do, I can not tell since packman's alias and names are the same (at least the way I originally added the repo). Look at 'repo-oss' in the example below.

    Code:
    # zypper lr
    
    #  | Alias                            | Name                             | Enabled | Refresh
    ---+----------------------------------+----------------------------------+---------+--------
     1 | Updates-for-openSUSE-11.4-11.4-0 | Updates for openSUSE 11.4 11.4-0 | Yes     | Yes    
     2 | google-earth                     | google-earth                     | Yes     | Yes    
     3 | nvidia                           | nvidia                           | Yes     | No     
     4 | openSUSE-11.4-11.4-0             | openSUSE-11.4-11.4-0             | No      | No     
     5 | openSUSE:Contrib                 | openSUSE:Contrib                 | Yes     | No     
     6 | openSUSE:Education               | openSUSE:Education               | Yes     | No     
     7 | openSUSE:Games                   | openSUSE:Games                   | Yes     | No     
     8 | openSUSE:Gnome:Community         | openSUSE:Gnome:Community         | Yes     | No     
     9 | openSUSE:Gnome:Contrib           | openSUSE:Gnome:Contrib           | Yes     | No     
    10 | openSUSE:Java                    | openSUSE:Java                    | Yes     | No     
    11 | openSUSE:KDE:Extra               | openSUSE:KDE:Extra               | Yes     | No     
    12 | openSUSE:Network:utilities       | openSUSE:Network:utilities       | Yes     | No     
    13 | openSUSE:PTA                     | openSUSE:PTA                     | Yes     | No     
    14 | openSUSE:Tools                   | openSUSE:Tools                   | Yes     | No     
    15 | openSUSE:Virtualization          | openSUSE:Virtualization          | Yes     | No     
    16 | openoffice                       | openoffice                       | Yes     | No     
    17 | packman                          | packman                          | Yes     | No     
    18 | repo-debug                       | openSUSE-11.4-Debug              | No      | Yes    
    19 | repo-debug-update                | openSUSE-11.4-Update-Debug       | No      | Yes    
    20 | repo-non-oss                     | openSUSE-11.4-Non-Oss            | Yes     | Yes    
    21 | repo-oss                         | openSUSE-11.4-Oss                | Yes     | Yes    
    22 | repo-source                      | openSUSE-11.4-Source             | Yes     | Yes    
    23 | virtualbox                       | virtualbox                       | Yes     | No     
    
    
    
    # find /etc/zypp/repos.d/ -type f
    
    /etc/zypp/repos.d/openSUSE:Contrib.repo
    /etc/zypp/repos.d/virtualbox.repo
    /etc/zypp/repos.d/repo-debug-update.repo
    /etc/zypp/repos.d/openSUSE:Tools.repo
    /etc/zypp/repos.d/openSUSE:Virtualization.repo
    /etc/zypp/repos.d/openSUSE:Gnome:Community.repo
    /etc/zypp/repos.d/repo-source.repo
    /etc/zypp/repos.d/repo-debug.repo
    /etc/zypp/repos.d/openSUSE:Games.repo
    /etc/zypp/repos.d/openSUSE:PTA.repo
    /etc/zypp/repos.d/openSUSE:Gnome:Contrib.repo
    /etc/zypp/repos.d/openSUSE:Java.repo
    /etc/zypp/repos.d/repo-oss.repo
    /etc/zypp/repos.d/openoffice.repo
    /etc/zypp/repos.d/openSUSE:Education.repo
    /etc/zypp/repos.d/Updates-for-openSUSE-11.4-11.4-0.repo
    /etc/zypp/repos.d/google-earth.repo
    /etc/zypp/repos.d/openSUSE:KDE:Extra.repo
    /etc/zypp/repos.d/packman.repo
    /etc/zypp/repos.d/openSUSE-11.4-11.4-0.repo
    /etc/zypp/repos.d/nvidia.repo
    /etc/zypp/repos.d/repo-non-oss.repo
    /etc/zypp/repos.d/openSUSE:Network:utilities.repo
    
    
    # find -H /var/cache/zypp/packages -maxdepth 1 -mindepth 1 -type d 
    
    /var/cache/zypp/packages/PTA
    /var/cache/zypp/packages/i386
    /var/cache/zypp/packages/repo-oss
    /var/cache/zypp/packages/google-earth
    /var/cache/zypp/packages/openSUSE:Gnome:Contrib
    /var/cache/zypp/packages/openSUSE:PTA
    /var/cache/zypp/packages/openSUSE:Virtualization
    /var/cache/zypp/packages/openSUSE:Contrib
    /var/cache/zypp/packages/_tmpRPMcache_
    /var/cache/zypp/packages/openSUSE:Education
    /var/cache/zypp/packages/openSUSE-11.4-11.4-0
    /var/cache/zypp/packages/InstallationImage
    /var/cache/zypp/packages/repo-non-oss
    /var/cache/zypp/packages/openSUSE:Games
    /var/cache/zypp/packages/virtualbox
    /var/cache/zypp/packages/openSUSE:Tools
    /var/cache/zypp/packages/nvidia
    /var/cache/zypp/packages/x86_64
    /var/cache/zypp/packages/Updates-for-openSUSE-11.4-11.4-0
    /var/cache/zypp/packages/packman
    /var/cache/zypp/packages/openSUSE:Network:utilities
    /var/cache/zypp/packages/openSUSE:Gnome:Community
    /var/cache/zypp/packages/openSUSE:KDE:Extra

  5. #5
    Join Date
    Feb 2009
    Location
    Spain
    Posts
    25,547

    Default Re: updaterepos - skipping repos that are down and allowing multipleurls

    On 2011-05-15 13:36, please try again wrote:
    >
    > robin_listas;2339750 Wrote:



    >> So zypper is not consistent. For some filenames it uses "alias", for
    >> other it uses "names". Interesting!
    >>

    >
    > The alias seems to be used in both cases for the download directory and
    > repo file names. However if you edit the repo file afterwards as I do, I
    > can not tell since packman's alias and names are the same (at least the
    > way I originally added the repo). Look at 'repo-oss' in the example
    > below.


    Ah, yes, you don't notice the difference because you have both name and
    alias the same. I alter the alias, so I noticed.

    See my case for three of the repos:

    Code:
    
    > #  | Alias                          | Name                                 | Enabled | Refresh | Priority | Type   | URI                                                                                    | Service
    > ---+--------------------------------+--------------------------------------+---------+---------+----------+--------+----------------------------------------------------------------------------------------+--------
    > 1  | EXT.P_NVidia                   | NVIDIA Repository                    | Yes     | Yes     |   99     | rpm-md | ftp://download.nvidia.com/opensuse/11.2/                                               |
    > 2  | Ext_Packman                    | Packman Repository                   | Yes     | Yes     |  120     | rpm-md | http://ftp.gwdg.de/pub/linux/misc/packman/suse/11.2/                                   |
    > 3  | Ext_Videolan                   | VideoLan Repository                  | No      | No      |  125     | rpm-md | http://download.videolan.org/pub/videolan/vlc/SuSE/11.2/                               |
    
    
    cer@Telcontar:~> find /etc/zypp/repos.d/ -type f | grep -i
    "nvidia\|videolan\|packman"
    /etc/zypp/repos.d/download.videolan.org-SuSE.repo
    /etc/zypp/repos.d/Packman_Repository.repo
    /etc/zypp/repos.d/download.nvidia.com-opensuse.repo
    
    > cer@Telcontar:~> ls -l /var/cache/zypp/packages  | grep -i "nvidia\|videolan\|packman"
    > lrwxrwxrwx 1 root root   60 2011-02-20 01:27 EXT.P_NVidia -> /data/storage_c/repositorios_zypp/11_2/packages/EXT.P_NVidia
    > lrwxrwxrwx 1 root root   41 2010-09-21 21:31 Ext_Packman -> /var/cache/zypp/packages.link/Ext_Packman
    > lrwxrwxrwx 1 root root   42 2010-09-21 21:44 Ext_Videolan -> /var/cache/zypp/packages.link/Ext_Videolan

    The packages dirs in my case are symlinks, because I export them via NFS
    and share the download on two computers. But the names are put by zypper,
    so you can see that it is using "alias" for the package dirs, and "name"
    for the repo metadata you use.


    The case of your repo-oss is curious, though. Maybe it can use either of
    alias or name. Intriguing.

    --
    Cheers / Saludos,

    Carlos E. R.
    (from 11.2 x86_64 "Emerald" at Telcontar)

  6. #6

    Default Re: updaterepos - skipping repos that are down and allowing multipleurls

    This is weird. In my case the packages dirs are not symlinks but the parent directory /var/cache/zypp/packages is a symlink to a directory on my file server. I didn't install any of the openSUSE-11.4xxx (name) or repo-xxx (alias) myself. Those are installed by default. I installed all the other repos from my own installation script, using the syntax:
    Code:
    zypper ar repo_url alias
    meaning I always explicitely specify the alias. The fact that I chose the alias 'packman' while the repo was also named 'packman' is actually a coincidence.

  7. #7
    Join Date
    Feb 2009
    Location
    Spain
    Posts
    25,547

    Default Re: updaterepos - skipping repos that are down and allowing multipleurls

    On 2011-05-15 17:36, please try again wrote:
    >
    > This is weird. In my case the packages dirs are not symlinks but the
    > parent directory /var/cache/zypp/packages is a symlink to a directory on
    > my file server.


    I let zypper create the real directory, then I move it to the nfs mount and
    symlink it. This way it works even if in another install it is named
    differently.

    > I didn't install any of the openSUSE-11.4xxx (name) or
    > repo-xxx (alias) myself. Those are installed by default. I installed all
    > the other repos from my own installation script, using the syntax:
    >
    > Code:
    > --------------------
    > zypper ar repo_url alias
    > --------------------
    >
    > meaning I always explicitely specify the alias.


    I change both name and alias after the creation, which I typically create
    from YaST (with yast you can define the name, not the alias). Only in YaST
    I can see which repos have the keep packages option enabled (and not at a
    glance).

    --
    Cheers / Saludos,

    Carlos E. R.
    (from 11.2 x86_64 "Emerald" at Telcontar)

  8. #8

    Default Re: updaterepos - skipping repos that are down and allowing multipleurls

    Quote Originally Posted by robin_listas View Post
    I let zypper create the real directory,
    Me too!

    Quote Originally Posted by robin_listas View Post
    Only in YaST I can see which repos have the keep packages option enabled (and not at a glance).
    What about:
    Code:
    find /etc/zypp/repos.d -type f -exec  awk -F "=" '/name=|keeppackages=/ { printf "%s =",  $2 } ; END { printf "\n"}' "{}" ";" | awk -F "=" 'BEGIN { printf "%-45s keeppackages\n", "repo" } ; { K=$2 ; if ( $2 == "" ) K="-" ;  printf "%50-s %s\n", $1, K}'
    ?

  9. #9

    Default Re: updaterepos - skipping repos that are down and allowing multipleurls

    packman.unixheads.com now responds to curl but the repo is still unavailable. I had to modify the script, so that it will switch back to the former mirror if zypper refresh fails. Don't worry about the error message while trying to refresh packman. This is normal.
    Code:
    #! /bin/bash
    #
    #: Title       : updaterepos
    #: Date Created: Fri May 13 10:51:42 PDT 2011
    #: Last Edit   : Fri May 13 12:52:04 PDT 2011
    #: Author      : please_try_again 
    #: Version     : 1.0
    #: Description : Refreshes all enabled repos, while skipping repos that are down after a defined timeout
    #                Replaces repos that are down by the next available repo defined for their alias - if any  
    #                Replaces repos that are up   by the first repo defined for their alias if it is back online 
    
    # You can define the timeout
    TimeOut=3
    
    # check if we are root
    [ $UID -eq 0 ] || exec echo "You have to run this script as root or sudo"
    
    # use lsb_release to get openSUSE version
    which lsb_release >/dev/null 2>&1 || zypper -n in lsb-release
    
    suseRel=$(lsb_release -rs)
    
    # You can define multiples urls for each repo's alias, as in the packman example below.
    
    packman="http://packman.unixheads.com/suse/openSUSE_$suseRel/ \
    http://ftp.uni-erlangen.de/pub/mirrors/packman/suse/openSUSE_$suseRel/ \
    http://ftp.gwdg.de/pub/linux/misc/packman/suse/openSUSE_$suseRel/ \
    http://ftp.halifax.rwth-aachen.de/packman/suse/openSUSE_$suseRel/ \
    http://mirror.geht-schon.de/packman.links2linux.de/suse/openSUSE_$suseRel/ \
    http://packman.jacobs-university.de/suse/openSUSE_$suseRel/ \
    http://packman.inode.at/suse/openSUSE_$suseRel/"
    
    eval $(zypper lr -u | tr -d " " | awk -F "|" '/^[1-9]/{ E=tolower($4) ; printf "ralias[%s]=%s;rurl[%s]=%s;enable[%s]=%s;", $1, $2, $1, $6, $1, E}')
    
    i=0
    while [ $i -le ${#ralias[*]} ] ; do
            let i++
            uri=${rurl[$i]} ; alias=${ralias[$i]}
            [ "${uri#http*}" != "$uri" -o "${uri#ftp*}" != "$uri" ] || continue     
            [ "${enable[$i]}" == "yes" ] || continue
            rfile=/etc/zypp/repos.d/$alias.repo      
            [ -f $rfile ] || continue
            curl --connect-timeout $TimeOut $uri > /dev/null 2>&1 && { 
            if [ "${!alias}" ] ; then
                    for mirror in ${!alias} ; do
                            if [ "$uri" != "$mirror" ] ; then
                                    curl --connect-timeout $TimeOut $mirror > /dev/null 2>&1 && sed -i "s|$uri|$mirror|" $rfile
                            fi
                            break
                    done
            fi
            } || {
                    for mirror in ${!alias} ; do
                            if [ "$uri" != "$mirror" ] ; then
                                    curl --connect-timeout $TimeOut $mirror > /dev/null 2>&1 && { 
                                            sed -i "s|$uri|$mirror|" $rfile
                                            break
                                    }
                            fi
                    done
            } 
            zypper -n refresh -r "$alias" || {      
                    for mirror in ${!alias} ; do
                            if [ "$uri" != "$mirror" ] ; then
                                    curl --connect-timeout $TimeOut $mirror > /dev/null 2>&1 && sed -i "s|$uri|$mirror|" $rfile
                                    break
                            fi
                    done
                    zypper -n refresh -r "$alias"
            }
    done

  10. #10
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    32,325
    Blog Entries
    15

    Default Re: updaterepos - skipping repos that are down and allowingmultiple urls

    Quote Originally Posted by please try again
    packman.unixheads.com now responds to curl but the repo is still
    unavailable. I had to modify the script, so that it will switch back to
    the former mirror if zypper refresh fails. Don't worry about the error
    message while trying to refresh packman. This is normal.
    Hi
    You should pull the links on the fly from this, if/when it updates you
    will always be current;
    ftp://packman.links2linux.de/pub/packman/MIRRORS


    --
    Cheers Malcolm °¿° (Linux Counter #276890)
    openSUSE 11.4 (x86_64) Kernel 2.6.37.6-0.5-desktop
    up 1 day 2:16, 4 users, load average: 0.09, 0.18, 0.16
    GPU GeForce 8600 GTS Silent - Driver Version: 270.41.06


Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •