Results 1 to 3 of 3

Thread: Save bandwidth with squid, adzap and intelligentmirror!

  1. #1

    Smile Save bandwidth with squid, adzap and intelligentmirror!

    I have mentioned in the past that I get my internet from my cell phone...this results in faster than dialup speeds but slower than a good dsl connection....roughly 25k per second downloads on an edge connection with T-Mobile. 3G is not yet available in my area for a faster connection. Over the years I have played with various cacheing proxies in the hopes of speeding up my surfing...wwwoffle, privoxy (for ad filtering), polipo....etc.....recently I decided to create one master cache for the entire house and setup transparent proxying...getting the traffic to my pc was easy by setting my wifi router to forward all port 80 traffic to my main pc. I've always done adfiltering to not download any more than I have to and I wanted to do those same things. I did a little bit of playing around with both squid 2.7 and squid 3.0...the configuration files are different between the two versions so you can't share the config files. I decided to use "adzap" as the squid plugin to do ad blocking....and as time went by I noticed if I was downloading rpm's from "download.opensuse.org" and it directed me to mirror A and my pc in front office got sent to mirror B it tried to download the files all over again...this resulted in alot of wasted space in the cache. After some googling I came across a product used by fedora users called "intelligentmirrors"...it worked as a plugin for squid and basically any time it encountered an rpm it would store it under a new url pointing to the local machine....this allowed you to build a cache and if you had to update other pc's at same house they'd get fed all the files from the cache. It was exactly what I wanted but the latest most efficient version only runs on squid 2.7...not the 3.0 I had upgraded to....so once again I downgraded...got transparent proxying working...added in adzap so the ads would get blocked and played with intelligentmirror. It promptly killed squid. intelligentmirror was written by and designed for fedora users...it uses calls to their package manager called "YUM"....after some searching on rpmfind.net I was able to locate a recent copy of yum for my opensuse x86_64 system and get it installed...and it appears to be working fine. I also made some tweaks to my squid to ensure video's and pics etc would get cached as long as possible while still getting current data. I'll past some of the pertinent parts of my squid.conf to show what I am talking about. A dnscache such as dnsmasq with the cache entries set to 1000+ entries should help alot too especially if you have it using the opendns servers instead of your isp's.

    The adzap plugin came from: Ad Zapping With Squid

    The intelligentmirror plugin came from: https://fedorahosted.org/intelligent...elligentMirror I installed the noarch rpm and then added the entries to config file after installing YUM which isn't listed as a dependency but is necessary to provide needed python modules.


    I changed the refresh patterns in squid to keep the graphics/files longer...the refresh patterns for that are:

    Code:
    refresh_pattern ^ftp: 1440 20% 10080
    refresh_pattern ^gopher: 1440 0% 1440
    refresh_pattern -i \.(gif|png|jpg|jpeg|ico)$ 10080 90% 43200 override-expire ignore-no-cache ignore-private
    refresh_pattern -i \.(iso|avi|wav|mp3|mp4|mpeg|swf|flv|x-flv)$ 43200 90% 432000 override-expire ignore-no-cache ignore-priv$
    refresh_pattern -i \.(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|tiff)$ 10080 90% 43200 override-expire ignore-no-cache ig$
    refresh_pattern -i \.index.(html|htm)$ 0 40% 10080
    refresh_pattern -i \.(html|htm|css|js)$ 1440 40% 40320
    refresh_pattern . 0 40% 40320 reload-into-ims
    You can add "ignore-no-store" to the lines that have "ignore-no-cache" if running squid 3.0 or higher. 2.7 doesn't support that directive. The patterns used came from this article on linux.com:
    Linux.com :: Speed up your Internet access using Squid's refresh patterns

    The line to call wrapzap (which calls adzap) and the intelligent mirror ones are at the bottom of my config.

    Code:
    # windows_ipaddrchangemonitor on
    
    redirect_program /usr/local/adzap/scripts/wrapzap
    
    # --BEGIN-- squid config for intelligentmirror #
           storeurl_rewrite_program /usr/bin/python /etc/squid/intelligentmirror/intelligentmirror.py
           storeurl_rewrite_children 15
           acl store_rewrite_list urlpath_regex -i .rpm$
    #       acl store_rewrite_list urlpath_regex -i .deb$
           storeurl_access allow store_rewrite_list
           storeurl_access deny all
    # --END-- squid config for intelligentmirror #
    The program supports .deb's....spose it could easily support other formats but I commented the .deb out since I only needed rpm's....when the intelligent mirror intercepts a rpm download it's log file shows:

    Code:
    2009-10-17 00:13:57,283 INFO NEW_URL RPM gtk2-metatheme-sonar-11.2.5-1.1.noarch.rpm http://192.168.2.2/rpm/gtk2-metatheme-sonar-11.2.5-1.1.noarch.rpm                                                                                                   
    2009-10-17 00:14:05,355 INFO NEW_URL RPM gftp-lang-2.0.19-1.20.noarch.rpm http://192.168.2.2/rpm/gftp-lang-2.0.19-1.20.noarch.rpm                                                                                                                       
    2009-10-17 00:14:09,394 INFO NEW_URL RPM gftp-lang-2.0.19-1.20.noarch.rpm http://192.168.2.2/rpm/gftp-lang-2.0.19-1.20.noarch.rpm                                                                                                                       
    2009-10-17 00:14:09,395 INFO NEW_URL RPM gftp-lang-2.0.19-1.20.noarch.rpm http://192.168.2.2/rpm/gftp-lang-2.0.19-1.20.noarch.rpm                                                                                                                       
    2009-10-17 00:15:23,762 INFO NEW_URL RPM command-not-found-0.1.0-33.1.noarch.rpm http://192.168.2.2/rpm/command-not-found-0.1.0-33.1.noarch.rpm                                                                                                         
    2009-10-17 00:15:27,943 INFO NEW_URL RPM command-not-found-0.1.0-33.1.noarch.rpm http://192.168.2.2/rpm/command-not-found-0.1.0-33.1.noarch.rpm                                                                                                         
    2009-10-17 00:15:35,496 INFO NEW_URL RPM zenity-2.28.0-1.2.i586.rpm http://192.168.2.2/rpm/zenity-2.28.0-1.2.i586.rpm       
    2009-10-17 00:15:38,573 INFO NEW_URL RPM zenity-2.28.0-1.2.i586.rpm http://192.168.2.2/rpm/zenity-2.28.0-1.2.i586.rpm
    You can see where it's rewriting them to show my local 192.168.2.2 pc. This makes keeping multiple pc's synced up alot easier. The actual version of Yum I installed is ftp://ftp5.gwdg.de/pub/opensuse/repo...2.2.x86_64.rpm

    I'll try to attach my squid.conf..it's not tiny and I'm gonna leave the comments in there so you know what you are changing.....the transparent proxying was relatively easy.....I added the word "transparent" after the 3128 port and setup to iptables rules on router/pc:

    iptables -t nat -A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3128
    iptables -t nat -A PREROUTING -i eth0 -p udp -m udp --dport 80 -j REDIRECT --to-ports 3128


    I also added some lines to my dd-wrt router which acts as my local dns server to FORCE all pc's on the network to use the router for dns....and it in return uses opendns....this helps alot...those lines are:

    iptables -t nat -A PREROUTING -p udp -i br0 --dport 53 -j DNAT --to $(nvram get lan_ipaddr)
    iptables -t nat -A PREROUTING -p tcp -i br0 --dport 53 -j DNAT --to $(nvram get lan_ipaddr)

    You could easily change the nvram command to just be 192.168.2.1 which is my routers ip address.


    I will try to help if someone has questions but I may refer you to the same tutorials I read on google such as how to setup the transparent proxy....etc. Hope this helps someone and I'd love input/comments.


    My actual squid.conf is available at http://famewolf.tripod.com/squid.conf because it was to large to attach to the msg.
    Fa/\/\e\/\/olf

    "There is no music here. There is no laughter. There is no joy here, in my here after." -- FW

  2. #2

    Default Re: Save bandwidth with squid, adzap and squirm!

    Regetfully I have to make an update to this post because intelligentmirror did not work out like I had hoped. It was getting an io error when trying to write the file so it always considered the rpm new.

    I installed squirm which is a squid redirector and setup rules that made it always use the same mirror for downloading rpm's when using download.opensuse.org....you could do the same by changing each pc to use a specific mirror but it was easier to place it in one place since they all go through the transparent proxy.
    Fa/\/\e\/\/olf

    "There is no music here. There is no laughter. There is no joy here, in my here after." -- FW

  3. #3
    Join Date
    Sep 2008
    Location
    Toronto,Canada
    Posts
    549

    Default Re: Save bandwidth with squid, adzap and intelligentmirror!

    All I can say is that I think you should be a very good network architect/designer? if you are not already one. Because you didn't have the facilities some of us have you found yourself a way to improve the way linux, in general works. There are always people which will improve and make things better and I think you are one of them. I hope other guys which are in your situation will see the post to get the benefits of using it. Congratulation!

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •