Packages cache for local network using Squid proxy server
FedoraForum.org - Fedora Support Forums and Community
Results 1 to 4 of 4
  1. #1
    Join Date
    Oct 2011
    Posts
    1,909
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Packages cache for local network using Squid proxy server

    If it is wanted to save internet bandwidth it is possible to setup local mirror to share packages among all computers on network so that download happens only once. Standard mirroring downloads whole repositories which may be disk and network consuming. In addition to that, not all packages will get used by computers on the network so it is synchronizing stuff that we are never going to use. This is usually overkill for home or small office use.

    By using Squid on the other hand, it is possible to cache only packages we are using, so next time some computer on the network requires it, it will get it from cache. But there is a catch, this does not work as is; Since packages managers uses mirrors, links to packages also change based on mirror used so default Squid setup is not seeing a package as same from different mirrors. So we need some way to tell squid to store packages files by package name, packages rarely or never change and we don't care from what url it came from as it is most likely the same file. Squid 3.4 implemented feature Store ID that allows to map files to custom ID. We can decide if we want to pass url to a helper program based on rule. Helper program further decides what to do with link and responds to Squid with a storage ID for that file. The most simple way we can do what we want is to trim url to package name in helper program and return that to Squid as storage ID.


    Squid setup:

    If using Fedora for a server just install Squid from default repositories.

    Code:
    $ sudo dnf install squid
    CentOS 7 provides older Squid 3.3 version that doesn't support Store ID feature so third party repo is required. Squid wiki lists some repositories that can be used. CentOS 7.3 got Squid 3.5 that has the required functionality.

    Install store_id_program.py from here, to /usr/local/bin/ and make it executable. Modify Squid configuration /etc/squid/squid.conf based on setting from link. 10000 (10GB) represents cache size to use for storing files and may be adjusted to appropriate size you have available, but it is recommended to not use more then 70% of space available.

    Code:
    #                 3 month    12 month
    refresh_pattern . 129600 33% 525600
    Code:
    cache_dir ufs /var/spool/squid 10000 16 256
    
    store_id_program /usr/local/bin/store_id_program.py
    store_id_children 5 startup=1
    
    # have not seen a larger RPM yet
    maximum_object_size 1 GB
    
    # cache RPMs only
    acl rpm_only urlpath_regex \.rpm
    cache allow rpm_only
    cache deny all
    After that we need to start/restart squid and enable service:
    Code:
    $ sudo systemctl start squid
    $ sudo systemctl enable squid
    Network setup:

    The network setup is pretty simple, all we need to do is to tell each machine on a network to use our proxy/cache server. We do that by configuring dnf proxy= line. Server running squid must have its TCP port open to be able to accept such requests. Default proxy port for squid is 3128.

    Configure dnf for each machine on the network with our proxy: Add a line to /etc/dnf/dnf.conf:
    Code:
    proxy=http://<server ip>:3128
    Where <server ip> is ip address of server running Squid cache.

    Open squid's port 3128 on server:
    Code:
    $ sudo firewall-cmd --permanent --add-port=3128/tcp
    $ sudo firewall-cmd --reload


    For troubleshooting purposes or to see if our cache is working as expected we can monitor Squid's log file:
    Code:
    $ sudo tail -f /var/log/squid/access.log


    Client(s) setup:

    Lately more and more mirrors are providing both http and https connection and dnf seems to prefer to https. This is an issue for squid since it can't read encrypted connection thus it just "tunnels" the connection and package gets downloaded again. The solution for that is to add "&protocol=http" to all the clients metalink URL.
    Last edited by srakitnican; 25th February 2018 at 08:34 AM.

  2. #2
    Join Date
    Jul 2006
    Posts
    10
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Packages cache for local network using Squid proxy server

    Changing the regex to:
    acl rpm_only urlpath_regex \.[d]?rpm

    Should also match delta RPMs...

    Thanks,
    Richard

  3. #3
    Join Date
    Oct 2011
    Posts
    1,909
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Packages cache for local network using Squid proxy server

    Hi,

    Just changing squid configuration is not enough, store_id_program needs that modification as well, right?

    I didn't want to bother with drpms since they are small already and can not be applied in all cases, if packages went out of date too much for example.

  4. #4
    Join Date
    Oct 2007
    Posts
    414
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Packages cache for local network using Squid proxy server

    For me the various mirrors caused some issues, so I did a quick hack..
    My solution is here: http://kalfaoglu.com/wp/how-to-creat...h-squid-proxy/

Similar Threads

  1. Squid 3.1.1 proxy server will not FTP
    By AlphaTwin in forum Servers & Networking
    Replies: 4
    Last Post: 22nd September 2011, 07:49 AM
  2. yum proxy/cache server
    By dangets in forum Installation, Upgrades and Live Media
    Replies: 1
    Last Post: 17th March 2011, 01:04 AM
  3. Squid proxy server
    By biggsk in forum Servers & Networking
    Replies: 2
    Last Post: 21st December 2007, 04:45 PM
  4. Squid Proxy Server
    By 1slorunner in forum Servers & Networking
    Replies: 1
    Last Post: 23rd December 2005, 08:48 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •