Squid Proxy running inside DD with HD usb like a big cache

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware
Goto page 1, 2, 3  Next
Author Message
stalonge
DD-WRT Guru


Joined: 21 Jul 2006
Posts: 1870
Location: Fortaleza Ce Brazil

PostPosted: Tue Oct 12, 2010 21:59    Post subject: Squid Proxy running inside DD with HD usb like a big cache Reply with quote
It is possible to do this ?

If yes , someone has a how to ?

_________________
DDwrt ...it rocks ....

1 R7800 46069 AP Wireguard webserver JFFS SAMBA FTP usb HD Mesh
1 R7800 46069 Cli Mesh
1 WZR1750 45690 AP Webserver OpenVPN Wireguard
1 DDx86 Vmware 40260
1 E3000 41027 K3 @500 WAP wifi client
Sponsor
icewaere
DD-WRT User


Joined: 09 Sep 2010
Posts: 86

PostPosted: Wed Oct 13, 2010 0:10    Post subject: Reply with quote
Olla,

yes it works absolutly great ;)



install per optware

ipkg-opt install squid

this is 2.6xxx Version. little bit old but works fine
"i have tested to install the 2.7 from openwrt but dont works ;( dunno why."

after install edit this: /opt/etc/squid/squid.conf

insert this:

Code:
cache_effective_user nobody
cache_mgr bigbrother@unslung_squid 
visible_hostname ddwrt

#cache_replacement_policy heap LFUDA  duno why dont work with 2.6
#memory_replacement_policy LFUDA      same

ipcache_size 2048
cache_swap_low 90
cache_swap_high 95
maximum_object_size_in_memory 100 KB

#If you have 64/8MB Router you can use 16MB cache_mem if smaler router use 8MB
cache_mem 16 MB

#this is cachedir you can change if you want. 400 meams 400MB cache. size you can change what you want
cache_dir ufs /opt/var/squid/cache 400 16 256

logfile_rotate 10
fqdncache_size 2048
memory_pools off
maximum_object_size 16384 KB
quick_abort_min 0 KB
quick_abort_max 0 KB
log_icp_queries off
client_db off
buffered_logs on
half_closed_clients off
negative_dns_ttl 10 second           
connect_timeout 60 second             
read_timeout 80 second
request_timeout 80 second

#Logs you can change dir what you want
cache_access_log /opt/var/squid/logs/access.log
cache_log /opt/var/squid/logs/debug
cache_store_log /opt/var/squid/logs/storage

hierarchy_stoplist on

#Proxy Ip same as ddwrt RuterIp. change how you want
http_port 192.168.1.1:3128 transparent

# Global ACL-Definitions (Access control lists)
acl idents ident REQUIRED
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl all src 0.0.0.0/0.0.0.0
acl intern dst 192.168.1.0/255.255.255.0
acl FTP proto FTP
always_direct allow FTP

#Allowed working outgoing Ports and LanIps
acl Allowed_Ports port 80 99 443 21 563 488 777 210 1025-65535
acl yourLAN src 192.168.1.0/255.255.255.0 

#http_access definition
http_access allow idents
http_access allow all
http_access allow intern
http_access deny manager all
http_access allow yourLAN
http_access deny all
icp_access deny all
miss_access allow all
always_direct allow intern


now you must add a "nobody" user. root not allowed

Code:
grep -q ^nobody /etc/group  || echo "nobody:x:99:" >> /etc/group
grep -q ^nobody /etc/passwd || echo "nobody:*:99:99:nobody:/var:/bin/false" >> /etc/passwd



next to do is create cachedir and swapadding:

Code:
squid -z


Now we can check if squid works ;)

start squid with that:

Code:
/opt/etc/init.d/S80squid start


now you can check if it runs with "ps" or "top"


Thats all, be happy with great CacheProxy Wink Great internetspeedup for user with 3G/Umts Connections!


Here Autostartscript:

Add this in WebGui of ddwrt to Admin->Diag->StartupCommand
Code:
grep -q ^nobody /etc/group  || echo "nobody:x:99:" >> /etc/group
grep -q ^nobody /etc/passwd || echo "nobody:*:99:99:nobody:/var:/bin/false" >> /etc/passwd
/opt/etc/init.d/S80squid start



And if you want to use Proxy as Transparent or Reverse Proxy you must add things to Firewall/Iptables read this:
http://www.dd-wrt.com/wiki/index.php/Transparent_web_proxy

greetz@all Ice


plz if anyone knows how i can install newer version of Squid plz let me know.

_________________
Netgear Nighthawk R7000@1400mhz / KongAC-ARM-build (25530M)
Netgear WNDR4500 DD-WRT v24-sp2 (09/27/13) kongac-build 23430 / OTRW t2 Squid 3.1.8 / 64gb SSD
Belkin Play Max F7D4301v1@533mhz / 500GB USB / 3G Huawei K3765-HV / DD-WRT 18777
Linksys E3000 with Tomato Victek
R.I.P. WRT160NL
R.I.P. WRT54GL
crashfly
DD-WRT Guru


Joined: 24 Feb 2009
Posts: 2026
Location: Sol System > Earth > USA > Arkansas

PostPosted: Wed Oct 13, 2010 0:20    Post subject: Reply with quote
Thanks for the information on this, icewaere.

I will likely implement this soon on my router.

_________________
E3000 22200M KongVPN K26
WRT600n v1.1 refirb mega 18767 BS K24 NEWD2 [not used]
WRT54G v2 16214 BS K24 [access point]

Try Dropbox for syncing files - get 2.5gb online for free by signing up.

Read! Peacock thread
*PLEASE* upgrade PAST v24SP1 or no support.
icewaere
DD-WRT User


Joined: 09 Sep 2010
Posts: 86

PostPosted: Wed Oct 13, 2010 1:34    Post subject: Reply with quote
np,

if works fine for many users i can do a wiki for that.

greetz

_________________
Netgear Nighthawk R7000@1400mhz / KongAC-ARM-build (25530M)
Netgear WNDR4500 DD-WRT v24-sp2 (09/27/13) kongac-build 23430 / OTRW t2 Squid 3.1.8 / 64gb SSD
Belkin Play Max F7D4301v1@533mhz / 500GB USB / 3G Huawei K3765-HV / DD-WRT 18777
Linksys E3000 with Tomato Victek
R.I.P. WRT160NL
R.I.P. WRT54GL
Monode
DD-WRT Novice


Joined: 02 Nov 2010
Posts: 8

PostPosted: Wed Nov 10, 2010 5:20    Post subject: Reply with quote
Thanks! I was looking for exactly this. Smile
AirFart
DD-WRT User


Joined: 06 Mar 2008
Posts: 92
Location: South East Asia

PostPosted: Fri Dec 17, 2010 9:25    Post subject: Reply with quote
i follow your guide to set up on my rt-n16

the problem i got is when i type squid -z

i got this message

Code:

2010/12/16 22:57:44| Creating Swap Directories
FATAL: Failed to make swap directory /opt/var/squid/cache/00: (13) Permission denied
Squid Cache (Version 2.6.STABLE23): Terminated abnormally.
CPU Usage: 0.000 seconds = 0.000 user + 0.000 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 0

_________________
1xWAP54G : Bricked
1xWHR-G54S : Repeater @ 251mW (2.2km link)DD-WRT Installation Guide
1xRT-N12 : Repeater Bridge (15962)
1xRT-N16 : Router + Optware (TomatoUSB DualWAN)
1xEOC-2611P : Bricked (RMA)
1xWHR-HP-GN : WDS AP @ 23dBm (15962)
2xWHR-HP-G300N : WDS Station @ 21-28dBm (15962)
1xTL-WA5110G : AP @ 26dBm (Stock)
|Free WiFi for all|
icewaere
DD-WRT User


Joined: 09 Sep 2010
Posts: 86

PostPosted: Fri Dec 17, 2010 11:33    Post subject: Reply with quote
hey,

you use ext3 ?

checkup if you set write access to that directionarys?

you have added nobody user?

greetz

_________________
Netgear Nighthawk R7000@1400mhz / KongAC-ARM-build (25530M)
Netgear WNDR4500 DD-WRT v24-sp2 (09/27/13) kongac-build 23430 / OTRW t2 Squid 3.1.8 / 64gb SSD
Belkin Play Max F7D4301v1@533mhz / 500GB USB / 3G Huawei K3765-HV / DD-WRT 18777
Linksys E3000 with Tomato Victek
R.I.P. WRT160NL
R.I.P. WRT54GL
stalonge
DD-WRT Guru


Joined: 21 Jul 2006
Posts: 1870
Location: Fortaleza Ce Brazil

PostPosted: Fri Dec 17, 2010 12:15    Post subject: Reply with quote
icewaere wrote:
np,

if works fine for many users i can do a wiki for that.

greetz



Ok , i have a wrt 610 N v2 i want to see the wiki to do this .. Now i have a squid proxy running in a linux box .

Thanks in advance

Very Happy Very Happy

_________________
DDwrt ...it rocks ....

1 R7800 46069 AP Wireguard webserver JFFS SAMBA FTP usb HD Mesh
1 R7800 46069 Cli Mesh
1 WZR1750 45690 AP Webserver OpenVPN Wireguard
1 DDx86 Vmware 40260
1 E3000 41027 K3 @500 WAP wifi client
AirFart
DD-WRT User


Joined: 06 Mar 2008
Posts: 92
Location: South East Asia

PostPosted: Fri Dec 17, 2010 13:15    Post subject: Reply with quote
icewaere wrote:
hey,

you use ext3 ?

checkup if you set write access to that directionarys?

you have added nobody user?

greetz


yes ext3 format. I already manage to run transmission on the optware. i dont know why it denied access... i didn't set any write access to that directories. Can you write complete wiki guide ... maybe i missed other step not mention here

thanks Smile

_________________
1xWAP54G : Bricked
1xWHR-G54S : Repeater @ 251mW (2.2km link)DD-WRT Installation Guide
1xRT-N12 : Repeater Bridge (15962)
1xRT-N16 : Router + Optware (TomatoUSB DualWAN)
1xEOC-2611P : Bricked (RMA)
1xWHR-HP-GN : WDS AP @ 23dBm (15962)
2xWHR-HP-G300N : WDS Station @ 21-28dBm (15962)
1xTL-WA5110G : AP @ 26dBm (Stock)
|Free WiFi for all|
icewaere
DD-WRT User


Joined: 09 Sep 2010
Posts: 86

PostPosted: Fri Dec 17, 2010 14:03    Post subject: Reply with quote
hmm,

ok first take a look with telnet/ssh or scp console if /opt/var/squid/cache/
and
/opt/var/squid/logs/
exist.

if not creat it manualy

mkdir /opt/var/squid/
mkdir /opt/var/squid/cache/
mkdir /opt/var/squid/logs/

then set rights:

chmod 777 /opt/var/squid/

after that check squid -z again

greetz

_________________
Netgear Nighthawk R7000@1400mhz / KongAC-ARM-build (25530M)
Netgear WNDR4500 DD-WRT v24-sp2 (09/27/13) kongac-build 23430 / OTRW t2 Squid 3.1.8 / 64gb SSD
Belkin Play Max F7D4301v1@533mhz / 500GB USB / 3G Huawei K3765-HV / DD-WRT 18777
Linksys E3000 with Tomato Victek
R.I.P. WRT160NL
R.I.P. WRT54GL
frater
DD-WRT Guru


Joined: 07 Jun 2006
Posts: 2777

PostPosted: Fri Dec 17, 2010 18:39    Post subject: Reply with quote
If I have time I can add it to OTRW with some appropriate startup scripts and maybe even an option to make it transparent for all IP's.

This way you don't have to make firewall rules rin rc_firewall that become invalid when you turn off the proxy.

Let me know how many testers I will get.

I'm currently busy with Zabbix and DD-WRT. It's amazing what you can do with it...

_________________
Asus RT16N + OTRW
Kingston 4GB USB-disk 128 MB swap + 1.4GB ext3 on /opt + 2 GB ext3 on /mnt
Copperjet 1616 modem in ZipB-config
Asterisk, pixelserv & Pound running on router
Another Asus RT16N as WDS-bridge

DD-WRT v24-sp2 vpn (c) 2010 NewMedia-NET GmbH
Release: 12/16/10 (SVN revision: 15758M)
icewaere
DD-WRT User


Joined: 09 Sep 2010
Posts: 86

PostPosted: Fri Dec 17, 2010 21:02    Post subject: Reply with quote
your welcome frater ;)

another thing you can plzplzplz double plz add pyload to optware?

and if possible newer version of squid.

greetz and thx

_________________
Netgear Nighthawk R7000@1400mhz / KongAC-ARM-build (25530M)
Netgear WNDR4500 DD-WRT v24-sp2 (09/27/13) kongac-build 23430 / OTRW t2 Squid 3.1.8 / 64gb SSD
Belkin Play Max F7D4301v1@533mhz / 500GB USB / 3G Huawei K3765-HV / DD-WRT 18777
Linksys E3000 with Tomato Victek
R.I.P. WRT160NL
R.I.P. WRT54GL
frater
DD-WRT Guru


Joined: 07 Jun 2006
Posts: 2777

PostPosted: Fri Dec 17, 2010 21:53    Post subject: Reply with quote
icewaere wrote:
your welcome frater ;)

another thing you can plzplzplz double plz add pyload to optware?

and if possible newer version of squid.

greetz and thx

I'm not responsible for optware itself nor do I want people to think this.

_________________
Asus RT16N + OTRW
Kingston 4GB USB-disk 128 MB swap + 1.4GB ext3 on /opt + 2 GB ext3 on /mnt
Copperjet 1616 modem in ZipB-config
Asterisk, pixelserv & Pound running on router
Another Asus RT16N as WDS-bridge

DD-WRT v24-sp2 vpn (c) 2010 NewMedia-NET GmbH
Release: 12/16/10 (SVN revision: 15758M)
AirFart
DD-WRT User


Joined: 06 Mar 2008
Posts: 92
Location: South East Asia

PostPosted: Wed Dec 22, 2010 5:28    Post subject: Reply with quote
icewaere wrote:
hmm,

ok first take a look with telnet/ssh or scp console if /opt/var/squid/cache/
and
/opt/var/squid/logs/
exist.

if not creat it manualy

mkdir /opt/var/squid/
mkdir /opt/var/squid/cache/
mkdir /opt/var/squid/logs/

then set rights:

chmod 777 /opt/var/squid/

after that check squid -z again

greetz


ok thanks.
i did as you told. Now when i type squid -z it will create the swap directory. :)

But...

when i try to use the squid proxy. Proxy is not working.

Can you help me to check my squid.conf file

or can you give me your working squid.conf file.



squid.tar.gz
 Description:

Download
 Filename:  squid.tar.gz
 Filesize:  93.78 KB
 Downloaded:  1957 Time(s)


_________________
1xWAP54G : Bricked
1xWHR-G54S : Repeater @ 251mW (2.2km link)DD-WRT Installation Guide
1xRT-N12 : Repeater Bridge (15962)
1xRT-N16 : Router + Optware (TomatoUSB DualWAN)
1xEOC-2611P : Bricked (RMA)
1xWHR-HP-GN : WDS AP @ 23dBm (15962)
2xWHR-HP-G300N : WDS Station @ 21-28dBm (15962)
1xTL-WA5110G : AP @ 26dBm (Stock)
|Free WiFi for all|
icewaere
DD-WRT User


Joined: 09 Sep 2010
Posts: 86

PostPosted: Wed Dec 22, 2010 15:37    Post subject: Reply with quote
ok here is my working conf:

only thing you must change is that:

cache_dir ufs /opt/var/squid/cache 10000 24 256

10000 = 10gb cache
1000 = 1gb cache

change what you need Wink
greetz



squid.rar
 Description:

Download
 Filename:  squid.rar
 Filesize:  803 Bytes
 Downloaded:  1727 Time(s)


_________________
Netgear Nighthawk R7000@1400mhz / KongAC-ARM-build (25530M)
Netgear WNDR4500 DD-WRT v24-sp2 (09/27/13) kongac-build 23430 / OTRW t2 Squid 3.1.8 / 64gb SSD
Belkin Play Max F7D4301v1@533mhz / 500GB USB / 3G Huawei K3765-HV / DD-WRT 18777
Linksys E3000 with Tomato Victek
R.I.P. WRT160NL
R.I.P. WRT54GL
Goto page 1, 2, 3  Next Display posts from previous:    Page 1 of 3
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum