ad-blocking with DD-WRT revisited (simple)

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware
Goto page Previous  1, 2, 3, ... 12, 13, 14  Next
Author Message
frater
DD-WRT Guru


Joined: 07 Jun 2006
Posts: 2777

PostPosted: Wed Feb 25, 2009 7:36    Post subject: Reply with quote
You should not write to /etc/hosts
It already contains data and it's write protected.

What are you trying to gain.
My code is well thought of and it became this way because I saw the problems with the code from the other thread.
Every bit of code is done with a reason.

If it's not working it's because of one of the reasons I already explained.

BTW. I can't delete your posts...

_________________
Asus RT16N + OTRW
Kingston 4GB USB-disk 128 MB swap + 1.4GB ext3 on /opt + 2 GB ext3 on /mnt
Copperjet 1616 modem in ZipB-config
Asterisk, pixelserv & Pound running on router
Another Asus RT16N as WDS-bridge

DD-WRT v24-sp2 vpn (c) 2010 NewMedia-NET GmbH
Release: 12/16/10 (SVN revision: 15758M)
Sponsor
phuque99
DD-WRT User


Joined: 20 Oct 2008
Posts: 446

PostPosted: Wed Feb 25, 2009 7:37    Post subject: Reply with quote
frater's script explicitly use /tmp/dlhosts for efficiency reason. What's the fuss about /tmp/hosts I wonder.
mac913
DD-WRT Guru


Joined: 02 May 2008
Posts: 1230
Location: Canada

PostPosted: Wed Feb 25, 2009 9:11    Post subject: Reply with quote
MKruer wrote:
frater, I can confirm that your script does work on my LinkSys WRT54GSv4


I just installed the code and settings that frater created and it works with my Buffalo WHR-HP-G54. Thanks frater!!! Very Happy

For others installing this the 1st time, I ran into problems that were my own. In the router Basic Setup I setup the Gateway and DNS Server with the same IP address as the router (before setup it was 0.0.0.0) and also selected "Use DNSMasq for DHCP" & "Use DNSMasq for DNS". Another issue was I had my computer setup with static IP addresses, Subnet and DNS servers IPs from my cable company, I setup my NIC for DHCP. After I finished configuring my router and rebooted it, I aslo disconnected the PC's network cable from the Network Switch waited for the router to be up and running than plugged-in the network cable to be assigned an IP Address and DNS IP Address (DNS IP same as router IP). Did the ipconfig /all and also nslookup showning DNS Server as the router IP. Also tested "nslookup a.abnad.net" and it came back with Address: 0.0.0.0! Working like a charm! Very Happy

_________________
Home Network on Telus PureFibre - Wired GigaBit Backbone
| SSH | DDNS | DNSCrypt-Proxy | DHCP/Static IPs | Ad-Blocking |
| USB Storage - Scripts Only | VLANs/BRs | WiFi - APs/VAPs |
| OpenVPN-Client | YAMon3 | Telus-IPv6 | ebtables | ip6tables |

Multi 2x R7000 Setup - KONGAC 35550M Release 2018-03-28 (k4.4.124#548SMP)

______________________
YAMon 3.0 | Ad-Blocking for DD-WRT
DDWRT 60-day Timeline/changelogs

Kong Builds | DD-WRT Beta Builds
gaganwalia
DD-WRT User


Joined: 24 Dec 2007
Posts: 449

PostPosted: Wed Feb 25, 2009 13:26    Post subject: Reply with quote
hos is it possible to join 2 files into one file without duplicates. is there any command in linux that can be used on the router.

i have 2 diff. hos files and would like to combine them, but there are lots of duplicates in there too.
frater
DD-WRT Guru


Joined: 07 Jun 2006
Posts: 2777

PostPosted: Wed Feb 25, 2009 13:52    Post subject: Reply with quote
dnsmasq uses /etc/hosts
If you use that command (addn-hosts=/tmp/dlhosts) it will use that file too.

If you have an USB-stick you could use that space for your dlhosts.

You also need to issue the command killall -HUP dnsmasq this will force dnsmasq to reread its config, but this is covered in the script.

Adding /jffs/hosts or /opt/etc/hosts to /tmp/dlhosts is also covered in my script. You can put your own hostnames there.

You should not alter /etc/hosts
/etc/hosts is just a symbolic link to /tmp/hosts and this file is filled with entries from DHCP-clients. This is done because the directory /etc is read-only.
Leave them alone as they are special entries!

I, for one, am using the data in that file to automatically configure my reverse-proxy!

_________________
Asus RT16N + OTRW
Kingston 4GB USB-disk 128 MB swap + 1.4GB ext3 on /opt + 2 GB ext3 on /mnt
Copperjet 1616 modem in ZipB-config
Asterisk, pixelserv & Pound running on router
Another Asus RT16N as WDS-bridge

DD-WRT v24-sp2 vpn (c) 2010 NewMedia-NET GmbH
Release: 12/16/10 (SVN revision: 15758M)


Last edited by frater on Wed Feb 25, 2009 13:56; edited 1 time in total
phuque99
DD-WRT User


Joined: 20 Oct 2008
Posts: 446

PostPosted: Wed Feb 25, 2009 13:52    Post subject: Reply with quote
gaganwalia wrote:
hos is it possible to join 2 files into one file without duplicates. is there any command in linux that can be used on the router.

i have 2 diff. hos files and would like to combine them, but there are lots of duplicates in there too.


uniq is the tool. However, if you're changing frater's script or making up your own stuff, a different thread would be better than confusing other readers who might be interested in using frater's script that is working perfectly.
frater
DD-WRT Guru


Joined: 07 Jun 2006
Posts: 2777

PostPosted: Wed Feb 25, 2009 15:17    Post subject: Reply with quote
mmm...
I didn't understand your question.
this should do it.
Code:
cat file1 + file2 | sort -u >file3

_________________
Asus RT16N + OTRW
Kingston 4GB USB-disk 128 MB swap + 1.4GB ext3 on /opt + 2 GB ext3 on /mnt
Copperjet 1616 modem in ZipB-config
Asterisk, pixelserv & Pound running on router
Another Asus RT16N as WDS-bridge

DD-WRT v24-sp2 vpn (c) 2010 NewMedia-NET GmbH
Release: 12/16/10 (SVN revision: 15758M)
kenja00
DD-WRT Novice


Joined: 04 Aug 2008
Posts: 18

PostPosted: Wed Feb 25, 2009 16:30    Post subject: Reply with quote
I've been using the same code gaganwalia is using... I'm pretty pragmatic -- it's been working for me so I'll stick to it for now... The old man always said, "If it's not broke, don't fix it!"

But thanks for posting this code too -- if I run into problems with my current code in the future, I can use this code as well... if it's more efficient as someone mentioned, I might switch over anyways...
frater
DD-WRT Guru


Joined: 07 Jun 2006
Posts: 2777

PostPosted: Wed Feb 25, 2009 16:59    Post subject: Reply with quote
It's more efficient.
It will try to download until it succeeds.
Your WAN might not be up when it tries to download the file and your host file will be empty.

Furthermore it doesn't stop doing other things as it is running as a daemon. It also tellls your dnsmasq to reload settings.

_________________
Asus RT16N + OTRW
Kingston 4GB USB-disk 128 MB swap + 1.4GB ext3 on /opt + 2 GB ext3 on /mnt
Copperjet 1616 modem in ZipB-config
Asterisk, pixelserv & Pound running on router
Another Asus RT16N as WDS-bridge

DD-WRT v24-sp2 vpn (c) 2010 NewMedia-NET GmbH
Release: 12/16/10 (SVN revision: 15758M)
gaganwalia
DD-WRT User


Joined: 24 Dec 2007
Posts: 449

PostPosted: Wed Feb 25, 2009 17:00    Post subject: Reply with quote
frater, there is nothing to be mad at. the purpose is to use the best method with the least resources to accomplish something.

your method is creating extra files and also the user has to set many more options.

the way i got it to work with less options and less diskspace usage and less work is as follows:

under the setup page, router setup, use local dns as your router ip.

save the following to your startup

wget -q -O - http://www.mvps.org/winhelp2002/hosts.txt | grep "^127.0.0.1" | gre
p -v localhost | awk '{print "192.168.6.1\t"$2}' | tr -d '\015' > /tmp/hosts
stopservice dnsmasq
startservice dnsmasq


make sure you replace 192.168.6.1 with your router IP or with 0.0.0.0 (i used my router ip because i am using port 80 to host a html file that tells the user that the router has blocked the site)

thats it, and it works.

now you said that the hosts file has a lot of other stuff in it. true. when you stop and start the service, those extra entries are appended to the hosts file that was just created. (I verified this) so nothing is overwritten or lost.

and this saves disk space on the router, and also the user does not have to create all the extra files.

this is working just great.

AND I am really sorry for messing up your thread. Wink
frater
DD-WRT Guru


Joined: 07 Jun 2006
Posts: 2777

PostPosted: Wed Feb 25, 2009 17:36    Post subject: Reply with quote
The /tmp/hosts file is filled with real hosts on your network. I have, for instance, software that makes use of this info. ( http://www.dd-wrt.com/phpBB2/viewtopic.php?t=37958 )
Even if you only append to this file it's the same as messing up.

My code merely writes a 200 byte file which is far from wasting space compared to the 0.5 MB which is the size of the custom host-file.

I don't think you really know what it is doing or you wouldn't make these kind of statements.

Your code not only messes up /etc/hosts but if it fails to download the file (after a powercut your modem could still be offline) it doesn't do anything about it.

There is NO waste of space.


BTW. killall -HUP dnsmasq is something else than stopservice dnsmasq and startservice dnsmasq (if we are talking about efficient coding)

_________________
Asus RT16N + OTRW
Kingston 4GB USB-disk 128 MB swap + 1.4GB ext3 on /opt + 2 GB ext3 on /mnt
Copperjet 1616 modem in ZipB-config
Asterisk, pixelserv & Pound running on router
Another Asus RT16N as WDS-bridge

DD-WRT v24-sp2 vpn (c) 2010 NewMedia-NET GmbH
Release: 12/16/10 (SVN revision: 15758M)
frater
DD-WRT Guru


Joined: 07 Jun 2006
Posts: 2777

PostPosted: Thu Feb 26, 2009 7:21    Post subject: Reply with quote
gaganwalia wrote:

frater, there is nothing to be mad at. the purpose is to use the best method with the least resources to accomplish something.

That's the reason I created this seperate thread. It's exactly the code from which I wished to distance. Your example is NOT the best method and it does NOT use less resources. It just appears to you this way. Your code is simpler in design (it leaves away stuff), but in fact it puts more strain on your router and in some way it's even erroneous.
gaganwalia wrote:

and this saves disk space on the router, and also the user does not have to create all the extra files.

Can you tell me what the advantage is of that? This is Linux and a file system is nothing more than a database. A 300~400 byte file and you're downloading more than 500 KB? In the other thread they didn't even care about translating spaces into tabs like I did. I'm seperating static and dynamic data which is best practice.

The 2 files in /opt/etc/hosts and /jffs/hosts are optional and this part is not needed (I'm now giving 2 examples in my first post). It is in fact the thing you were probably planning to do (it's for merging in your manual entries).

Because /etc/hosts (or in fact /tmp/hosts) are left untouched, the file stays small and can be quickly parsed and maintained by the system. This would be slowed down if it suddenly became a >500 KB file. You even gave the reason why it was inefficient:
gaganwalia wrote:

when you stop and start the service, those extra entries are appended to the hosts file that was just created.

How much extra resources is this taking compared to maintaining the small /etc/hosts?

Stopping dnsmasq and starting dnsmasq puts a lot of strain on the resources anyhow and during startup all those resources are needed. I even considered placing the sleep before downloading the file and am getting more convinced it is the best thing to do. It will make the router responsive a bit sooner at startup so it can finish its more important tasks.

Your code also stops the router from doing other stuff when it starts its one and only try because it doesn't detach itself. My code detaches itself.

There is more to this code than it appears at first glance and I did it this way for various reasons. Reasons others didn't even consider. I'm not holding it against you that you don't see the reasons why I'm doing it this way, but I do when you're claiming your way is more efficient without really understanding why I'm doing it like this.

_________________
Asus RT16N + OTRW
Kingston 4GB USB-disk 128 MB swap + 1.4GB ext3 on /opt + 2 GB ext3 on /mnt
Copperjet 1616 modem in ZipB-config
Asterisk, pixelserv & Pound running on router
Another Asus RT16N as WDS-bridge

DD-WRT v24-sp2 vpn (c) 2010 NewMedia-NET GmbH
Release: 12/16/10 (SVN revision: 15758M)
gaganwalia
DD-WRT User


Joined: 24 Dec 2007
Posts: 449

PostPosted: Thu Feb 26, 2009 8:53    Post subject: Reply with quote
well, true frater, your post makes sense. and you guys are the gurus, i am not going to argue about that.

me being a MS junkie had trouble with your code and didnt understand it. moreover it was dealing with /etc/hosts and /jffs/hosts and i saw nothing like that on my file system. so i figured to go the shortcut way.

is it possible for you to explain your code line by line and WHY it is like that. that would make much more sense to people like me.

I try to get things running in a shortcut because it seems easier, but maybe in the long run it is better to properly think out the code and why it is to be done in a certain way.

I would appreciate a full breakdown of your code if you dont mind, and we can only ask this from the gurus, so please do us a favor by explaining your code as you would explain this to a young kid.

remember, we are not experts at these things.
frater
DD-WRT Guru


Joined: 07 Jun 2006
Posts: 2777

PostPosted: Thu Feb 26, 2009 10:37    Post subject: Reply with quote
gaganwalia wrote:
is it possible for you to explain your code line by line and WHY it is like that. that would make much more sense to people like me.

Fair enough
This is the IP where you want everything directed to. It's on a seperate line to avoid messing up the code itself.
Code:
_rogue=0.0.0.0

Now it starts to write the script /tmp/write_dlhosts and make it executable
Code:
echo -e "#!/bin/sh\nn=1\nwhile sleep 60\ndo\n\twget -q -O - http://www.mvps.org/winhelp2002/hosts.txt | grep \"^127.0.0.1\" | grep -v localhost | awk '{print \"$_rogue\\\t\"\$2}' | tr -d '\\\015' >/tmp/dlhosts\n" >/tmp/write_dlhosts
echo -e "\t[ \`grep -il doubleclick /tmp/dlhosts\` ] && break\n\t[ \$n -gt 5 ] && break\n\tlet n+=1\ndone\nkillall -HUP dnsmasq" >>/tmp/write_dlhosts
chmod +x /tmp/write_dlhosts

Now it starts the script as a daemon (using the &). This means it will continue the startup script.
Code:
/tmp/write_dlhosts &


Using this method (script writing script) I'm able to run it as a daemon so it doesn't stop the router with its startup script.
This is the script which is created:
Code:

#!/bin/sh
n=1
while sleep 60
do
 wget -q -O - http://www.mvps.org/winhelp2002/hosts.txt | grep "^127.0.0.1" | grep -v localhost | awk '{print "0.0.0.0\t"$2}' | tr -d '\015' >/tmp/dlhosts"
 [ `grep -il doubleclick /tmp/dlhosts` ] && break
 [ $n -gt 5 ] && break
 let n+=1
done
killall -HUP dnsmasq


It's a loop which runs for a maximum of 5 times until it succeeds to download a file containing "doubleclick" after every download it will wait a minute.
I'm using a seperate file for these entries because I don't want to mess up the entries in /etc/hosts.

With killall -HUP dnsmasq I'm telling dnsmasq to reload its data.

As I said it might even be better to move the "sleep 60" to the beginning and relieve the router somewhat because it's already quite busy during startup. Furthermore there's a bigger chance the modem is up by that time.

The 2 extra lines were:
Code:
[ -e /jffs/hosts ] && cat /jffs/hosts >>/tmp/dlhosts
[ -e /opt/etc/hosts ] && cat /opt/etc/hosts >>/tmp/dlhost

They add the contents of /opt/etc/hosts and /jffs/hosts to the list if they exist.

If you need more explanation you'll get it.

The code I'm running myself will also start a 2nd instance of the http-server on dd-wrt to respond. This is not as good as running pixelserv on a big machine (It's perl)

_________________
Asus RT16N + OTRW
Kingston 4GB USB-disk 128 MB swap + 1.4GB ext3 on /opt + 2 GB ext3 on /mnt
Copperjet 1616 modem in ZipB-config
Asterisk, pixelserv & Pound running on router
Another Asus RT16N as WDS-bridge

DD-WRT v24-sp2 vpn (c) 2010 NewMedia-NET GmbH
Release: 12/16/10 (SVN revision: 15758M)


Last edited by frater on Thu Feb 26, 2009 17:59; edited 1 time in total
frater
DD-WRT Guru


Joined: 07 Jun 2006
Posts: 2777

PostPosted: Thu Feb 26, 2009 10:47    Post subject: Reply with quote
I have moved "sleep 60" to the beginning of the loop. This will put less stress on router during startup and give it more time to get its WAN running properly (a powercut could have put the modem offline).

Code:
while sleep 60
do
  ....
  ....
done

instead of
Code:
while true
do
  ....
  ....
  sleep 60
done

_________________
Asus RT16N + OTRW
Kingston 4GB USB-disk 128 MB swap + 1.4GB ext3 on /opt + 2 GB ext3 on /mnt
Copperjet 1616 modem in ZipB-config
Asterisk, pixelserv & Pound running on router
Another Asus RT16N as WDS-bridge

DD-WRT v24-sp2 vpn (c) 2010 NewMedia-NET GmbH
Release: 12/16/10 (SVN revision: 15758M)
Goto page Previous  1, 2, 3, ... 12, 13, 14  Next Display posts from previous:    Page 2 of 14
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum