My Ad/Malware block script

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Advanced Networking
Goto page Previous  1, 2, 3, 4, 5  Next
Author Message
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 989
Location: Appalachian mountains, USA

PostPosted: Wed Sep 25, 2019 23:18    Post subject: Reply with quote
Edited 9 Dec 2019 to add wait loop before curls, to make sure the network is available, and an outer subshell (first two noncomment lines and a final closing parenthesis) that can be exited early if a lockfile shows it's already been run. You likely won't need either tweak in a shell version that you run with cron.

So while I am in a fixing mood, and continuing my idiotic tradition of posting new versions way too soon and after minimal testing of changes (a dd-wrt tradition!), here's the latest adblocker code for GUI>Administration>Commands, Startup. Note the first comment re the other GUI edit required.

The main point here is just cleanup for readability. However, there are actual changes here to that earlier hurried kluge of syslog-entry machinery. This version is cleaner. Pointless group-command braces in the Download function have been removed as well. While I've badmouthed cats in pipelines, even in this very thread, I don't actually see a cleaner way here (in the syslog-entry code). So there it lurks.
Code:
#Hosts to block in dnsmasq using Add'l Config: addn-hosts=/tmp/badhosts
#Alozaros URLs 7/8/18 at https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=315773
#sbc.io URL 1st at https://github.com/StevenBlack/hosts/blob/master/readme.md
( STARTED=/tmp/root/StartedAdBlocker
  [[ -f $STARTED ]] && exit || touch $STARTED
  ( cd /tmp; touch badhosts; sleep 30
    until ping -c 1 -w 1 &>/dev/null cloudflare.com; do sleep 120; done
    sed 's/\./\\./g;s/.*/\/( |\\.)&$\/d/' <<'ENDWHITE' >badhosts.whitelist
hulu.com
huluad.com
ENDWHITE
    awk '{print "0.0.0.0 "$1}' <<'ENDBLACK' >badhosts.blacklist
graph.facebook.com
ads.facebook.com
connect.facebook.net
ENDBLACK
    { Download(){ curl -kf $1; EX="$EX $?"; }; EX=
      Download http://winhelp2002.mvps.org/hosts.txt
      Download http://sbc.io/hosts/hosts
      Download https://someonewhocares.org/hosts/zero/hosts
      echo $EX > badhosts.codes
    } 2>badhosts.log \
    | sed 's/\t/ /g; /^0\.0\.0\.0 /!d; s/ *\#.*$//; s/\r//' \
    | sort -u - badhosts.blacklist \
    | sed -Ef badhosts.whitelist > badhosts
    wc -l badhosts \
    | awk '{printf "%s has %d lines, exit codes ",$2,$1}' \
    | cat - badhosts.codes \
    | logger -t "startup-code adblocker"
  ) && killall -HUP dnsmasq & )

Any URLs containing $ or & (maybe some other characters also) must be enclosed in single quotes like 'https://blah.com/foo?bar&bat' and such quotes never hurt. Don't use double quotes. Note as before that the sleep may need lengthening for some router/build/config combinations. Try it and check the log entry, looking for 0 0 0 for exit codes (or look at /tmp/badhosts.codes in the CLI) and a decent-length badhosts file. As of today -- it can vary a bit day to day as the files posted at the URLs are edited -- I have 43276 lines. On the first try it's a good idea to look at /tmp/badhosts.log in the CLI to verify that the curls are doing something reasonable.

If you have an old or small build that doesn't have curl, you can replace curl -kf with wget -O - but be careful to include that final hyphen, and note that O is the letter "Oh" and not a zero. You'll likely get nothing but an error from the third Download line, as the wget in dd-wrt does not, by default anyway, handle https sites. You can just delete that third Download line and get by just fine on two, with a slightly smaller badhosts file resulting.

The whitelist entries here allow hulu ads, without which hulu will not stream, and the blacklist entries hamper some of facebook's spying and ad displays. The connect.facebook.net entry will likely break logging into other sites using facebook credentials in addition to hampering their spying. Edit: It appears that blacklisting graph.facebook.com kills facebook messenger, also in addition to hampering their spying. Boo hoo.

Whitelist entries can be either domains, as above, or linux extended regular expressions that expand to domains when each "." is escaped. The latter means the two entries here could be condensed to the one line hulu(ad)?.com or hulu(|ad).com if desired. A whitelisted domain foo.com implicitly whitelists x.foo.com and such as well. All that flexibility is absent for the blacklist, however. Those names must be domains. Either list can be empty, with zero lines.

_________________
Five WRT1900ACSv2's on 42926, 44048.
VLANs, multiple VAPs, NAS, client-mode travel router, OpenVPN client (AirVPN), DDNS, wireguard server, two wireguard clients (AzireVPN), use of two DNSCrypt DNS providers (inc Quad9) through OpenVPN and wireguard clients.


Last edited by SurprisedItWorks on Sat Dec 28, 2019 18:27; edited 2 times in total
Sponsor
pikasso
DD-WRT Novice


Joined: 06 Nov 2019
Posts: 2

PostPosted: Wed Nov 06, 2019 23:40    Post subject: Reply with quote
Hi there..
SurprisedItWorks.. I registered today to ddwrt just to say thank you.
This method is so simple and automatic, no need of flash drive, tuning something in addition, and just simply works.
I must say I had a lot of fun watching the posts updates, from one method then using curl.. then wget.. then optimizing the sequences.. really perfect mindset and script.

Thank you !
Pierrick
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 989
Location: Appalachian mountains, USA

PostPosted: Thu Nov 07, 2019 5:13    Post subject: Reply with quote
Thank you, Pierrick! Unusual that people bother with that. It is appreciated.

And so here's a bonus, something I just added to my routers a couple of weeks ago. Add this as a new line after the sleep and before the first sed:

until ping -c 1 -w 10 &> /dev/null cloudflare.com; do sleep 110; done

This prevents things from going further until the router is able to ping cloudflare.com. Tries the ping every two minutes.

I was moved to add this when I had a power failure at home for an hour, and when the power came back up, the router booted before the modem. This meant the router was trying to do the Download steps before success was possible, and as a result all the Downloads failed and badhosts remained empty. This way it won't try the downloads until the network is up with DNS functioning.

_________________
Five WRT1900ACSv2's on 42926, 44048.
VLANs, multiple VAPs, NAS, client-mode travel router, OpenVPN client (AirVPN), DDNS, wireguard server, two wireguard clients (AzireVPN), use of two DNSCrypt DNS providers (inc Quad9) through OpenVPN and wireguard clients.
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 3959
Location: UK, London, just across the river..

PostPosted: Thu Nov 07, 2019 6:39    Post subject: Reply with quote
does this line keeps pinging even when WAN is on....??

until ping -c 1 -w 10 &> /dev/null cloudflare.com; do sleep 110; done

instead I usually extend adblocker sleep time in startup script as it's placed in the custom section, so it loads independently and its the last thing to load at the end when WAN is available anyway ..

_________________
Atheros
TP-Link WR740Nv1 -----DD-WRT 44538 BS AP,NAT
TP-Link WR740Nv4 -----DD-WRT 44251 BS WAP/Switch
TP-Link WR1043NDv2 ---DD-WRT 44715 BS AP,NAT,AP Isolation,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---DD-WRT 44772 BS AP,NAT,AD Block,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---Gargoyle OS 1.12.0 AP,NAT,QoS,Quotas
Qualcomm/IPQ8065
Netgear R7800 -----DD-WRT 44849 BS AP,NAT,AD-Block,AP&Net Isolation,VLAN's,Firewall,Local DNS,DoT
Broadcom
Netgear R7000 -----DD-WRT 44849 BS AP,Wi-Fi OFF,NAT,AD-Block,Firewall,Local DNS,Forced DNS,DoT,VPN
-----------------------------------------------------------------------------------------------
Stubby for DNS over TLS I DNSCrypt v2 by mac913
pikasso
DD-WRT Novice


Joined: 06 Nov 2019
Posts: 2

PostPosted: Thu Nov 07, 2019 10:12    Post subject: Reply with quote
SurprisedItWorks wrote:
Thank you, Pierrick! Unusual that people bother with that. It is appreciated.


Being around xda and other communities, I know that so well.. I have been having my router since one year but now, I need also more parental control and ads management that I switched recently to ddwrt.. I begin to have fun with it Very Happy

SurprisedItWorks wrote:

until ping -c 1 -w 10 &> /dev/null cloudflare.com; do sleep 110; done


Updated successfully.

For the records.. I had previously a host file on my windows machine.. but now, all my devices are under the same configuration, and with white listing, it saves also some time in case of future forced *inclusion*.

Thanks
ctclark1
DD-WRT Novice


Joined: 08 Feb 2012
Posts: 1

PostPosted: Wed Nov 27, 2019 3:31    Post subject: Reply with quote
This is awesome, thank you! I was getting fed up with Privoxy not really blocking much of anything ad-wise except for whole sites, and stumbled across the "other" thread with your trials, found my way here, and managed to somehow get it working! Laughing I will say it took me a few tries, for some reason the first few times I tried curl I wouldn't get any responses in the logs at all, then wget was always failing even with the extra "-", but then mysteriously curl worked when I went back to trying it, so I wanted to say thank you!

As a comment on it - I ended up dropping the ping wait at the beginning from 110 seconds to 30, hopefully that doesn't cause a problem. Fingers crossed.

I should also add, for some who might encounter it, that graph.facebook.com being blacklisted not only broke my Messenger (on mobile only, not desktop) but also kept the facebook settings on mobile from working at all, so I unfortunately had to whitelist those, but oh well.
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 989
Location: Appalachian mountains, USA

PostPosted: Wed Nov 27, 2019 17:09    Post subject: Reply with quote
ctclark1 wrote:
This is awesome, thank you! I was getting fed up with Privoxy not really blocking much of anything ad-wise except for whole sites, and stumbled across the "other" thread with your trials, found my way here, and managed to somehow get it working! Laughing I will say it took me a few tries, for some reason the first few times I tried curl I wouldn't get any responses in the logs at all, then wget was always failing even with the extra "-", but then mysteriously curl worked when I went back to trying it, so I wanted to say thank you!

As a comment on it - I ended up dropping the ping wait at the beginning from 110 seconds to 30, hopefully that doesn't cause a problem. Fingers crossed.

Thanks for the support! Also, there's absolutely no harm in using a smaller delay (as you did) in that "until..." loop. It just means you'll attempt to ping cloudflare more often while looking for the network to come up.
Quote:
I should also add, for some who might encounter it, that graph.facebook.com being blacklisted not only broke my Messenger (on mobile only, not desktop) but also kept the facebook settings on mobile from working at all, so I unfortunately had to whitelist those, but oh well.

For graph.facebook.com, it should be enough to not blacklist it, since (at least at present) it's not showing up in badhosts anyway. I don't blacklist it here anymore either. I'm currently blacklisting only ads.facebook.com and connect.facebook.net, but the inadequate evidence here so far suggests the latter may disable facebook messenger. I don't use any facebook tools/spyware myself anyway and refuse to install it or even browse to it on my own devices, so to discover problems I have to wait for the rest of the family to swawk.
Twisted Evil

_________________
Five WRT1900ACSv2's on 42926, 44048.
VLANs, multiple VAPs, NAS, client-mode travel router, OpenVPN client (AirVPN), DDNS, wireguard server, two wireguard clients (AzireVPN), use of two DNSCrypt DNS providers (inc Quad9) through OpenVPN and wireguard clients.
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 989
Location: Appalachian mountains, USA

PostPosted: Mon Dec 09, 2019 16:27    Post subject: Reply with quote
It turns out that sometimes Startup code is run more than once by dd-wrt. I believe Apply on some GUI pages will rerun it, for example. So as presented above, the adblocker may get rerun. That's pretty harmless unless two runs of it overlap in time in just the wrong way, but to be super safe, you can replace the whole <Adblocker Code> with this:
Code:
( STARTED=/tmp/root/StartedAdBlocker
  [[ -f $STARTED ]] && exit || touch $STARTED
  <Adblocker Code> )

Of course you can name the file whatever you like. It will be an empty file that exists only so that the second and later runs of the Startup code will "realize" not to execute your code. The variable STARTED is local to the enclosing subshell (the parentheses), so you can re-use the name elsewhere if inclined, just not in the <Adblocker>.

_________________
Five WRT1900ACSv2's on 42926, 44048.
VLANs, multiple VAPs, NAS, client-mode travel router, OpenVPN client (AirVPN), DDNS, wireguard server, two wireguard clients (AzireVPN), use of two DNSCrypt DNS providers (inc Quad9) through OpenVPN and wireguard clients.
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 3959
Location: UK, London, just across the river..

PostPosted: Mon Dec 09, 2019 18:13    Post subject: Reply with quote
got lost here what is all about Razz

just either revise it on the top or post the all revised adblocker as you believe it should be....
here at the end... this article gets interesting as
there are many suggestions all working and useful..

by the way ive never seen it executed twice...as i also keep it in custom script instead...

_________________
Atheros
TP-Link WR740Nv1 -----DD-WRT 44538 BS AP,NAT
TP-Link WR740Nv4 -----DD-WRT 44251 BS WAP/Switch
TP-Link WR1043NDv2 ---DD-WRT 44715 BS AP,NAT,AP Isolation,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---DD-WRT 44772 BS AP,NAT,AD Block,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---Gargoyle OS 1.12.0 AP,NAT,QoS,Quotas
Qualcomm/IPQ8065
Netgear R7800 -----DD-WRT 44849 BS AP,NAT,AD-Block,AP&Net Isolation,VLAN's,Firewall,Local DNS,DoT
Broadcom
Netgear R7000 -----DD-WRT 44849 BS AP,Wi-Fi OFF,NAT,AD-Block,Firewall,Local DNS,Forced DNS,DoT,VPN
-----------------------------------------------------------------------------------------------
Stubby for DNS over TLS I DNSCrypt v2 by mac913
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 989
Location: Appalachian mountains, USA

PostPosted: Mon Dec 09, 2019 18:41    Post subject: Reply with quote
Alozaros wrote:
got lost here what is all about Razz

just either revise it on the top

Good point. I've gone back and revised the latest full version with the more recent updates.
Quote:
or post the all revised adblocker as you believe it should be....
here at the end... this article gets interesting as
there are many suggestions all working and useful..

by the way ive never seen it executed twice...as i also keep it in custom script instead...

I have seen other startup code executed twice. Yesterday the second execution, due to a bug in my (other) code, crashed my router so completely that I needed a reset/restore. It got my attention! And this morning I finally noticed that the scripting section in the wiki warns us to plan for startup to run more than once. Of course if you call your custom script from startup, you may need something like
Code:
( STARTED=/tmp/root/StartedCustom
  [[ -f $STARTED ]] && exit || touch $STARTED
  sh /tmp/custom.sh & )

If you don't call the adblocker from startup, I can't see any reason you'd need to care about double running, so you can pretty much ignore today's posts. Or if you are looking back to the last posted full version of the code (https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=321121&start=42), strip off the new two first (noncomment) lines AND the new final parenthesis on the last line. Gotta keep those parens in pairs!

BTW all, yesterday this adblocker saved my wife's computer from tags.bluekai.com, supposedly a nasty site that infects browsers with a popup creation engine that fills one's life with ads. There's lots of stuff online about it and how to remove it. Just for grins, I checked and discovered that Quad9 DNS does not block it but that Adguard DNS does.

_________________
Five WRT1900ACSv2's on 42926, 44048.
VLANs, multiple VAPs, NAS, client-mode travel router, OpenVPN client (AirVPN), DDNS, wireguard server, two wireguard clients (AzireVPN), use of two DNSCrypt DNS providers (inc Quad9) through OpenVPN and wireguard clients.
EnigmaSol
DD-WRT Novice


Joined: 07 Jan 2020
Posts: 5

PostPosted: Tue Jan 07, 2020 16:47    Post subject: Adchoices Ads Reply with quote
Hi all,

I have copied and saved the script as startup (Thank you all who created/contributed to the script), DNSmasq is enabled but Local DNS is disabled.

The Adchoices ads and google ads are still able to slip through (yahoo.com). Am I doing something wrong (noob)? It seems all the sites I frequent are served by Adchoices/Google ads and there is no solution? even Ad Block plus extension works only sometimes.

Thanks again!
tinkeruntilitworks
Guest





PostPosted: Tue Jan 07, 2020 22:12    Post subject: Reply with quote
can't speak for this script but i run something somewhat similar. hosts blocking has become harder. https sites can inject ads
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 3959
Location: UK, London, just across the river..

PostPosted: Wed Jan 08, 2020 8:35    Post subject: Re: Adchoices Ads Reply with quote
EnigmaSol wrote:
Hi all,

I have copied and saved the script as startup (Thank you all who created/contributed to the script), DNSmasq is enabled but Local DNS is disabled.

The Adchoices ads and google ads are still able to slip through (yahoo.com). Am I doing something wrong (noob)? It seems all the sites I frequent are served by Adchoices/Google ads and there is no solution? even Ad Block plus extension works only sometimes.

Thanks again!


Local DNS must be enabled (turned on)
as this script is using a, local host based scanning list....

_________________
Atheros
TP-Link WR740Nv1 -----DD-WRT 44538 BS AP,NAT
TP-Link WR740Nv4 -----DD-WRT 44251 BS WAP/Switch
TP-Link WR1043NDv2 ---DD-WRT 44715 BS AP,NAT,AP Isolation,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---DD-WRT 44772 BS AP,NAT,AD Block,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---Gargoyle OS 1.12.0 AP,NAT,QoS,Quotas
Qualcomm/IPQ8065
Netgear R7800 -----DD-WRT 44849 BS AP,NAT,AD-Block,AP&Net Isolation,VLAN's,Firewall,Local DNS,DoT
Broadcom
Netgear R7000 -----DD-WRT 44849 BS AP,Wi-Fi OFF,NAT,AD-Block,Firewall,Local DNS,Forced DNS,DoT,VPN
-----------------------------------------------------------------------------------------------
Stubby for DNS over TLS I DNSCrypt v2 by mac913
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 989
Location: Appalachian mountains, USA

PostPosted: Wed Jan 08, 2020 15:51    Post subject: Re: Adchoices Ads Reply with quote
EnigmaSol wrote:
Hi all,

I have copied and saved the script as startup (Thank you all who created/contributed to the script), DNSmasq is enabled but Local DNS is disabled.

The Adchoices ads and google ads are still able to slip through (yahoo.com). Am I doing something wrong (noob)? It seems all the sites I frequent are served by Adchoices/Google ads and there is no solution? even Ad Block plus extension works only sometimes.

Thanks again!

There is no 100% solution, as many ads are embedded now in the material you are trying to view and don't involve separate DNS lookups. All we can do is try to reduce the numbers of ads as best we can!

_________________
Five WRT1900ACSv2's on 42926, 44048.
VLANs, multiple VAPs, NAS, client-mode travel router, OpenVPN client (AirVPN), DDNS, wireguard server, two wireguard clients (AzireVPN), use of two DNSCrypt DNS providers (inc Quad9) through OpenVPN and wireguard clients.
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 989
Location: Appalachian mountains, USA

PostPosted: Wed Jan 08, 2020 16:08    Post subject: Re: Adchoices Ads Reply with quote
Alozaros wrote:
Local DNS must be enabled (turned on)
as this script is using a, local host based scanning list....

Have you actually tested this both ways, with and without Local DNS enabled? I have not, but I don't believe Local DNS should be required. The addn-hosts=... dnsmasq feature that this all hinges on simply extends the usual unix use of an /etc/hosts file, which is not related to a local-DNS capability in particular.

_________________
Five WRT1900ACSv2's on 42926, 44048.
VLANs, multiple VAPs, NAS, client-mode travel router, OpenVPN client (AirVPN), DDNS, wireguard server, two wireguard clients (AzireVPN), use of two DNSCrypt DNS providers (inc Quad9) through OpenVPN and wireguard clients.
Goto page Previous  1, 2, 3, 4, 5  Next Display posts from previous:    Page 3 of 5
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Advanced Networking All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum