My Ad/Malware block script

Post new topic   Reply to topic    DD-WRT Forum Index -> Advanced Networking
Goto page 1, 2, 3, 4, 5, 6  Next
Author Message
Sunshine
DD-WRT Novice


Joined: 16 Aug 2017
Posts: 1

PostPosted: Mon Sep 02, 2019 19:11    Post subject: My Ad/Malware block script Reply with quote
Hey guys, hope this is the right place to post this. Just wanted to share my ad/malicious block script. Been using it for a couple of years with good success, though I'm sure it can be improved upon. There's no connectivity or error checking, and there should be a wan check before running. Of course this assumes your using dnsmasq as local dns server. Anyway hope someone finds it useful, and if you improve or add host sources, please do post!

Code:

#!/bin/sh
#---Ad block script for DD-WRT using dnsmasq

hostspath="/tmp"
tmpfile="$hostspath/tmphosts"
hostfile="$hostspath/adhosts"
conffile="$hostspath/dnsmasq.conf"
logfile="$hostspath/adblock.log"

>$tmpfile
>$hostfile
>$logfile

sleep 20

wget -O - "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&mimetype=plaintext" >> $tmpfile 2>>$logfile
wget -O - "http://www.mvps.org/winhelp2002/hosts.txt" >> $tmpfile 2>>$logfile
wget -O - "http://www.malwaredomainlist.com/hostslist/hosts.txt" >> $tmpfile 2>>$logfile


sed -e 's/#.*$//' -e 's/127.0.0.1/0.0.0.0/g' -e 's/\r//' $tmpfile | grep 0.0.0.0 |
grep -v localhost | sort | uniq -u >> $hostfile

#Uncomment to remove temp hosts file, use to debug output before formatting.
#rm $tmpfile

#Added to allow downloading of files hosted behind and adfly url.
sed -i /"adf.ly"/d "$hostfile"

echo "addn-hosts=$hostfile" >> $conffile

#Kill and reload dnsmasq. For some reason if user not specified dnsmasq will
#start as nobody (cron?). Must explicitly define user,group, and config file.
#These are the options DD-WRT uses at startup
killall dnsmasq
sleep 1
dnsmasq -u root -g root --conf-file=$conffile
Sponsor
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 1366
Location: Appalachian mountains, USA

PostPosted: Mon Sep 02, 2019 21:15    Post subject: Reply with quote
Edited 25 September 2019 (date of my post below pointing out the correction) to add the -f flag to each curl, to provide correct behavior when a download site doesn't exist at the URL given. But don't use this version of the code... Find the most recent one, further down the thread.

It's amazing how similar in spirit your script and mine are. I add addn-hosts=/tmp/badhosts to Additional Dnsmasq Options in the GUI, then in the Startup Commands in the GUI I add the script. I have a scheduled reboot every night and so running it at boot is enough to keep things current.
Code:
#hosts to block in dnsmasq (Alozaros 7/8/18 post in
#https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=315773)
  ( cd /tmp ; \
    touch badhosts ; \
    sleep 30 ; \
    ( ERRA=0 ; ERRB=0 ; ERRC=0 ; \
      curl -sf http://winhelp2002.mvps.org/hosts.txt \
        2>badhosts.log || ERRA=$? ; \
      curl -sf http://sbc.io/hosts/hosts \
        2>>badhosts.log || ERRB=$? ; \
      curl -skf https://someonewhocares.org/hosts/zero/hosts \
        2>>badhosts.log || ERRC=$? ; \
      echo $ERRA $ERRB $ERRC > badhosts.errcodes \
    ) | sed 's/\t/ /g; /^0\.0\.0\.0 /!d; s/ *\#.*$//; s/\r//' \
      | sort -u \
      | sed -E '/\.hulu(|ad)\./d' \
      > badhosts \
    ) && stopservice dnsmasq && startservice dnsmasq &

Ideally badhosts.errcodes ends up looking like 0 0 0 to indicate that the three curls had no errors. The positions of any nonzeros show which curls failed, and in that case badhosts.log will be nonempty and will contain the error messages. The sleep is to give my DNSCrypt system time to get going before the curls need to use it. The touch is to make sure there is a file there to read when DNSmasq looks for it too early, before badhosts has been properly built. The hulu sed line near the end wires in an exception for the two hulu ad sites needed for hulu streaming, at least here, to work at all. The earlier sed converts Windows line endings to linux ones, eliminates any line that some evildoer might splice into one of the source files like 34.23.12.3 goodbank.com, where of course the issue is that 34.23.12.3 might actually be the IP of badbank.com. That sed also strips comments for the benefit of the sort -u which needs comments gone to reliably recognize when two entries are really identical.

I've only been using this script for a month or so, so it's a work in progress. As you can see in the comment at its beginning, the inspiration is due to another posted script by Alozaros.

_________________
5 Linksys WRT1900ACSv2 routers on 50474: VLANs, VAPs, NAS, client mode, OpenVPN client (AirVPN), wireguard servers (via AirVPN port forwarding) and clients (AzireVPN and AirVPN), 3 DNSCrypt DNS providers via VPN clients.


Last edited by SurprisedItWorks on Wed Sep 25, 2019 21:07; edited 1 time in total
PavelVD
DD-WRT User


Joined: 26 Jul 2019
Posts: 86

PostPosted: Fri Sep 06, 2019 13:42    Post subject: Reply with quote
The same method using JFFS - I give a link so as not to get lost in a related topic.
A bit of criticism:
I have already encountered that one or two links to download "curl" did not work about the schedule. As a result, instead of the expected 40,000 entries in the badhosts file, it turned out to be a little more than 10,000.
It’s bad that any of the download links may die temporarily or permanently.
In general, I ask the author to consider changing the script so that if the download of any file fails, the last successful copy is used.
Thank you for your work!
PS
On the expanses of this form, I got a link to a good applicant for your script:
Quote:
# Title: StevenBlack/hosts with the social extension
#
# This hosts file is a merged collection of hosts from reputable sources,
# with a dash of crowd sourcing via Github
#
# Date: 06 September 2019 04:21:56 (UTC)
# Extensions added to this file: social
# Number of unique domains: 42,119
#
# Fetch the latest version of this file: https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/social/hosts
# Project home page: https://github.com/StevenBlack/hosts
# Project releases: https://github.com/StevenBlack/hosts/releases
#
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 1366
Location: Appalachian mountains, USA

PostPosted: Fri Sep 06, 2019 15:20    Post subject: Reply with quote
PavelVD wrote:
https://forum.dd-wrt.com/phpBB2/viewtopic.php?p=1175113

Yes, thanks for your work there!
Quote:
I have already encountered that one or two links to download "curl" did not work about the schedule. As a result, instead of the expected 40,000 entries in the badhosts file, it turned out to be a little more than 10,000. It’s bad that any of the download links may die temporarily or permanently.

Yes, it certainly is! However, the only times I've had the curls in my script fail (that I caught) was when I was experimenting with a dd-wrt build that gave me lots of other problems as well. Never had any issue on solid builds for my router (e.g. 40009). On one of the newer builds (40890?), I had serious curl trouble unless I upped the startup delay to 90 sec or more. Finally gave up on that build for that and other reasons.
Quote:
In general, I ask the author to consider changing the script so that if the download of any file fails, the last successful copy is used.

If I were willing to start using /jffs or were interested in long dd-wrt uptime runs and therefore a cron job to repeat a script periodically, I'd do that for sure. Neither of those applies to me though. At this point my dd-wrt config is strictly /jffs free with nightly reboots.
Quote:
On the expanses of this form, I got a link to a good applicant for your script:

Nice! I'll have a look soon and see about integrating it into my script.

_________________
5 Linksys WRT1900ACSv2 routers on 50474: VLANs, VAPs, NAS, client mode, OpenVPN client (AirVPN), wireguard servers (via AirVPN port forwarding) and clients (AzireVPN and AirVPN), 3 DNSCrypt DNS providers via VPN clients.
tinkeruntilitworks
Guest





PostPosted: Fri Sep 06, 2019 19:44    Post subject: Reply with quote
the others host files are in the stevenblack file

you just need the one http://sbc.io/hosts/hosts



if you scroll down a bit on this link he shows all the sources
https://github.com/StevenBlack/hosts
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 1366
Location: Appalachian mountains, USA

PostPosted: Sat Sep 07, 2019 15:31    Post subject: Reply with quote
tinkeruntilitworks wrote:
the others host files are in the stevenblack file

you just need the one http://sbc.io/hosts/hosts

if you scroll down a bit on this link he shows all the sources
https://github.com/StevenBlack/hosts

Just tested this. Turns out the sbc.io file gives me 39641 lines (once a couple of hulu lines are deleted by my script), adding the winhelp2002.mvps.org file gives the same total, making the latter file redundant, but including the someonewhocares.org file adds 122 lines. My grand total today is 39763 lines, somewhat smaller than in the original tests, which yielded a total over 42K. The source files do get updated, so someone seems to have cleaned out a couple thousand entries!

_________________
5 Linksys WRT1900ACSv2 routers on 50474: VLANs, VAPs, NAS, client mode, OpenVPN client (AirVPN), wireguard servers (via AirVPN port forwarding) and clients (AzireVPN and AirVPN), 3 DNSCrypt DNS providers via VPN clients.
tinkeruntilitworks
Guest





PostPosted: Sat Sep 07, 2019 15:43    Post subject: Reply with quote
i think the difference in numbers on the stevenblack he makes an effort not to break stuff and he leaves some items commented

so doubling up with the someonewhocares might be a good idea for stricter blocking

ultimately whatever works best is different for everyone. was just noting other list was an all in one
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 1366
Location: Appalachian mountains, USA

PostPosted: Sat Sep 07, 2019 16:20    Post subject: Reply with quote
tinkeruntilitworks wrote:
he leaves some items commented
Indeed. My script deletes all comments just to avoid confusion.

Agree there are many paths to reasonable success here, not just one!

_________________
5 Linksys WRT1900ACSv2 routers on 50474: VLANs, VAPs, NAS, client mode, OpenVPN client (AirVPN), wireguard servers (via AirVPN port forwarding) and clients (AzireVPN and AirVPN), 3 DNSCrypt DNS providers via VPN clients.
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 1366
Location: Appalachian mountains, USA

PostPosted: Sun Sep 08, 2019 22:58    Post subject: Reply with quote
Edited 25 September 2019 (date of my post below pointing out the correction) to add the -f flag to each curl, to provide correct behavior when a download site doesn't exist at the URL given. Don't use this code though. Find the latest version further down this thread.

Turns out I had a bug in my adblocker startup code that caused it to never log error information. In this corrected version, I correct that (and log info on success as well), add a syslog entry showing the curl exit codes, and reinitialize dnsmasq with -HUP without actually stopping/starting it. There's a bit of general coding cleanup as well.
Code:
#hosts to block in dnsmasq (inspired by Alozaros 7/8/18 post
#in https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=315773)
  ( cd /tmp; touch badhosts; sleep 30
    ( EXA=0 ; EXB=0 ; EXC=0
      curl -f  http://winhelp2002.mvps.org/hosts.txt || EXA=$?
      curl -f  http://sbc.io/hosts/hosts || EXB=$?
      curl -kf https://someonewhocares.org/hosts/zero/hosts || EXC=$?
      echo $EXA $EXB $EXC > badhosts.codes
    ) 2>badhosts.log \
    | sed 's/\t/ /g; /^0\.0\.0\.0 /!d; s/ *\#.*$//; s/\r//' | sort -u \
    | sed -E '/\.hulu(|ad)\./d' > badhosts
    logger -t ': adblocker curl exit codes' <badhosts.codes
  ) && killall -HUP dnsmasq &

I decided to leave in all three curls, even though the second one has almost all the entries. I figure that the redundancy is good in that if one or more of those sites goes offline, there will still be a decent badhosts list put together from the others.

And Sunshine, thanks for starting this off. Never meant for it to eat your whole thread though. Crying or Very sad

_________________
5 Linksys WRT1900ACSv2 routers on 50474: VLANs, VAPs, NAS, client mode, OpenVPN client (AirVPN), wireguard servers (via AirVPN port forwarding) and clients (AzireVPN and AirVPN), 3 DNSCrypt DNS providers via VPN clients.


Last edited by SurprisedItWorks on Wed Sep 25, 2019 21:09; edited 1 time in total
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 5392
Location: UK, London, just across the river..

PostPosted: Sun Sep 08, 2019 23:06    Post subject: Reply with quote
can you i use it with wget instead of curl??

can i add one line for 127.0.0.1 like this?

| sed 's/\t/ /g; /^127\.0\.0\.1 /!d; s/ *\#.*$//; s/\r//' \

_________________
Atheros
TP-Link WR740Nv1 ---DD-WRT 50500 WAP
TP-Link WR1043NDv2 -DD-WRT 50963 Gateway,DNS,AP Isolation,Ad-Block,Firewall,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 -DD-WRT 50963 Gateway,DNS,Ad-Block,Firewall,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 -Gargoyle OS 1.13.0 AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear R7800 --DD-WRT 50963 Gateway,DNS,AD-Block,AP&Net Isolation,VLAN's,Firewall,DoT,Vanilla
Netgear R9000 --DD-WRT 50927 Gateway,DNS,AD-Block,AP Isolation,Firewall,Forced DNS,DoT,2,4Ghz only,Vanilla
Broadcom
Netgear R7000 ---DD-WRT 50963 Gateway,DNS,AD-Block,Firewall,Forced DNS,VLAN's,DoT,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby for DNS over TLS I DNSCrypt v2 by mac913
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 1366
Location: Appalachian mountains, USA

PostPosted: Sun Sep 08, 2019 23:49    Post subject: Reply with quote
Alozaros wrote:
can you i use it with wget instead of curl??

I believe you can replace each curl with wget -O - except for the last curl. That one is pointed at an https site, which I don't believe our dd-wrt wget can access. So you'd delete that third curl line and the related uses of variable EXC. Here's the thing to try:
Code:
( cd /tmp; touch badhosts; sleep 30
  ( EXA=0 ; EXB=0
    wget -O - http://winhelp2002.mvps.org/hosts.txt || EXA=$?
    wget -O - http://sbc.io/hosts/hosts || EXB=$?
    echo $EXA $EXB > badhosts.codes
  ) 2>badhosts.log \
  | sed 's/\t/ /g; /^0\.0\.0\.0 /!d; s/ *\#.*$//; s/\r//' | sort -u \
  | sed -E '/\.hulu(|ad)\./d' > badhosts
  logger -t ': adblocker wget exit codes' <badhosts.codes
) && killall -HUP dnsmasq &
I'll leave actual experimenting to you.
Quote:
can i add one line for 127.0.0.1 like this?

| sed 's/\t/ /g; /^127\.0\.0\.1 /!d; s/ *\#.*$//; s/\r//' \

You wouldn't want to replace the existing sed command of /^0\.0\.0\.0 /!d because that command protects you from malicious table entries. You could replace sed -E '/\.hulu(|ad)\./d' with sed -E '/\.hulu(|ad)\./d;1s/^/127.0.0.1\n/' but it is not needed, because in the dnsmasq config we have set addn-hosts=/tmp/badhosts. That's additional hosts. I believe the original hosts file at /tmp/hosts is still read, and it contains a "127.0.0.1 localhost" line, which apparently is all that dd-wrt needs. I'm really fuzzy on the loopback business, so please educate me if I'm missing something, but my system has been running fine without /tmp/badhosts containing a 127.0.0.1 line.

_________________
5 Linksys WRT1900ACSv2 routers on 50474: VLANs, VAPs, NAS, client mode, OpenVPN client (AirVPN), wireguard servers (via AirVPN port forwarding) and clients (AzireVPN and AirVPN), 3 DNSCrypt DNS providers via VPN clients.
tinkeruntilitworks
Guest





PostPosted: Sun Sep 08, 2019 23:58    Post subject: Reply with quote
sometimes you can get wget to work with https
by just removing the s from the url

not sure if it would work here
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 1366
Location: Appalachian mountains, USA

PostPosted: Mon Sep 09, 2019 1:12    Post subject: Reply with quote
tinkeruntilitworks wrote:
sometimes you can get wget to work with https
by just removing the s from the url

not sure if it would work here

Yeah. Tried that. No luck.

_________________
5 Linksys WRT1900ACSv2 routers on 50474: VLANs, VAPs, NAS, client mode, OpenVPN client (AirVPN), wireguard servers (via AirVPN port forwarding) and clients (AzireVPN and AirVPN), 3 DNSCrypt DNS providers via VPN clients.
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 5392
Location: UK, London, just across the river..

PostPosted: Mon Sep 09, 2019 9:54    Post subject: Reply with quote
whats the pourpouse of this line, more likely, whats is the expected outcome/print...? Is it really needed...?
logger -t ': adblocker curl exit codes' <badhosts.codes
i can see a some results in the log...

but, yep its working with wget...only http sites, as not always https sites have a mirror http...

_________________
Atheros
TP-Link WR740Nv1 ---DD-WRT 50500 WAP
TP-Link WR1043NDv2 -DD-WRT 50963 Gateway,DNS,AP Isolation,Ad-Block,Firewall,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 -DD-WRT 50963 Gateway,DNS,Ad-Block,Firewall,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 -Gargoyle OS 1.13.0 AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear R7800 --DD-WRT 50963 Gateway,DNS,AD-Block,AP&Net Isolation,VLAN's,Firewall,DoT,Vanilla
Netgear R9000 --DD-WRT 50927 Gateway,DNS,AD-Block,AP Isolation,Firewall,Forced DNS,DoT,2,4Ghz only,Vanilla
Broadcom
Netgear R7000 ---DD-WRT 50963 Gateway,DNS,AD-Block,Firewall,Forced DNS,VLAN's,DoT,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby for DNS over TLS I DNSCrypt v2 by mac913
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 1366
Location: Appalachian mountains, USA

PostPosted: Mon Sep 09, 2019 12:12    Post subject: Reply with quote
Alozaros wrote:
whats the pourpouse of this line, more likely, whats is the expected outcome/print...? Is it really needed...?
logger -t ': adblocker curl exit codes' <badhosts.codes
It can be omitted. It's just there to put the exit codes in the syslog, just so that when scanning the log, you can see without other digging that the exit codes returned by the curls or wgets are all zero, meaning they ran fine. Do change curl to wget in your version. It seems I was in too big a hurry. It's just a text label for the log, so you can put anything you like between the single quotes, as long as it is not too long, which would result in truncation.
_________________
5 Linksys WRT1900ACSv2 routers on 50474: VLANs, VAPs, NAS, client mode, OpenVPN client (AirVPN), wireguard servers (via AirVPN port forwarding) and clients (AzireVPN and AirVPN), 3 DNSCrypt DNS providers via VPN clients.
Goto page 1, 2, 3, 4, 5, 6  Next Display posts from previous:    Page 1 of 6
Post new topic   Reply to topic    DD-WRT Forum Index -> Advanced Networking All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum