[Solution] Blocking all Youtube Video Advertisements

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Advanced Networking
Goto page Previous  1, 2, 3, 4
Author Message
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 685
Location: Appalachian mountains, USA

PostPosted: Mon Jan 27, 2020 20:54    Post subject: Reply with quote
Here is the new, improved version of my adblocker. The reason to use it, even if you want only youtube blocking, is to get the logging and diagnostics and safety features. If you don't like it downloading other block lists, just delete the Download URL lines for the lists you don't want to use! Don't want a blacklist? Delete its entries. I've split the whitelist here into two lists, the WHITE list and the ALLWHITE list. Putting foo.com in the WHITE lists just whitelists foo.com. Putting it in the ALLWHITE list whitelists both foo.com and *.foo.com. Using two lists simplified practical whitelisting. Don't like the whitelists? Delete all entries from those lists. It's fine for all these lists to be empty.

There is a new separate PTRLIST here of sites that need special treatment like we've been giving manifest.google.com. These are ad-related sites that in the public DNS system have reverse lookups that don't point back to the original domain but instead point to weird-looking domain names seem to be somehow useful in their ad systems. Don't want those two other sites in that list? Delete them. However, to block youtube ads, you must keep manifest.google.com in that list.
Code:
#Adblocker.  Requires dnsmasq Add'l Config lines
#"addn-hosts=/tmp/badhosts" and "address=/1e100.net/0.0.0.0"
#Re manifest.googlevideo.com for youtube ad blocker, see
#https://forum.dd-wrt.com/phpBB2/viewtopic.php?p=1187268 and
#https://www.reddit.com/r/pihole/comments/9w5swx/i_think_ive_managed_to_block_youtube_ads_with/
( STARTED=/tmp/root/StartedAdBlocker
  [[ -f $STARTED ]] && exit || touch $STARTED
  ( cd /tmp; touch badhosts;
    sleep 30; until ping -c 1 -w 1 &>/dev/null cloudflare.com; do sleep 30; done; sleep 60
    cat <<'ENDPTR' >badhosts.ptr
manifest.googlevideo.com
media.trafficjunky.net
static.trafficjunky.net
ENDPTR
    sed 's/\./\\./g;s/.*/\/ &$\/d/' <<'ENDWHITE' - badhosts.ptr >badhosts.whitelist
ipleak.net
secret.[^g]+oogle.com
ny(mag|post).com
ENDWHITE
    sed 's/\./\\./g;s/.*/\/( |\\.)&$\/d/' <<'ENDALLWHITE' >>badhosts.whitelist
hulu(ad)?.com
ENDALLWHITE
    awk '{print "0.0.0.0 "$1}' <<'ENDBLACK' >badhosts.blacklist
ads.facebook.com
connect.facebook.net
ENDBLACK
    { Download(){ { curl -kf $1; echo -n " $?" >> badhosts.codes; } }
      #sbc.io URL:  https://github.com/StevenBlack/hosts/blob/master/readme.md
        Download http://sbc.io/hosts/hosts
      #Alozaros URLs: 7/8/18 at https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=315773
        Download http://winhelp2002.mvps.org/hosts.txt
        Download https://someonewhocares.org/hosts/zero/hosts
      #CHEF-KOCH trackers: https://github.com/CHEF-KOCH/NSABlocklist/tree/master/HOSTS/Trackers
        CK=https://raw.githubusercontent.com/CHEF-KOCH/NSABlocklist/master/HOSTS/Trackers
        Download $CK/trackers.txt
        Download $CK/Canvas%20font%20fingerprinting.txt
        Download $CK/audio%20fingerprinting%20pages.txt
        Download $CK/canvas%20fingerprinting%20pages.txt
        Download $CK/webrtc%20tracking.txt
      #youtube from Handley, stuart engineering, barcelona: https://gist.github.com/seanhandley
        Download https://gist.githubusercontent.com/seanhandley/d76c51099a32287c0a9074d5f15ac7d0/raw/2bde83a93e68208585d602df787d782036271438/gistfile1.txt
      #other youtube URLs: https://forum.dd-wrt.com/phpBB2/viewtopic.php?p=1186953
        Download https://api.hackertarget.com/hostsearch/?q=googlevideo.com | awk -F , '{print "0.0.0.0",$1}'
        Download https://raw.githubusercontent.com/anudeepND/youtubeadsblacklist/master/domainlist.txt | sed '/---/d;s/^/0.0.0.0 &/'
    } 2>badhosts.log \
    | sed 's/\t/ /g; /^0\.0\.0\.0 /!d; s/ *\#.*$//; s/\r//' \
    | sort -u | sed -Ef badhosts.whitelist \
    | cat - badhosts.blacklist >> badhosts
    #defeat redirecting PTR lookups
      A='^Address [0-9]+: '; I='[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+'
      while read D; do
        { nslookup $D 2>/dev/null; echo -n " $?" >>badhosts.codes; } \
        | sed -nE "/$A$I/{s/$A($I).*/\1 $D/p;q}"
      done <badhosts.ptr >>badhosts
    echo '' >> badhosts.codes
    wc -l badhosts \
    | awk '{printf "%s has %d lines, exit codes",$2,$1}' \
    | cat - badhosts.codes \
    | logger -t "startup-code adblocker"
  ) && killall -HUP dnsmasq & )

This script is meant to be run just once, in Startup at boot time. If I don't reboot for a few days, I start to see youtube ads again. I suspect this is because the badhosts line for manifest.google.com has become too old. But modifying this script so that it could be rerun periodically, say by cron, would require some thought. This is because if you use "Forced DNS Redirection," like I do here, nslookup (in the script) will obtain the wrong IP for manifest.google.com (it will obtain 0.0.0.0) if the blocker is already running. On an ordinary linux system, nslookup could be run with a -port=blah argument to get around the forcing, but on dd-wrt nslookup does not accept the -port argument. So this script works best on systems that get rebooted daily or near daily.

_________________
Six Linksys WRT1900ACSv2 (39144/40009/41954):
VLANs, multiple VAPs, NAS, QoS, client-mode travel router, OpenVPN client/PBR (AirVPN), two DNSCrypt servers (incl Quad9) routed through vpn.
Sponsor
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 685
Location: Appalachian mountains, USA

PostPosted: Wed Jan 29, 2020 18:35    Post subject: Reply with quote
I just did some tweaking of the short version of my youtube blocker above at https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=322606&start=52. See the edit comment just after the code.
_________________
Six Linksys WRT1900ACSv2 (39144/40009/41954):
VLANs, multiple VAPs, NAS, QoS, client-mode travel router, OpenVPN client/PBR (AirVPN), two DNSCrypt servers (incl Quad9) routed through vpn.
Rickz
DD-WRT Novice


Joined: 01 Jan 2020
Posts: 31

PostPosted: Thu Feb 13, 2020 2:42    Post subject: Reply with quote
@SurprisedItWorks
your script never work with me, in both i got IP 0.0.0.0 so all went down, will be better to specific which router model you have and firmware, i believe this is not for any router. Laughing
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 685
Location: Appalachian mountains, USA

PostPosted: Thu Feb 13, 2020 21:59    Post subject: Reply with quote
Rickz wrote:
@SurprisedItWorks
your script never work with me, in both i got IP 0.0.0.0 so all went down, will be better to specific which router model you have and firmware, i believe this is not for any router. Laughing

As my sig says, I'm on Linksys WRT1900ACSv2 routers. They are pretty fast and have a decent amount of memory, so an older, smaller router might indeed be an issue, especially if you try to keep my long list of Download sites. Maybe commenting out many of the downloads to try a small version would make sense. You also might want to experiment with longer sleep times up front. Some older releases for smaller routers do not have curl on them. If you are in that boat, replace curl -kf with wget -O - (including that final hyphen) in the definition of Download.

I've run this on releases 40009, 40784, and 41954. Haven't seen any build-specific issues.

I'm not in a position to say much more. If you have shell-script skills, it shouldn't be too hard to test it out in pieces in ssh to perhaps discover where the issue lies.

_________________
Six Linksys WRT1900ACSv2 (39144/40009/41954):
VLANs, multiple VAPs, NAS, QoS, client-mode travel router, OpenVPN client/PBR (AirVPN), two DNSCrypt servers (incl Quad9) routed through vpn.
Rickz
DD-WRT Novice


Joined: 01 Jan 2020
Posts: 31

PostPosted: Mon Mar 02, 2020 17:45    Post subject: Reply with quote
@SurprisedItWorks

i got the WRT1900ACv2

i edited your script to use it like this:
Quote:
#Youtube adblocker. Requires dnsmasq Add'l Config lines
#"addn-hosts=/tmp/badhosts" and "address=/1e100.net/0.0.0.0"
cat <<'EOF' >/tmp/root/adblocker
#!/bin/sh
cd /tmp
D=manifest.googlevideo.com
DD=$(echo $D | sed 's/\./\\./g')
{ curl -kf http://sbc.io/hosts/hosts
} 2>badhosts.log | sed "s/\t/ /g; /^0\.0\.0\.0 /!d; s/ *\#.*$//; s/\r//; / $DD$/d" | sort -u > badhosts
killall -HUP dnsmasq; sleep 5
A='^Address [0-9]+: '; I='[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+'
nslookup $D 2>/dev/null | sed -nE "/$A$I/{s/$A($I).*/\1 $D/p;q}" >>badhosts
killall -HUP dnsmasq
EOF
chmod +x /tmp/root/adblocker
( sleep 90; /tmp/root/adblocker ) &


how do i know if i have to edit it accordingly your last comment :
Quote:
This code is a stripped-down version of what I've been running successfully for a couple of weeks now, but in stripped-down form it has been tested only modestly. Take a look at the last line of badhosts (perhaps using tail /tmp/badhosts) to check the key manifest.googlevideo.com line. On some earlier dd-wrt builds (even as recently as early 2019), /tmp/root/adblocker will not run the file as a shell script as it should. In that case, do sh /tmp/root/adblocker instead.


how i know if i have to use:
Quote:
sh /tmp/root/adblocker

and which lines i should replace in the script ?

also i have a doubt about which is the best, stripped-down version or should i use the full long version ??

thanks in advance

Laughing
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 685
Location: Appalachian mountains, USA

PostPosted: Tue Mar 03, 2020 17:12    Post subject: Reply with quote
Rickz wrote:
@SurprisedItWorks... i got the WRT1900ACv2... i edited your script to use it like this:
Quote:
#Youtube adblocker. Requires dnsmasq Add'l Config lines
#"addn-hosts=/tmp/badhosts" and "address=/1e100.net/0.0.0.0"
cat <<'EOF' >/tmp/root/adblocker
#!/bin/sh
cd /tmp
D=manifest.googlevideo.com
DD=$(echo $D | sed 's/\./\\./g')
{ curl -kf http://sbc.io/hosts/hosts
} 2>badhosts.log | sed "s/\t/ /g; /^0\.0\.0\.0 /!d; s/ *\#.*$//; s/\r//; / $DD$/d" | sort -u > badhosts
killall -HUP dnsmasq; sleep 5
A='^Address [0-9]+: '; I='[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+'
nslookup $D 2>/dev/null | sed -nE "/$A$I/{s/$A($I).*/\1 $D/p;q}" >>badhosts
killall -HUP dnsmasq
EOF
chmod +x /tmp/root/adblocker
( sleep 90; /tmp/root/adblocker ) &
how do i know if i have to edit it accordingly your last comment :
Quote:
This code is a stripped-down version of what I've been running successfully for a couple of weeks now, but in stripped-down form it has been tested only modestly. Take a look at the last line of badhosts (perhaps using tail /tmp/badhosts) to check the key manifest.googlevideo.com line. On some earlier dd-wrt builds (even as recently as early 2019), /tmp/root/adblocker will not run the file as a shell script as it should. In that case, do sh /tmp/root/adblocker instead.
how i know if i have to use:
Quote:
sh /tmp/root/adblocker
and which lines i should replace in the script ?

also i have a doubt about which is the best, stripped-down version or should i use the full long version ??

thanks in advance

Laughing

Greetings, RickZ. In the CLI, simply do ls -l /tmp/badhosts* after you are sure more than 90 seconds has passed since boot. If the script didn't run (so that you need the sh approach), you'll get a message that there are no such files. (It's also possible that you'll have dnsmasq trouble, since you've told it to look for badhosts. I'm not sure on that point.). But if ls lists /tmp/badhosts and /tmp/badhosts.log, then certainly the script ran and you don't need the sh.

In the latter case do cat /tmp/badhosts.log to have a look at the script's download log. The curl command curl in the script should have written three lines (at least in my version of dd-wrt) to this file, two of header labels and one line showing how the download went. On the left look for 100% of the file to have been downloaded. If that looks good, for a bit more confidence building, do wc -l /tmp/badhosts to see how many lines (one per domain blocked) are in the badhosts[/b] file. Should be thousands I believe, for that particular file. Finally do tail /tmp/badhosts to see the last ten lines of the [i]badhosts file. All the lines except the last should look like "0.0.0.0 " followed by a domain name, and the last line should be some nonzero IP address followed by manifest.googlevideo.com.

Those are the basic checks. Since as presented, that super-abbreviated form of the script does not download the thousands of youtube-specific domains that people identified early in this thread, it may or may not do anything useful for youtube blocking. But it should certainly block a lot of other ad-related nonsense. Try nslookup (I'm not sure of the precise Windows syntax, but it's easy to google) of one of the sites you saw zeros for in the tail... command above to verify that it gives you a 0.0.0.0 IP address. (I don't know that I'd try visiting one in a browser, because some malware sites can abuse a simple visit. No problem if the blocking works, right?)

If the files are not present, before you look at the sh option, try increasing the 90 second number at the end to 300 seconds, reboot, and wait 5 min before checking things out. That'd be just to be sure the problem was not a timing issue.

I haven't waded through the details of your edited version of the script to see how it differs from what I posted, so I can't comment on that. I'm not sure what you are asking when you say "which lines i should replace in the script?" Replace with what goal? As to which version to use. Try this simple version, see if it works for what you want. If youtube is still a problem, I believe with the WRT1900ACv2 you'll be good to try the full version (as that router is almost identical to mine), though be sure you have at least 4K (more would be safer) of free nvram space (see the status page in the GUI) before you try installing it in the Startup section. Really, really bad things happen when you run out of nvram space. I like the long version of the script for the exception lists but also for the greater diagnostics: a [b]badhosts.codes[b] file, a couple of syslog entries, etc. And it is easier to add or delete download sites in that version, so you could cut it down a bit and use only the sbc.io site of your version above plus the youtube-related sites, if you still want to tilt towards the minimal. It'd be easy enough to zero out the whitelists and blacklist and keep only the manifest.google.com entry in the other list as well.

_________________
Six Linksys WRT1900ACSv2 (39144/40009/41954):
VLANs, multiple VAPs, NAS, QoS, client-mode travel router, OpenVPN client/PBR (AirVPN), two DNSCrypt servers (incl Quad9) routed through vpn.
Goto page Previous  1, 2, 3, 4 Display posts from previous:    Page 4 of 4
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Advanced Networking All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum