Posted: Thu Jan 02, 2020 15:45 Post subject: [Solution] Blocking all Youtube Video Advertisements
Hi Everyone,
I was working on blocking all Youtube video ADs. I guess I found a solution and I would like to share with DDWRT community. For my tests my solution perfectly blocks all video ads on Youtube doesn't matter Mobile, Android TV or Web Browser.
Youtube pushes ADs thru thousands of hostnames. I blocked all of them via Unbound DNS. Official Youtube app shows yellow AD fractions on timeline of the video but video will never display. Here is the step by step tutorial.
Requirments:
* DDWRT router with Unbound (Recursive DNS) feature (You can check SETUP screen. Make sure you will see and checked "Recursive DNS Resolving (Unbound)" feature on DDWRT). I use DD-WRT v3.0-r38060 std 12/20/18 firmware and WRT1900AC v1 router.
* PHP Hosting
There are several Youtube ADs hostnames lists on the internet. My code merges two of them. One dynamic (https://api.hackertarget.com/hostsearch/?q=googlevideo.com) other one static list. By the way JFFS is my external drive:
/tmp/mnt/sda1/Backups/jffs/unbound/named.cache
local-data: "r1---sn-25ge7n76.googlevideo.com A 127.0.0.1"
local-data: "r1.sn-25ge7n76.googlevideo.com A 127.0.0.1"
local-data: "r2---sn-25ge7n76.googlevideo.com A 127.0.0.1"
local-data: "r2.sn-25ge7n76.googlevideo.com A 127.0.0.1"
local-data: "r3---sn-25ge7n76.googlevideo.com A 127.0.0.1"
local-data: "r3.sn-25ge7n76.googlevideo.com A 127.0.0.1"
local-data: "r4---sn-25ge7n76.googlevideo.com A 127.0.0.1"
local-data: "r4.sn-25ge7n76.googlevideo.com A 127.0.0.1"
local-data: "r5---sn-25ge7n76.googlevideo.com A 127.0.0.1"
Above code stops Unbound service and starts with new configuration. You can change config file location accordingly.
STEP THREE: ADD FOLLOWING UNBOUND CONFIG FILE TO YOUR EXTERNAL DRIVE JFFS OR SOMEWHERE IN YOUR SYSTEM
Your local hosts and settings should be different than mine. So the best way just copy the original /tmp/unbound.conf file and add lines that I indicated below:
server:
verbosity: 1
interface: 0.0.0.0
interface: ::0
outgoing-num-tcp: 10
incoming-num-tcp: 10
msg-buffer-size: 8192
msg-cache-size: 1m
num-queries-per-thread: 30
rrset-cache-size: 2m
infra-cache-numhosts: 200
username: ""
pidfile: "/var/run/unbound.pid"
root-hints: "/tmp/mnt/sda1/Backups/jffs/unbound/named.cache" #Download updated named.cache and relocate accordingly your system Download from here https://www.internic.net/domain/named.root
target-fetch-policy: "2 1 0 0 0 0"
harden-short-bufsize: yes
harden-large-queries: yes
auto-trust-anchor-file: "/etc/unbound/root.key"
key-cache-size: 100k
neg-cache-size: 10k
num-threads: 2
so-reuseport: yes
msg-cache-slabs: 2
rrset-cache-slabs: 2
infra-cache-slabs: 2
key-cache-slabs: 2
outgoing-range: 462
access-control: 127.0.0.0/8 allow
access-control: 192.168.20.1/24 allow
access-control: 192.168.21.1/24 allow
local-data: "localhost A 127.0.0.1"
local-data: "WRT1900AC A 192.168.20.1"
local-data: "WRT1900AC.local A 192.168.20.1"
local-data: "VPN-SERVER.local A 192.168.20.2"
local-data: "WDMYCLOUD.local A 192.168.20.3"
local-data: "ASUS-N550JK.local A 192.168.20.100"
local-data: "COMPANY.local A 192.168.20.101"
local-data: "PLAYSTATION-3.local A 192.168.20.104"
local-data: "CANON-MG6450.local A 192.168.20.105"
local-data: "SOFTETHER_VPN.local A 192.168.20.106"
local-data: "XIAOMI-MIBOX.local A 192.168.20.107"
local-data: "EP-3703.local A 192.168.20.108"
local-data: "LGWEBOSTV.local A 192.168.20.109"
# ADD FOLLOWING LINES TO YOUR ORIGINAL UNBOUND.CONF FILE
# Blocking Ad Server domains. Google's AdSense, DoubleClick and Yahoo
# account for a 70 percent share of all advertising traffic. Block them.
local-zone: "doubleclick.net" redirect
local-data: "doubleclick.net A 127.0.0.1"
local-zone: "ads.youtube.com" redirect
local-data: "ads.youtube.com A 127.0.0.1"
local-zone: "adserver.yahoo.com" redirect
local-data: "adserver.yahoo.com A 127.0.0.1"
local-zone: "manifest.googlevideo.com" redirect
local-data: "manifest.googlevideo.com A 172.217.19.238"
local-data-ptr: "172.217.19.238 manifest.googlevideo.com"
local-zone: "1e100.net" redirect
local-data: "1e100.net A 127.0.0.1"
include: "/tmp/mnt/sda1/Backups/jffs/unbound/youtube.conf" #Relocate accordingly your system.
python:
remote-control:
STEP FOUR: DOWNLOAD UPDATED YOUTUBE.CONF FILE
Add custom script to your router. This code pulls regenerated Youtube Ad block rules from PHP file.
Posted: Thu Jan 02, 2020 18:56 Post subject: [Solution] Blocking all Youtube Video Advertisements thread
Hi there mkaand!
Thank you very much for sharing this.
I have tried this and it actually works.
Actually I added the hosts you mention in "Additional Dnsmasq Options" in form of:
P.S Why are there "duplicates" in form of:
local-data: "r1---sn-25ge7n76.googlevideo.com A 127.0.0.1"
etc?
Are you aware why the "---" addresses exist? _________________ Netgear R7800 - Firmware: DD-WRT v3.0-r41811 std (12/28/19)
TP-Link AC1750 as Repeater- Firmware: DD-WRT v3.0 r44187 std (08/13/2020)
OpenVPN PBR + Privoxy = IP EXPOSURE:https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=322517&start=0
Thank you very much for nice comments. You are free to improve or fork my work. It is open license. You an create dnsmasq version or you can modify for other systems like PiHole. _________________ Kaan's World | @mkaand | PLEX Archive | Trakt.tv
Posted: Fri Jan 03, 2020 0:01 Post subject: Re: [Solution] Blocking all Youtube Video Advertisements thr
t81 wrote:
P.S Why are there "duplicates" in form of:
local-data: "r1---sn-25ge7n76.googlevideo.com A 127.0.0.1"
etc?
Are you aware why the "---" addresses exist?
I compared two lists via online text compare tools but They couldn't find any common lines. I don't know meaning of --- but I prefer to keep AS IS. Because they are real subdomains. _________________ Kaan's World | @mkaand | PLEX Archive | Trakt.tv
Joined: 04 Aug 2018 Posts: 1447 Location: Appalachian mountains, USA
Posted: Fri Jan 03, 2020 2:53 Post subject:
mkaand wrote:
There are several Youtube ADs hostnames lists on the internet. My code merges two of them. One dynamic (https://api.hackertarget.com/hostsearch/?q=googlevideo.com) other one static list.
Very nice to see this! I am looking forward to trying it out.
I see only 500 lines in the dynamic list you name though, so presumably the rest of those thousands are in the fixed list. What is the URL for the fixed list? I can't find it in your materials! _________________ 2x Netgear XR500 and 3x Linksys WRT1900ACSv2 on 53544: VLANs, VAPs, NAS, station mode, OpenVPN client (AirVPN), wireguard server (AirVPN port forward) and clients (AzireVPN, AirVPN, private), 3 DNSCrypt providers via VPN.
@SurprisedItWorks Thank you very much and reminder
I updated my youtube.php credits. Now you can easily see sources of these lists.
Code:
/*
YOUTUBE ADS VIDEO HOST BLOCK RULE GENERATOR v1.0
Created by CryptoYakari
02.01.2020
https://twitter.com/CryptoYakari
This PHP script generates Unbound blocking rules for Youtube Advertisement Hosts
Static list https://raw.githubusercontent.com/anudeepND/youtubeadsblacklist/master/domainlist.txt
Dynamic list https://api.hackertarget.com/hostsearch/?q=googlevideo.com
*/
I simplified my youtube.php rule generator script. Here is the final code:
Code:
<?
/*
YOUTUBE ADS VIDEO HOST BLOCK RULE GENERATOR v1.0
Created by CryptoYakari
02.01.2020
https://twitter.com/CryptoYakari
This PHP script generates Unbound blocking rules for Youtube Advertisement Hosts
Static list https://raw.githubusercontent.com/anudeepND/youtubeadsblacklist/master/domainlist.txt
Static list Mirror https://raw.githubusercontent.com/mkaand/youtube-ads-blocker-ddwrt/master/domainlist.txt
Dynamic list https://api.hackertarget.com/hostsearch/?q=googlevideo.com
*/
Posted: Fri Jan 03, 2020 6:50 Post subject: Re: [Solution] Blocking all Youtube Video Advertisements thr
mkaand wrote:
t81 wrote:
P.S Why are there "duplicates" in form of:
local-data: "r1---sn-25ge7n76.googlevideo.com A 127.0.0.1"
etc?
Are you aware why the "---" addresses exist?
I compared two lists via online text compare tools but They couldn't find any common lines. I don't know meaning of --- but I prefer to keep AS IS. Because they are real subdomains.
Ok it seems that these "---" addresses do not resolve to anything.
Try pinging or tracert to any of them.
I assume they are safe to delete. _________________ Netgear R7800 - Firmware: DD-WRT v3.0-r41811 std (12/28/19)
TP-Link AC1750 as Repeater- Firmware: DD-WRT v3.0 r44187 std (08/13/2020)
OpenVPN PBR + Privoxy = IP EXPOSURE:https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=322517&start=0
Thank you very much for nice comments. You are free to improve or fork my work. It is open license. You an create dnsmasq version or you can modify for other systems like PiHole.
Thank you very much for nice comments. You are free to improve or fork my work. It is open license. You an create dnsmasq version or you can modify for other systems like PiHole.
Excellent work. I updated my youtube.php again. It seems you are right. --- hostnames are invalid. My new code ignores them. Please use latest version on your fork. Thanks.
Code:
<?
/*
YOUTUBE ADS VIDEO HOST BLOCK RULE GENERATOR v1.0
Created by CryptoYakari
02.01.2020
https://twitter.com/CryptoYakari
This PHP script generates Unbound blocking rules for Youtube Advertisement Hosts
Static list https://raw.githubusercontent.com/anudeepND/youtubeadsblacklist/master/domainlist.txt
Static list Mirror https://raw.githubusercontent.com/mkaand/youtube-ads-blocker-ddwrt/master/domainlist.txt
Dynamic list https://api.hackertarget.com/hostsearch/?q=googlevideo.com
*/
Joined: 04 Aug 2018 Posts: 1447 Location: Appalachian mountains, USA
Posted: Fri Jan 03, 2020 20:14 Post subject:
I have your URLs up and working in a slightly modified version of the adblocker linked in my sig. Soon I'll modify that post to include these new youtube-blocking features (and cite the present thread). Because the blocking machinery is there in my existing script already, adding these URLs was pretty simple. Adding the URLs themselves was four lines, two for the downloading and two for a bit of postprocessing with awk (for one URL) and sed (for the other) to get them into the right form. Keeping that postprocessing from causing problems elsewhere required require minor modification of two of the existing script lines as well, so it was a six-line effort overall. Still I spent a few hours on it as something to do while down with a cold. The foggy brain from the cold made it slow, but also a good part of it was experimenting with some scripting features I was not up on to figure out how things worked in our shell. Good educational experience then!
Now the catch: While I can see with nslookup that I am mapping these youtube ad domains to 0.0.0.0 as I intended, I find when I start a youtube video, sometimes I see an ad and sometimes I don't. It seems not far from half and half. I don't know whether youtube is just being "kind" and not showing me ads more than N times per minute as I hop from video to video or whether perhaps they are using different ad domains in different parts of the world. I'm in the eastern US, and mkaand, your posts show you as being in İstanbul, so maybe very different internet/googly contexts. Or maybe the list is just incomplete so that whether I see an ad simply depends on whether it streams from a site we block or one we don't. I have also seen claims online -- I no longer know where -- that youtube has moved to embedding ads right in the main video stream, to defeat adblockers. Or maybe they do that in some of their servers but not others, again bringing geography into it. _________________ 2x Netgear XR500 and 3x Linksys WRT1900ACSv2 on 53544: VLANs, VAPs, NAS, station mode, OpenVPN client (AirVPN), wireguard server (AirVPN port forward) and clients (AzireVPN, AirVPN, private), 3 DNSCrypt providers via VPN.
Thank you very much for sharing your results. For my tests I can see yellow AD fractions on timeline bar but ADs never show up. I have %100 rate. But I want to indicate some important points for you:
1. I use Unbound (Recursive DNS) not dnsmasq. They looks like doing same thing but I guess Unbound better than dnsmasq.
2. I want to highlight some important settings on my unbound.conf:
local-zone: "manifest.googlevideo.com" redirect
local-data: "manifest.googlevideo.com A 172.217.19.238"
local-data-ptr: "172.217.19.238 manifest.googlevideo.com"
local-zone: "1e100.net" redirect
local-data: "1e100.net A 127.0.0.1"
First three lines: I found 172.217.19.238 IP when i ping manifest.googlevideo.com on my remote server (It is in Paris) If I ping this at my home I see different IP. First three lines are very important. Please read this:
Joined: 04 Aug 2018 Posts: 1447 Location: Appalachian mountains, USA
Posted: Sun Jan 05, 2020 21:35 Post subject:
Hi Kaan! Thanks for taking the time to educate me a bit more here. Your comments here got me going down quite a rabbit hole.
mkaand wrote:
1. I use Unbound (Recursive DNS) not dnsmasq. They looks like doing same thing but I guess Unbound better than dnsmasq.
Yes, it looks like Unbound is much more sophisticated. But I'm trying to stay with dnsmasq for now and not get into anything requiring persistent storage. Also at this point I have a lot of things going on in dnsmasq and am in no hurry to start over!
Quote:
2. I want to highlight some important settings on my unbound.conf:
local-zone: "manifest.googlevideo.com" redirect
local-data: "manifest.googlevideo.com A 172.217.19.238"
local-data-ptr: "172.217.19.238 manifest.googlevideo.com"
local-zone: "1e100.net" redirect
local-data: "1e100.net A 127.0.0.1"
First three lines: I found 172.217.19.238 IP when i ping manifest.googlevideo.com on my remote server (It is in Paris) If I ping this at my home I see different IP. First three lines are very important. Please read this:
Yes, that site is excellent, though reading comments is important, as there's an key error early on, and the comments make some things clearer as well.
Quote:
Last two lines blocks all *.1e100.net via unbound. Blocking youtube ads doesn't mean only blocks certain hosts. I added also PTR record.
3. For your test you have to clear the client dns caches also.
The best test conditions I recommend use Unbound and use same settings that I use. I watch youtube videos without any ad.
You can see my sample youtube.php output here. I ignored --- included hostnames.
One preliminary note: This all assumes a pure IPv4 system. I disable IPv6 in dd-wrt, as at this point I have way too many customizations I'd need to untangle to ever go back and try to deal with IPv6.
My current dnsmasq experiment then is to see how much machinery is really required to eliminate the youtube ads. Right now I'm testing what looks for now like the simplest possible approach. I ignore the fixed and dynamic lists and even the *1e100.net material (though my testing environment already eliminates gx-in-f109.1e100.net, mu-in-f167.1e100.net, and yx-in-f108.1e100.net, it turns out) and focus on manifest.googlevideos.com, since that seems for me to be the part of all this that reaaaaally matters. Since I've tried this experiment, I have not seen a youtube ad. My sample size is small though, and it does seem that location matters, so I'll be curious what others experience.
The weird thing is that all that is done by the one line added to the router DNS setup by this simple approach is to assign to manifest.googlevideos.com the very same IP address, call it A, that nslookup reveals that it already has in the public DNS system when the latter is queried from your location at a given time. But as @mkaand shows us above, without this line, you can then do a reverse lookup on A, and the public DNS system will return one or more other domains, with particulars depending on your region and maybe the phase of the moon. Each such domain has a 3-letter airport code in it. It seems like these weird domain names are some sort of code needed by the youtube ad system. With this line added, the reverse lookup returns only manifest.googlevideo.com, and this stifles the ad machinery.
Now it is very clear after experimenting that nslookup manifest.googlevideo.com returns different IP addresses for users in different locations. I know this thanks both to a vpn and to the comments at the site in the quote above. I am not sure but certainly suspect it changes also at different times. So what I am doing is putting the nslookup right in the dd-wrt startup code. I reboot on schedule every night, so I'll always have a reasonably current result this way. The catch is that in dd-wrt I cannot find any alternative to nslookup, and I cannot find a way to have nslookup return just the IP address and nothing else. So I had to kludge and use sed to parse the nslookup output to extract the IP address. That's fine for now, but what if the output format changes in a future build? What if it has already changed, since I am working with build 40009, the linux kernel in which has already been superceded in recent dd-wrt builds? So if you try this, be sure to either look at the system log entry this creates or or go into the CLI and look at file youtubeAdsHost (in your home directory ~, aka /root or /tmp/root). Either way, verify that the format is an IP address, a space, then manifest.google.com.
To try this experiment (if you are a dnsmasq user) then, include a line addn-hosts=/tmp/root/youtubeAdsHost in Additional Dnsmasq Options in GUI>Services>Services and then in Startup in GUI>Administration>Commands, include this little code block (or for a preliminary test, paste it into the CLI):
There is a minimum 30 second delay before anything happens, and if needed, repeated two-minute delays then follow until networking and the DNS system are operational, as detected by being able to ping cloudlflare.com. Then nslookup is run, and you'll see that file youtubeAdsHost.exitCode then contains a 0 if it exited normally. The new contents of file youtubeAdsHost, which should be one line in /etc/hosts format, are logged in the system log for your amusement, and finally dnsmasq is reinitialized. It's all run in the background to not slow other activity, and it's isolated in a subshell so as not to be setting shell variables that may affect other things.
Zero-length file StartedYoutubeAdBlocker is there only to record that this code has been run so that if dd-wrt startup code is rerun on an Apply, this code block will not be run a second time. If this is a feature you'd rather do without, try replacing exit in the second line with echo "rerunning youtube ad blocker at $(date)" >>/tmp/root/YoutubeAdBlockerRunLog so that you'll have a record of any rerunning of the code. In that case, file youtubeAdsHost.exitCode will contain an nslookup exit code -- we hope only a zero -- for each run.
I tested mostly on an Android TV, which I had to reboot to clear its DNS cache before testing, but a little on an iPhone also, and I have yet to see a youtube ad.
Let me know if there are issues trying this or if you spot bugs, as the above is actually a stripped-down version of what I use here, and I haven't actually tried the precise version I'm sharing here. My own version does other ad blocking (besides youtube) as well. _________________ 2x Netgear XR500 and 3x Linksys WRT1900ACSv2 on 53544: VLANs, VAPs, NAS, station mode, OpenVPN client (AirVPN), wireguard server (AirVPN port forward) and clients (AzireVPN, AirVPN, private), 3 DNSCrypt providers via VPN.
This is excellent job. Good to hear you have yet to see youtube ad. I hope this study will help everyone. If they change their system in the future we can figure out again. My current setup works perfectly. No Youtube ADs, clear videos. I enjoy AD-free Youtube _________________ Kaan's World | @mkaand | PLEX Archive | Trakt.tv
Joined: 04 Aug 2018 Posts: 1447 Location: Appalachian mountains, USA
Posted: Mon Jan 06, 2020 15:39 Post subject:
Update: yesterday after installing my one-line youtube ad blocker above, I streamed dozens of youtube videos over the rest of the day and never saw an ad. This morning, however, after my router's overnight reboot and so redoing of my one line, the first video I tried, on my phone, came with an ad, though others tried immediately thereafter did not. So I checked what Quad9 DNS was returning for manifest.googlevideo.com, at least when queried from today's vpn server location, and discovered that it was different from the IP used at boot time to set up my one line. So I hacked in the change by hand to see what would happen. No ads so far, but we'll see how long that lasts.
More interesting perhaps is that I repeated the DNS query for manifest.google.com several times and found that I was often getting different IP addresses when the query was repeated, even when queries were separated by only seconds. A little looking at the TTL field -- certainly should have done that earlier! -- was more confusing than enlightening. Once I saw 300s in that field, repeated the query to 9.9.9.9 a few seconds later, and saw the same four IPs but now with a TTL of 6s. Different upstream servers?
And yes, the IPs. Most of the time I was getting four IP's, not one, and their reverse lookups shared the airport designator but with different XX numbers in the fXX component of the names. The little script I posted should pick the first one IP listed (in the forward lookup), but I have no idea what the significance is of their being four, and I have no clue what the rapid changes mean for us. No wonder others who have gone down this rabbit hole have come up looking drained!
Watch this space... _________________ 2x Netgear XR500 and 3x Linksys WRT1900ACSv2 on 53544: VLANs, VAPs, NAS, station mode, OpenVPN client (AirVPN), wireguard server (AirVPN port forward) and clients (AzireVPN, AirVPN, private), 3 DNSCrypt providers via VPN.
