Posted: Thu May 25, 2023 11:28 Post subject: Adblock with SmartDNS
Hello, I noticed that many people were having success with DNS-level adblocking, so I thought I'd give it a try. So far I've had it running for about a week without issue and it seems to work pretty well. Here's how I did it in case anybody else might want to do the same thing...
I'm using SmartDNS for this. It's attractive because it is very fast, caches and can automatically renew DNS results, tests for the fastest option when there are multiple results for a site, and uses DNS over TLS (DoT) (which is now supported by all major DNS services). I use it in a custom configuration, with one instance per openvpn client + for direct wan access (because each path to the internet has different fastest IP addresses), and fallback to dnsmasq for local addresses. But it should work perfectly well with the standard setup via the GUI if you simply enable smartdns. You will need a recent router with enough memory and reasonably fast processor. I'm using it on a Netgear R7800 and it runs very fast. With a basic adblock list of ~180,000 entries, it uses 13.1% of memory (62MB) vs 7.8% (37MB) without adblocking for the configuration I'm using (cache-size 2000). I would recommend you use an attached USB drive or USB key for additional storage, although it should be possible to use JFFS. You will need a minimum of about 15MB free to use a single adblock blacklist. SmartDNS is very fast at filtering urls, and there is more configuration info here
I put all my blocklist stuff is in /mnt/sda1/blocklists. Modify as desired in the below.
To setup:
1) Put the following script in /mnt/sda1/blocklists/fetch_blocklists.sh (or /jffs/bin, as you prefer):
Code:
#!/bin/sh
# get StevenBlack blocklists and convert them to the format used by smartdns
# See https://github.com/StevenBlack/hosts for more info. This is the default list used by Pi-hole
# requires md5sum awk sed & curl
cd /mnt/sda1/blocklists
restart=0
# take a .hosts file with the given base filename and convert it to a smartdns .conf format
convert_list () {
# remove entries before start (localhost etc), remove comments and blank lines, convert to "address /domain.to.ignore/#"
cat $1.hosts |
sed -n '/\# Start StevenBlack/,$p' |
sed '/^\(\#.*\|\s*\)$/d' |
awk '{print "address /"$2"/#"}' > $1.conf
}
# fetch .hosts file from URL $1, save it to $2.hosts, convert it to $2.conf
fetch_and_convert () {
local md5=$(md5sum $2.hosts)
echo -n "Downloading list: $2... "
curl -sSLo $2.hosts $1
if [ $(wc -c <"$2.hosts") -lt 1000000 ]; then
# file too small: likely contains html page showing error or did not download at all
echo " download apparently failed. ABORTING!"
exit
fi
if [ "$(md5sum $2.hosts)" = "$md5" ]; then
echo " $2 unchanged."
else
echo -n " converting... "
restart=1
convert_list $2
echo "done."
fi
}
if [ $restart = 1 ]; then
echo -n "Restarting smartdns... "
stopservice smartdns
sleep 1
startservice smartdns
echo "done."
fi
This code is set up to use a blocklist from Steven Black's Unified hosts files. These are the default lists used by Pi-Hole and seem to work reliably (you probably don't want overly aggressive lists because you'll have to manually override them to get sites working again, and that's a pain because there is no GUI for this. So far I haven't had to override anything.). You can choose from a variety of options at that website.
Towards the bottom of the script is:
Code:
fetch_and_convert [URL] [Base filename]
1b) Here you can choose which blocklist(s) to download and what to call it (them) on your router. (The script above uses a CDN URL that mirrors the github for the "Unified hosts" (most basic) blocklist. Any line starting with '#' is ignored.) Choose the link for "Raw hosts" for the one you want. Copy that into the 'fetch_and_convert' line and change the name as desired.
Note that each entry in the generated blocklist blocks the domain and any subdomains (so 'address /yahoo.com/#' would block mail.yahoo.com as well as yahoo.com, for both IPv4 and IPv6).
1c) Run 'chmod +x /mnt/sda1/blocklists/fetch_blocklists.sh' to make the script executable. Then run it for the first time: '/mnt/sda1/blocklists/fetch_blocklists.sh'. You should see something like:
Code:
--- Mon May 22 05:20:00 CEST 2023 Fetching Blocklists ---
Downloading list: malware+ads... converting... done.
Restarting smartdns... done.
2) Enable smartdns under services. Put the following in 'Additional Options':
This will fetch, convert, and install new block lists each week at 5:20am on monday morning.
All set. After the next reboot, lists will be updated automatically. You will see log entries at /mnt/sda1/blocklists/fetch.log that show how updating went.
If for some reason you are unable to access something you want, or a site is not working because it can't access a blocked resources, add it to the whitelist.conf file and restart smartdns. That should fix it.
P.S. It is possible to use any hosts-formatted adblock list with this script. You'll just need to remove the line "sed -n '/\# Start StevenBlack/,$p' | " from the script. That is there to skip over everything before the line "# Start StevebBlack", because in his lists there are some localhost entries that would otherwise be inadvertantly blocked.
P.P.S. If you want to use several blocklists, it's no problem. It is possible to fetch multiple by simply having multiple 'fetch_and_convert' lines with different names and URLs. (This would make no sense with StevenBlack lists since they are all overlapping, unless it were for seperate smartdns instances. But you might want blocklists from another source). Add them all to the SmartDNS options BEFORE the whitelist.
P.P.P.S If you want to debug anything with smartdns, you might want to enable logging. To do so, you'll need to get the 'opkg' version of smartdns, because the supplied version (silently) excludes logging functionality.
Finally, you might consider a simpler option: controld.com, alternate-dns.com, cleanbrowsing.org, opendns.com and others provide DNS servers that automatically block different classes of websites. These will be slower however, and a bit trickier to override when some site you want doesn't work.
Last edited by jtbr on Wed Jun 07, 2023 22:35; edited 2 times in total
Joined: 16 Nov 2015 Posts: 6435 Location: UK, London, just across the river..
Posted: Thu May 25, 2023 20:38 Post subject:
im not sure if this script is working at all, have you tried it...
fetch command does not exist on DDWRT... _________________ Atheros
TP-Link WR740Nv1 ---DD-WRT 55630 WAP
TP-Link WR1043NDv2 -DD-WRT 55723 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55779 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55779 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55779 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
I wrote it, and it does work for me. Let me know if you're having problems and I can help. But the fetch_and_convert function is defined within the script provided.
Thank you for that introduction to SmartDNS.
And luckily for me, just at the moment when I started looking at it.
I used your work as inspiration to do it in my own way, and it works like a charm (yes, it's magic).
So far, 201572 URL merged and blocked from 5 sources
That way, you download the file only if it's new.
Then I merge them all with all the grep en sed necessary, and finish with sort and uniq to shrink it.
Edit :
I also used this article to make the proper UI configuration. _________________ Netgear R7000P - r52720 : gateway behind ISP / DHCP server / VPN server / AdBlocking.
I'm an "advanced user", not an "admin expert", so I like when things work magically (I'm a Spring Boot developer, after all...), but I like to know a bit about the "how" (the "server=127.0.0.1#6053", actually, was what I needed to know).
That PDF solved my problem about USB booting, because I didn't understand why I had to restart SmartDNS.
Thanks, guys, for your precious help.
Also, I've read somewhere that "-host-name: cloudflare-dns.com -tls-host-verify: cloudflare-dns.com" is useless (SmartDNS apparently don't interpret it), and a lot of examples don't use it.
As I have no clew what it could be useful about and at worst it is not read, I left it anyway. _________________ Netgear R7000P - r52720 : gateway behind ISP / DHCP server / VPN server / AdBlocking.
Joined: 16 Nov 2015 Posts: 6435 Location: UK, London, just across the river..
Posted: Wed May 31, 2023 12:31 Post subject:
Maze_fr wrote:
I'm an "advanced user", not an "admin expert", so I like when things work magically (I'm a Spring Boot developer, after all...), but I like to know a bit about the "how" (the "server=127.0.0.1#6053", actually, was what I needed to know).
That PDF solved my problem about USB booting, because I didn't understand why I had to restart SmartDNS.
Thanks, guys, for your precious help.
Also, I've read somewhere that "-host-name: cloudflare-dns.com -tls-host-verify: cloudflare-dns.com" is useless (SmartDNS apparently don't interpret it), and a lot of examples don't use it.
As I have no clew what it could be useful about and at worst it is not read, I left it anyway.
-server=127.0.0.1#6053 - this is the default port for SmartDNS (6053)- and server= is the command that tells DNSmasq what server to use...in this case 127.0.0.1 is the loopback interface, you can change it to any other port..via smartdns.config box or via jffs config and than point it to 127.0.0.1#any other port in DNSmasq...
DNSmasq can work as a stub resolver, so the normal requests via port 53 tcp&udp will be unreplyed and will use 127.0.0.1 to forward the DNS requests directly to the DNS resolving server, so you can see 9.9.9.9 connected to port 443 or 853 directly...
- restart SmartDNS is not needed by default(im not doing it), but it depends from your scenario..
--tls-host-verify: cloudflare-dns.com is not used and i haven't saw it recommended neither, i just pasted it and deleted it(on the recap)..it will not hurt even if you leave it...i believe it could be a bit that some DNS resolvers support as an extra, like cloudflare for example... _________________ Atheros
TP-Link WR740Nv1 ---DD-WRT 55630 WAP
TP-Link WR1043NDv2 -DD-WRT 55723 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55779 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55779 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55779 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
Thank you Alozaros, but I wasn't asking for explanation about "server=127.0.0.1#6053", I understood it.
What I meant is that I didn't know about it, so I didn't know that DNSmasq was already connecting to SmartDNS. I admit I should have guessed so, because DD-WRT is really well thought about integration of its components to work all together.
The PDF has a very clear explanation about how all that work, and it's enough for my need.
In my scenario, I have "/jffs" on an USB stick, so I need SmartDNS to restart after USB mount on reboot. That I didn't know, and I didn't understand why I lost internet after rebooting.
Note : I installed DD-WRT this weekend, so I'm still discovering... but I love it. _________________ Netgear R7000P - r52720 : gateway behind ISP / DHCP server / VPN server / AdBlocking.
Joined: 16 Nov 2015 Posts: 6435 Location: UK, London, just across the river..
Posted: Wed May 31, 2023 18:35 Post subject:
Maze_fr wrote:
Thank you Alozaros, but I wasn't asking for explanation about "server=127.0.0.1#6053", I understood it.
What I meant is that I didn't know about it, so I didn't know that DNSmasq was already connecting to SmartDNS. I admit I should have guessed so, because DD-WRT is really well thought about integration of its components to work all together.
The PDF has a very clear explanation about how all that work, and it's enough for my need.
In my scenario, I have "/jffs" on an USB stick, so I need SmartDNS to restart after USB mount on reboot. That I didn't know, and I didn't understand why I lost internet after rebooting.
Note : I installed DD-WRT this weekend, so I'm still discovering... but I love it.
Well..yep DDWRT is great and has lots of potential...indeed !
In general as i said, for standard simple use of it you dont need to restart SmartDNS to look for config in jffs...just add your commands to the SmartDNS box in GUI
On boot/reboot in case if SmartDNS service is enabled, firmware will proceed:
-will look for USB jffs mounted
-will look for any SmartDNS config in jffs...(if any) and those will be parsed to its /tmp/smartdns.conf
-than it will proceed with loading its standard config.../tmp/smartdns.conf (if not jffs config is present)
-than it will look at the SmartDNS config box in the GUI for extra commands
-it will be up and running.
So, to make it up and running you dont need to fiddle with it, just use the GUI options and its box in GUI...
If you really need to paste any other more specific configs to SmartDNS, yes you can run jffs configs...but, those can be bound with troubles...
In general, all USB related stuff, needs to be saved in save USB script in GUI..so, it will be executed when USB is up and running/mounted (and those times can vary)...
But in your case, as you want to add stuff related to SmartDNS config and (on boot) when SmartDNS service is started it look at jffs for config..than just paste your config in jffs and hope it will be triggered on time...if not on time, than restart SmartDNS service and it will re-read jffs config and ect.
To be honest i haven't played with SmartDNS extra configs like this adblocker stuff that is around..
instead i use another add-blocker on dnsmasq level...that adds those to hosts..
i my give it a try when I have more spare time.. _________________ Atheros
TP-Link WR740Nv1 ---DD-WRT 55630 WAP
TP-Link WR1043NDv2 -DD-WRT 55723 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55779 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55779 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55779 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
Very interesting commit !
Well... now I know why I had to restart SmartDNS.
It checks if "Use NVRAM for Client Lease DB" is disabled, which, in my case, was enabled in the same time as "Use JFFS2 for Client Lease DB".
I thought "2 is better than only 1"
It shouldn't be 2 check boxes, then. It should be a radio button.
Or maybe testing "Use JFFS2 for Client Lease DB" being enabled instead of "Use NVRAM for Client Lease DB" being disabled.
Anyway, SmartDNS restart is very fast. And it prevent any problem related to conf and USB mount.
Originally, with the lack of documentation in DD-WRT wiki, I planed to install Entware and use DNScrypt or NextDNS or Stubby or Pixelserv-tls.
So SmartDNS saves me a lot of time, and probably also a lot of router performance.
I also considered using Privoxy, but it's not for HTTPS, so I would have needed one of the previous list anyway.
Maybe there should be a part of SmartDNS config in "Services -> Ad Blocking" to help configure it for AdBlocking. _________________ Netgear R7000P - r52720 : gateway behind ISP / DHCP server / VPN server / AdBlocking.
Joined: 16 Nov 2015 Posts: 6435 Location: UK, London, just across the river..
Posted: Thu Jun 01, 2023 15:06 Post subject:
Maze_fr wrote:
Very interesting commit !
Well... now I know why I had to restart SmartDNS.
It checks if "Use NVRAM for Client Lease DB" is disabled, which, in my case, was enabled in the same time as "Use JFFS2 for Client Lease DB".
I thought "2 is better than only 1"
It shouldn't be 2 check boxes, then. It should be a radio button.
Or maybe testing "Use JFFS2 for Client Lease DB" being enabled instead of "Use NVRAM for Client Lease DB" being disabled.
Anyway, SmartDNS restart is very fast. And it prevent any problem related to conf and USB mount.
Originally, with the lack of documentation in DD-WRT wiki, I planed to install Entware and use DNScrypt or NextDNS or Stubby or Pixelserv-tls.
So SmartDNS saves me a lot of time, and probably also a lot of router performance.
I also considered using Privoxy, but it's not for HTTPS, so I would have needed one of the previous list anyway.
Maybe there should be a part of SmartDNS config in "Services -> Ad Blocking" to help configure it for AdBlocking.
about:
-"Use NVRAM for Client Lease DB" is a bad idea..tons of threads on the subject (use search forum)
there is more useful way to save your clients db and paste those in advanced dnsmasq rules anytime
-SmartDNS ad-blocking - https://pymumu.github.io/smartdns/en/config/ad-block/ and consider this, that it is a binary that is not DDWRT stuff and has its own shine, so DDWRT wiki is not due but it will be a bonus..(needs lot of testing and approval)
-you can also create an adblock using IPset via either iptables or using dnsmasq
-privoxy its not the best way no idea why its still around but it does something that could be called ad-blocking
-use of SmartDNS does not surpass the security and options that DNScrypt-proxy v2 deliverers , but its presence is very good indeed, at list surpasses the Stubby.(never tried NextDNS via entware)
The interesting bit is, Openssl is abandoning the support of its libssl 1.1.1.x and concentrates over v3.x only..that needs to be implemented/incorporated or replaced with another ssl soon.. _________________ Atheros
TP-Link WR740Nv1 ---DD-WRT 55630 WAP
TP-Link WR1043NDv2 -DD-WRT 55723 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55779 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55779 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55779 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
My SmartDNS is up and functioning perfectly.
I precisely used the link you mentioned to help.
Such a great pleasure to use smartphone apps without adds...
I just discovered that DNScrypt-proxy is already included in DD-WRT.
As there is nothing about it in the GUI, and... well... the wiki being the wiki...
At my level of expertise, I don't know what DNScrypt-proxy would give me more than SmartDNS is giving me : blocking adds even in HTTPS.
I don't expect DD-WRT wiki to have everything, but a link here and there to external sources would be nice. As the wiki is very outdated, those sources would probably be more relevant.
Also, a link to the wiki in the "more..." help pages of the GUI would be nice too, because the wiki is "complicated" to search in.
A Google Search filtered on the wiki is more convenient, actually... Maybe adding an input to launch directly that kind of restricted search would be helpful in the header. _________________ Netgear R7000P - r52720 : gateway behind ISP / DHCP server / VPN server / AdBlocking.
Joined: 16 Nov 2015 Posts: 6435 Location: UK, London, just across the river..
Posted: Fri Jun 02, 2023 18:22 Post subject:
DNScrypt included in DDWRT is the old v1.95 (still operational)
The new version of DNScrypt-proxy v2.x has much more configurable options and by far is the most secure DNS solution, see the green link in my signature...
As far as the wiki not very updated, so if you need to know something and its not in the wiki, either ask google or search in the forum tons of useful threads....last resort, post a request thread on its belonging section...
When you ask for something always start with router model and current firmware number, provide as many details possible...
See the forum guidelines with helpful pointers about how to research your router, where and what firmware to download, where and how to post and many other helpful tips:
https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=324087 _________________ Atheros
TP-Link WR740Nv1 ---DD-WRT 55630 WAP
TP-Link WR1043NDv2 -DD-WRT 55723 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55779 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55779 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55779 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
That way, you download the file only if it's new.
Then I merge them all with all the grep en sed necessary, and finish with sort and uniq to shrink it.
Interesting, I didn't know curl could do that!
I'm curious about the de-duping you're doing with sort and uniq. I considered doing something like that too, but I guessed that smartdns must already be doing this (although it would save perhaps a bit of startup time and disk space in any case). Have you tried comparing how much memory usage smartdns uses with/without removing potential duplicates?
If it's useful, there is something else that could be done. SmartDNS blocklists block all subdomains, while (iirc) hosts files do not. Thus there are a lot of entries of subdomains that could be removed from the file. (eg: badsite.com is blocked, no need to also block another.badsite.com: smartdns already does that, but they're in the lists anyway).
, but its presence is very good indeed, at list surpasses the Stubby.(never tried NextDNS via entware)