Adblock with SmartDNS

Post new topic   Reply to topic    DD-WRT Forum Index -> Advanced Networking
Goto page Previous  1, 2
Author Message
Maze_fr
DD-WRT Novice


Joined: 26 May 2023
Posts: 8
Location: France

PostPosted: Thu Jun 08, 2023 9:31    Post subject: Reply with quote
jtbr wrote:
Maze_fr wrote:

Personally, I prefer to use something like :
Code:
curl --etag-compare etag.txt --etag-save etag.txt -sfL http://url | some grep stuff >> hosts.list

That way, you download the file only if it's new.
Then I merge them all with all the grep en sed necessary, and finish with sort and uniq to shrink it.

Interesting, I didn't know curl could do that!

Actually, it works perfectly on my desktop, but on the router, it's not the same behavior.
On my desktop, it keeps the files untouched, when on the router it creates empty files.
So I deactivated it (I remove etag files before "curling").

jtbr wrote:
I'm curious about the de-duping you're doing with sort and uniq. I considered doing something like that too, but I guessed that smartdns must already be doing this (although it would save perhaps a bit of startup time and disk space in any case). Have you tried comparing how much memory usage smartdns uses with/without removing potential duplicates?

It shrinks the file a lot because I merge 5 sources. And it doesn't take so much time on the router.
So I won't bother compare the memory, because I don't know how to do it and it would take me time to learn to do it that I can use to try to make that f#*@!ing L2TP/IPsec VPN SoftEther server work.

jtbr wrote:
If it's useful, there is something else that could be done. SmartDNS blocklists block all subdomains, while (iirc) hosts files do not. Thus there are a lot of entries of subdomains that could be removed from the file. (eg: badsite.com is blocked, no need to also block another.badsite.com: smartdns already does that, but they're in the lists anyway).

I disagree on that.
I had some taboola ads still showing, so I checked the file : I had subdomains but not the main domain, and my ads were from a subdomain that wasn't in the list.
Actually, I made a "simple" test : I gathered all domains like "^[a-z0-9_-]+\.(co\.)?[a-z]+$", and it was about 30% of the content of the file (taking all main domain isn't a good thing to do, because you have, for example "adservice.google.com" or "device-metrics-us.amazon.com").
The grep to filter those 70% facing the 30% as a filter file is incredibly long (about 7min on my desktop with high CPU usage...), then I add back the domain list, and it shrinks the global list by about 25 %.
I don't think it worses it, as the router doesn't have any performance problem, but that grep would be one.

I still have some ads going through and I don't know why : on the IMDB app, I have "amazon-adsystem.com" ads, but the subdomain and the main domain are in the list.
I wish I know how to log the URLs when I launch the IMDB app (only coming from the MAC of my smartphone), so I could know what isn't blocked exactly.
But if I have ads only when I use the IMDB app, I can live with it...

Note : before I flashed my router, I was using Pi-Hole, and I had a dedicated screen to see visited URLs.
But, well, if I can filter ads on my router instead of on another device, it's more convenient, more reliable, and more efficient. And I can dedicate my RPi4B to IoT.

_________________
Netgear R7000P - r52720 : gateway behind ISP / DHCP server / VPN server / AdBlocking.
Sponsor
jtbr
DD-WRT User


Joined: 09 Mar 2017
Posts: 100

PostPosted: Fri Jun 09, 2023 12:17    Post subject: Reply with quote
Maze_fr wrote:

jtbr wrote:
I'm curious about the de-duping you're doing with sort and uniq. I considered doing something like that too, but I guessed that smartdns must already be doing this (although it would save perhaps a bit of startup time and disk space in any case). Have you tried comparing how much memory usage smartdns uses with/without removing potential duplicates?

It shrinks the file a lot because I merge 5 sources. And it doesn't take so much time on the router.
So I won't bother compare the memory, because I don't know how to do it and it would take me time to learn to do it that I can use to try to make that f#*@!ing L2TP/IPsec VPN SoftEther server work.


If you just type 'top' from the shell, it will show you the memory usage for smartdns; you could compare with/without uniq. Still sounds helpful in your case due to the 5 independent lists.

Maze_fr wrote:

jtbr wrote:
If it's useful, there is something else that could be done. SmartDNS blocklists block all subdomains, while (iirc) hosts files do not. Thus there are a lot of entries of subdomains that could be removed from the file. (eg: badsite.com is blocked, no need to also block another.badsite.com: smartdns already does that, but they're in the lists anyway).

I disagree on that.
I had some taboola ads still showing, so I checked the file : I had subdomains but not the main domain, and my ads were from a subdomain that wasn't in the list.


Right, the behavior if you only have another.badsite.com on the list should be that s1.another.badsite.com is blocked, but www.badsite.com and badsite.com are not blocked.

Maze_fr wrote:
Actually, I made a "simple" test : I gathered all domains like "^[a-z0-9_-]+\.(co\.)?[a-z]+$", and it was about 30% of the content of the file (taking all main domain isn't a good thing to do, because you have, for example "adservice.google.com" or "device-metrics-us.amazon.com").
The grep to filter those 70% facing the 30% as a filter file is incredibly long (about 7min on my desktop with high CPU usage...), then I add back the domain list, and it shrinks the global list by about 25 %.
I don't think it worses it, as the router doesn't have any performance problem, but that grep would be one.


I think that a grep approach can't work, though it's helpful to get a handle on the scale of the question. It needs to check site-by-site. Like add all the TLDs, then check subdomains and ignore any that are already in the first list, then sub-subdomains, etc.

Maze_fr wrote:
I still have some ads going through and I don't know why : on the IMDB app, I have "amazon-adsystem.com" ads, but the subdomain and the main domain are in the list.
I wish I know how to log the URLs when I launch the IMDB app (only coming from the MAC of my smartphone), so I could know what isn't blocked exactly.
But if I have ads only when I use the IMDB app, I can live with it...


Not sure what's going on here, perhaps it's still cached on the device? Or the app has it hard-coded? If you use the entware version of smartdns (downloaded with 'opkg install smartdns' once entware is installed), you can enable logging and at the right verbosity, see each dns lookup as it happens. Good luck with that one Smile
Maze_fr
DD-WRT Novice


Joined: 26 May 2023
Posts: 8
Location: France

PostPosted: Fri Jun 09, 2023 20:31    Post subject: Reply with quote
jtbr wrote:
Maze_fr wrote:
I still have some ads going through and I don't know why : on the IMDB app, I have "amazon-adsystem.com" ads, but the subdomain and the main domain are in the list.
I wish I know how to log the URLs when I launch the IMDB app (only coming from the MAC of my smartphone), so I could know what isn't blocked exactly.
But if I have ads only when I use the IMDB app, I can live with it...

Not sure what's going on here, perhaps it's still cached on the device? Or the app has it hard-coded? If you use the entware version of smartdns (downloaded with 'opkg install smartdns' once entware is installed), you can enable logging and at the right verbosity, see each dns lookup as it happens. Good luck with that one Smile

Actually, the IMDB app is just like the website (I have the ads too), so I spied with the browser debugger... and uBlock request log...
So I think "https://www.imdb.com/_json/getads/..." this one is guilty.
I guess I need also an URL keyword filter service of some kind.

_________________
Netgear R7000P - r52720 : gateway behind ISP / DHCP server / VPN server / AdBlocking.
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 6410
Location: UK, London, just across the river..

PostPosted: Sat Jun 10, 2023 5:23    Post subject: Reply with quote
Some smart devices have hard coded DNS, that can go around and avoid DNS based ad blocking..
Solution is to use forced DNS or force those devices to use DNS you want via iptables rules..
The problem is, some devices may loose connectivity..but this is very rare...
Also some ads nowadays are blended inside the main stream, so those cannot be blocked that well..

If you use uBlock on PC level you don't really need ad block on router level...but yes for some devices it is useful...to use ad block router level.. lots of threads on the subject..
Its a cats and mouse game... Smile you can also use IPset rules to build adblock too..

_________________
Atheros
TP-Link WR740Nv1 ---DD-WRT 55179 WAP
TP-Link WR1043NDv2 -DD-WRT 55303 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55460 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55460 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55363 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
Maze_fr
DD-WRT Novice


Joined: 26 May 2023
Posts: 8
Location: France

PostPosted: Sat Jun 10, 2023 6:26    Post subject: Reply with quote
The router level, it is precisely for smartphones and tablets.
(Actually, I've read about AdBlocking apps for mobile devices, creating a local VPN server and connection, for filtering the ads)
Also, some websites require to deactivate AdBlocking plugin in order to work, and router level isn't an AdBlocking plugin...

Unfortunately, IMDB "blends the ads in" by using it's own HTTPS API to call Amazon ads.
So whatever I can do with IP / DNS won't work.
I would need an URL keyword filter working on HTTPS, and that would be too much work for too few ads.
Well... if I block almost all ads on my network, it's already a great success.

_________________
Netgear R7000P - r52720 : gateway behind ISP / DHCP server / VPN server / AdBlocking.
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 6410
Location: UK, London, just across the river..

PostPosted: Sun Jun 11, 2023 9:29    Post subject: Reply with quote
well...have you ever checked SmartDNS for DNS leaks... https://dnscheck.tools/#results
i keep noticing 'from time to time' it tends to leak a lot...even if i use VPN..
I guess its down to its querying algorithm..where Stubby or DNScrypt proxy v2.xx never leak...

Speaking of DNScrypt proxy v2.xx it has very advanced ad-blocking / IP blocking, black/white lists, or even time based domain allowance options...

_________________
Atheros
TP-Link WR740Nv1 ---DD-WRT 55179 WAP
TP-Link WR1043NDv2 -DD-WRT 55303 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55460 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55460 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55363 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
jtbr
DD-WRT User


Joined: 09 Mar 2017
Posts: 100

PostPosted: Mon Jun 12, 2023 15:15    Post subject: Reply with quote
Alozaros wrote:
well...have you ever checked SmartDNS for DNS leaks... https://dnscheck.tools/#results
i keep noticing 'from time to time' it tends to leak a lot...even if i use VPN..
I guess its down to its querying algorithm..where Stubby or DNScrypt proxy v2.xx never leak...


I haven't had any problem with DNS leaks; although I don't check it regularly I don't see why it would change. Note that this is part of why I have a smartdns instance for each vpn client bridge (plus the wan). Each one has its own unique DNS servers which can only be accessed via the appropriate outward link (ie, DNS server 9.9.9.9 for the vpn tunnel for one vpn client, 8.8.8.8 directly for the wan). To ensure that the DNS servers for the vpn client bridges are only accessed via the vpn (and not directly via the wan), the openvpn client config contains a routing line like this for each DNS server reserved for that VPN:

Code:
route 9.9.9.9 255.255.255.255 vpn_gateway


In order to have multiple smartDNS instances, you have to re-route requests for the default DNS server to the correct one for the bridge. (This is because there is no way for DNSMasq to tell DHCP clients to use a non-default port for DNS). Ie, direct the bridge with the subnet 192.168.10.* to use the (Smart)DNS server at port 8053:

Code:
#Redirect traffic to different SmartDNS servers depending upon source subnet
#for br1, forward port 53 to port 8053 DNS service (needs both udp & tcp)
iptables -t nat -I PREROUTING -p udp -m udp -s 192.168.10.0/24 -d 192.168.10.1 --dport 53 -j REDIRECT --to-ports 8053
iptables -t nat -I PREROUTING -p tcp -m tcp -s 192.168.10.0/24 -d 192.168.10.1 --dport 53 -j REDIRECT --to-ports 8053
...


You also need to start them all yourself, or dd-wrt will not reset them properly. For me, SmartDNS is not enabled in the GUI, but started in the startup commands.
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 6410
Location: UK, London, just across the river..

PostPosted: Tue Jun 13, 2023 7:30    Post subject: Reply with quote
"For me, SmartDNS is not enabled in the GUI, but started in the startup commands" - Although you can, nowadays you don't need to start it..this way...also there is an option where you can use smart 'Additional Servers Only' which will use exclusively those servers in smart DNS box and nothing else..and you still can use iptables to redirect...clients via iptables rules...
in the SMDNS box you can add any commands like port to use and ect. BS made the use of the command box very versatile...
The leak comes while im using SmartDNS via VPN as for some odd reason some specific dns servers are queried and the fastest comes from my real location (like NextDNS more often) instead of my VPN..

_________________
Atheros
TP-Link WR740Nv1 ---DD-WRT 55179 WAP
TP-Link WR1043NDv2 -DD-WRT 55303 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55460 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55460 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55363 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
jtbr
DD-WRT User


Joined: 09 Mar 2017
Posts: 100

PostPosted: Tue Jun 13, 2023 12:51    Post subject: Reply with quote
Alozaros wrote:

The leak comes while im using SmartDNS via VPN as for some odd reason some specific dns servers are queried and the fastest comes from my real location (like NextDNS more often) instead of my VPN..


If you're in windows, the problem may be that you don't have block-outside-dns in the client settings, or it may be a windows bug (not sure if it still exists or not but wouldn't be surprised): see https://superuser.com/a/1195156/713936
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 6410
Location: UK, London, just across the river..

PostPosted: Tue Jun 13, 2023 18:04    Post subject: Reply with quote
Razz naah im talking all from router side...VPN and SmartDNS both on router side...and using particular DNS servers{(router side) DNS leaks..and im not using always Windows...Im talking various clients..and my windows is configured to use exclusively only router's DNS Wink same as my other clients..it happens in very narrow scenarios..mostly using NextDNS resolver and some particular VPN servers...it happens not with all of them..but mostly using SmartDNS..

If i use Stubby DoT or Dnscrypt (it has some settings for querying types) i don't have DNS leaks...

_________________
Atheros
TP-Link WR740Nv1 ---DD-WRT 55179 WAP
TP-Link WR1043NDv2 -DD-WRT 55303 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55460 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55460 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55363 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
Goto page Previous  1, 2 Display posts from previous:    Page 2 of 2
Post new topic   Reply to topic    DD-WRT Forum Index -> Advanced Networking All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum