That way, you download the file only if it's new.
Then I merge them all with all the grep en sed necessary, and finish with sort and uniq to shrink it.
Interesting, I didn't know curl could do that!
Actually, it works perfectly on my desktop, but on the router, it's not the same behavior.
On my desktop, it keeps the files untouched, when on the router it creates empty files.
So I deactivated it (I remove etag files before "curling").
jtbr wrote:
I'm curious about the de-duping you're doing with sort and uniq. I considered doing something like that too, but I guessed that smartdns must already be doing this (although it would save perhaps a bit of startup time and disk space in any case). Have you tried comparing how much memory usage smartdns uses with/without removing potential duplicates?
It shrinks the file a lot because I merge 5 sources. And it doesn't take so much time on the router.
So I won't bother compare the memory, because I don't know how to do it and it would take me time to learn to do it that I can use to try to make that f#*@!ing L2TP/IPsec VPN SoftEther server work.
jtbr wrote:
If it's useful, there is something else that could be done. SmartDNS blocklists block all subdomains, while (iirc) hosts files do not. Thus there are a lot of entries of subdomains that could be removed from the file. (eg: badsite.com is blocked, no need to also block another.badsite.com: smartdns already does that, but they're in the lists anyway).
I disagree on that.
I had some taboola ads still showing, so I checked the file : I had subdomains but not the main domain, and my ads were from a subdomain that wasn't in the list.
Actually, I made a "simple" test : I gathered all domains like "^[a-z0-9_-]+\.(co\.)?[a-z]+$", and it was about 30% of the content of the file (taking all main domain isn't a good thing to do, because you have, for example "adservice.google.com" or "device-metrics-us.amazon.com").
The grep to filter those 70% facing the 30% as a filter file is incredibly long (about 7min on my desktop with high CPU usage...), then I add back the domain list, and it shrinks the global list by about 25 %.
I don't think it worses it, as the router doesn't have any performance problem, but that grep would be one.
I still have some ads going through and I don't know why : on the IMDB app, I have "amazon-adsystem.com" ads, but the subdomain and the main domain are in the list.
I wish I know how to log the URLs when I launch the IMDB app (only coming from the MAC of my smartphone), so I could know what isn't blocked exactly.
But if I have ads only when I use the IMDB app, I can live with it...
Note : before I flashed my router, I was using Pi-Hole, and I had a dedicated screen to see visited URLs.
But, well, if I can filter ads on my router instead of on another device, it's more convenient, more reliable, and more efficient. And I can dedicate my RPi4B to IoT. _________________ Netgear R7000P - r52720 : gateway behind ISP / DHCP server / VPN server / AdBlocking.
I'm curious about the de-duping you're doing with sort and uniq. I considered doing something like that too, but I guessed that smartdns must already be doing this (although it would save perhaps a bit of startup time and disk space in any case). Have you tried comparing how much memory usage smartdns uses with/without removing potential duplicates?
It shrinks the file a lot because I merge 5 sources. And it doesn't take so much time on the router.
So I won't bother compare the memory, because I don't know how to do it and it would take me time to learn to do it that I can use to try to make that f#*@!ing L2TP/IPsec VPN SoftEther server work.
If you just type 'top' from the shell, it will show you the memory usage for smartdns; you could compare with/without uniq. Still sounds helpful in your case due to the 5 independent lists.
Maze_fr wrote:
jtbr wrote:
If it's useful, there is something else that could be done. SmartDNS blocklists block all subdomains, while (iirc) hosts files do not. Thus there are a lot of entries of subdomains that could be removed from the file. (eg: badsite.com is blocked, no need to also block another.badsite.com: smartdns already does that, but they're in the lists anyway).
I disagree on that.
I had some taboola ads still showing, so I checked the file : I had subdomains but not the main domain, and my ads were from a subdomain that wasn't in the list.
Right, the behavior if you only have another.badsite.com on the list should be that s1.another.badsite.com is blocked, but www.badsite.com and badsite.com are not blocked.
Maze_fr wrote:
Actually, I made a "simple" test : I gathered all domains like "^[a-z0-9_-]+\.(co\.)?[a-z]+$", and it was about 30% of the content of the file (taking all main domain isn't a good thing to do, because you have, for example "adservice.google.com" or "device-metrics-us.amazon.com").
The grep to filter those 70% facing the 30% as a filter file is incredibly long (about 7min on my desktop with high CPU usage...), then I add back the domain list, and it shrinks the global list by about 25 %.
I don't think it worses it, as the router doesn't have any performance problem, but that grep would be one.
I think that a grep approach can't work, though it's helpful to get a handle on the scale of the question. It needs to check site-by-site. Like add all the TLDs, then check subdomains and ignore any that are already in the first list, then sub-subdomains, etc.
Maze_fr wrote:
I still have some ads going through and I don't know why : on the IMDB app, I have "amazon-adsystem.com" ads, but the subdomain and the main domain are in the list.
I wish I know how to log the URLs when I launch the IMDB app (only coming from the MAC of my smartphone), so I could know what isn't blocked exactly.
But if I have ads only when I use the IMDB app, I can live with it...
Not sure what's going on here, perhaps it's still cached on the device? Or the app has it hard-coded? If you use the entware version of smartdns (downloaded with 'opkg install smartdns' once entware is installed), you can enable logging and at the right verbosity, see each dns lookup as it happens. Good luck with that one
I still have some ads going through and I don't know why : on the IMDB app, I have "amazon-adsystem.com" ads, but the subdomain and the main domain are in the list.
I wish I know how to log the URLs when I launch the IMDB app (only coming from the MAC of my smartphone), so I could know what isn't blocked exactly.
But if I have ads only when I use the IMDB app, I can live with it...
Not sure what's going on here, perhaps it's still cached on the device? Or the app has it hard-coded? If you use the entware version of smartdns (downloaded with 'opkg install smartdns' once entware is installed), you can enable logging and at the right verbosity, see each dns lookup as it happens. Good luck with that one
Actually, the IMDB app is just like the website (I have the ads too), so I spied with the browser debugger... and uBlock request log...
So I think "https://www.imdb.com/_json/getads/..." this one is guilty.
I guess I need also an URL keyword filter service of some kind. _________________ Netgear R7000P - r52720 : gateway behind ISP / DHCP server / VPN server / AdBlocking.
Joined: 16 Nov 2015 Posts: 6410 Location: UK, London, just across the river..
Posted: Sat Jun 10, 2023 5:23 Post subject:
Some smart devices have hard coded DNS, that can go around and avoid DNS based ad blocking..
Solution is to use forced DNS or force those devices to use DNS you want via iptables rules..
The problem is, some devices may loose connectivity..but this is very rare...
Also some ads nowadays are blended inside the main stream, so those cannot be blocked that well..
If you use uBlock on PC level you don't really need ad block on router level...but yes for some devices it is useful...to use ad block router level.. lots of threads on the subject..
Its a cats and mouse game... you can also use IPset rules to build adblock too.. _________________ Atheros
TP-Link WR740Nv1 ---DD-WRT 55179 WAP
TP-Link WR1043NDv2 -DD-WRT 55303 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55460 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55460 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55363 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
The router level, it is precisely for smartphones and tablets.
(Actually, I've read about AdBlocking apps for mobile devices, creating a local VPN server and connection, for filtering the ads)
Also, some websites require to deactivate AdBlocking plugin in order to work, and router level isn't an AdBlocking plugin...
Unfortunately, IMDB "blends the ads in" by using it's own HTTPS API to call Amazon ads.
So whatever I can do with IP / DNS won't work.
I would need an URL keyword filter working on HTTPS, and that would be too much work for too few ads.
Well... if I block almost all ads on my network, it's already a great success. _________________ Netgear R7000P - r52720 : gateway behind ISP / DHCP server / VPN server / AdBlocking.
Joined: 16 Nov 2015 Posts: 6410 Location: UK, London, just across the river..
Posted: Sun Jun 11, 2023 9:29 Post subject:
well...have you ever checked SmartDNS for DNS leaks... https://dnscheck.tools/#results
i keep noticing 'from time to time' it tends to leak a lot...even if i use VPN..
I guess its down to its querying algorithm..where Stubby or DNScrypt proxy v2.xx never leak...
Speaking of DNScrypt proxy v2.xx it has very advanced ad-blocking / IP blocking, black/white lists, or even time based domain allowance options... _________________ Atheros
TP-Link WR740Nv1 ---DD-WRT 55179 WAP
TP-Link WR1043NDv2 -DD-WRT 55303 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55460 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55460 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55363 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
well...have you ever checked SmartDNS for DNS leaks... https://dnscheck.tools/#results
i keep noticing 'from time to time' it tends to leak a lot...even if i use VPN..
I guess its down to its querying algorithm..where Stubby or DNScrypt proxy v2.xx never leak...
I haven't had any problem with DNS leaks; although I don't check it regularly I don't see why it would change. Note that this is part of why I have a smartdns instance for each vpn client bridge (plus the wan). Each one has its own unique DNS servers which can only be accessed via the appropriate outward link (ie, DNS server 9.9.9.9 for the vpn tunnel for one vpn client, 8.8.8.8 directly for the wan). To ensure that the DNS servers for the vpn client bridges are only accessed via the vpn (and not directly via the wan), the openvpn client config contains a routing line like this for each DNS server reserved for that VPN:
Code:
route 9.9.9.9 255.255.255.255 vpn_gateway
In order to have multiple smartDNS instances, you have to re-route requests for the default DNS server to the correct one for the bridge. (This is because there is no way for DNSMasq to tell DHCP clients to use a non-default port for DNS). Ie, direct the bridge with the subnet 192.168.10.* to use the (Smart)DNS server at port 8053:
Code:
#Redirect traffic to different SmartDNS servers depending upon source subnet
#for br1, forward port 53 to port 8053 DNS service (needs both udp & tcp)
iptables -t nat -I PREROUTING -p udp -m udp -s 192.168.10.0/24 -d 192.168.10.1 --dport 53 -j REDIRECT --to-ports 8053
iptables -t nat -I PREROUTING -p tcp -m tcp -s 192.168.10.0/24 -d 192.168.10.1 --dport 53 -j REDIRECT --to-ports 8053
...
You also need to start them all yourself, or dd-wrt will not reset them properly. For me, SmartDNS is not enabled in the GUI, but started in the startup commands.
Joined: 16 Nov 2015 Posts: 6410 Location: UK, London, just across the river..
Posted: Tue Jun 13, 2023 7:30 Post subject:
"For me, SmartDNS is not enabled in the GUI, but started in the startup commands" - Although you can, nowadays you don't need to start it..this way...also there is an option where you can use smart 'Additional Servers Only' which will use exclusively those servers in smart DNS box and nothing else..and you still can use iptables to redirect...clients via iptables rules...
in the SMDNS box you can add any commands like port to use and ect. BS made the use of the command box very versatile...
The leak comes while im using SmartDNS via VPN as for some odd reason some specific dns servers are queried and the fastest comes from my real location (like NextDNS more often) instead of my VPN.. _________________ Atheros
TP-Link WR740Nv1 ---DD-WRT 55179 WAP
TP-Link WR1043NDv2 -DD-WRT 55303 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55460 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55460 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55363 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
The leak comes while im using SmartDNS via VPN as for some odd reason some specific dns servers are queried and the fastest comes from my real location (like NextDNS more often) instead of my VPN..
If you're in windows, the problem may be that you don't have block-outside-dns in the client settings, or it may be a windows bug (not sure if it still exists or not but wouldn't be surprised): see https://superuser.com/a/1195156/713936
Joined: 16 Nov 2015 Posts: 6410 Location: UK, London, just across the river..
Posted: Tue Jun 13, 2023 18:04 Post subject:
naah im talking all from router side...VPN and SmartDNS both on router side...and using particular DNS servers{(router side) DNS leaks..and im not using always Windows...Im talking various clients..and my windows is configured to use exclusively only router's DNS same as my other clients..it happens in very narrow scenarios..mostly using NextDNS resolver and some particular VPN servers...it happens not with all of them..but mostly using SmartDNS..
If i use Stubby DoT or Dnscrypt (it has some settings for querying types) i don't have DNS leaks... _________________ Atheros
TP-Link WR740Nv1 ---DD-WRT 55179 WAP
TP-Link WR1043NDv2 -DD-WRT 55303 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55460 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55460 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55363 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913