Is Ad-Blocking (Privoxy) Still Useful Nowadays?

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> General Questions
Goto page 1, 2  Next
Author Message
Timmy256
DD-WRT Novice


Joined: 06 Jul 2018
Posts: 3

PostPosted: Fri Jul 06, 2018 6:50    Post subject: Is Ad-Blocking (Privoxy) Still Useful Nowadays? Reply with quote
Hi there,

I was recently wondering about this and couldn't find any information about the topic. Since most of the websites are encrypted through HTTPS (and pretty much everything is migrating towards that), is Privoxy in the DD-WRT implementation still worth it? It's my understanding that it inspects and modifies HTML requests to block ads and such, but since said connections are HTTPS, I'm guessing it can't do it.

I've read in some places found via Google about installing some HTTPS handlers (or something like that) for standalone Prixovy, but I don't know if that would be possible to implement in DD-WRT.

Could anyone please shed some light into this? Thanks in advance for your help! Very Happy
Sponsor
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 4098
Location: UK, London, just across the river..

PostPosted: Sat Jul 07, 2018 5:39    Post subject: Reply with quote
yep its working and its present on the high flash size units but there are much better scripts for
ad blocking that work almost on every DD-WRT unit, personally i find privoxy a bit buggy

_________________
Atheros
TP-Link WR740Nv1 -----DD-WRT 44538 BS AP,NAT
TP-Link WR740Nv4 -----DD-WRT 44251 BS WAP/Switch
TP-Link WR1043NDv2 ---DD-WRT 45229 BS AP,NAT,AP Isolation,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---DD-WRT 45454 BS AP,NAT,AD Block,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---Gargoyle OS 1.12.0 AP,NAT,QoS,Quotas
Qualcomm/IPQ8065
Netgear R7800 -----DD-WRT 44719 BS AP,NAT,AD-Block,AP&Net Isolation,VLAN's,Firewall,Local DNS,DoT
Broadcom
Netgear R7000 -----DD-WRT 45454 BS AP,Wi-Fi OFF,NAT,AD-Block,Firewall,Local DNS,Forced DNS,DoT,VPN
-----------------------------------------------------------------------------------------------
Stubby for DNS over TLS I DNSCrypt v2 by mac913
Timmy256
DD-WRT Novice


Joined: 06 Jul 2018
Posts: 3

PostPosted: Sat Jul 07, 2018 23:35    Post subject: Reply with quote
Hi Alozaros, thanks for your reply!

I know that it actually works (for HTTP sites at least); I have it enabled for the past ~2 years on a Netgear R8000. What I was wondering is what happen with all the HTTPS sites, Privoxy on DD-WRT can't inspect and filter them, right?

Can you share the better scripts that you'd recommend over Privoxy? I've also found out it giving the gray "error" page a little bit too often; probably because it's filtering legal sites that it shouldn't be blocking (like "Unsuscribe" buttons that take you to ad networks when you actually want to unsuscribe from them or some not well-known e-commerce sites).

Thanks again!
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 4098
Location: UK, London, just across the river..

PostPosted: Sun Jul 08, 2018 7:17    Post subject: Reply with quote
add to start up script
wget -qO /tmp/mvps http://winhelp2002.mvps.org/hosts.txt
wget -qO /tmp/someonewhocares http://someonewhocares.org/hosts/zero/hosts
wget -qO /tmp/sbc http://sbc.io/hosts/hosts
stopservice dnsmasq && startservice dnsmasq

Under Setup/Services/DNSMasq/Additional DNSMasq Options:

addn-hosts=/tmp/mvps
addn-hosts=/tmp/someonewhocares
addn-hosts=/tmp/sbc

add to cronjob
0 12 * * * root /tmp/.rc_startup

and yes privoxi is buggy

_________________
Atheros
TP-Link WR740Nv1 -----DD-WRT 44538 BS AP,NAT
TP-Link WR740Nv4 -----DD-WRT 44251 BS WAP/Switch
TP-Link WR1043NDv2 ---DD-WRT 45229 BS AP,NAT,AP Isolation,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---DD-WRT 45454 BS AP,NAT,AD Block,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---Gargoyle OS 1.12.0 AP,NAT,QoS,Quotas
Qualcomm/IPQ8065
Netgear R7800 -----DD-WRT 44719 BS AP,NAT,AD-Block,AP&Net Isolation,VLAN's,Firewall,Local DNS,DoT
Broadcom
Netgear R7000 -----DD-WRT 45454 BS AP,Wi-Fi OFF,NAT,AD-Block,Firewall,Local DNS,Forced DNS,DoT,VPN
-----------------------------------------------------------------------------------------------
Stubby for DNS over TLS I DNSCrypt v2 by mac913
Timmy256
DD-WRT Novice


Joined: 06 Jul 2018
Posts: 3

PostPosted: Sun Jul 08, 2018 15:27    Post subject: Reply with quote
Great! Yes, I used one of those scripts on my previous, lower end router that didn't support the DD-WRT version that includes Privoxy. Actually, 75% of my upgrade to the R8000 was to get Privoxy built in Twisted Evil.

I'm using Adguard for DNS filtering nowadays, so I don't need to filter them myself anymore. But I was happy to add another layer with Privoxy locally.

So, if anybody can share some thoughts on if it's useful with all the HTTPS connections (considering that Privoxy doesn't touch tem), I'll be really helpful. Also, if there is some way to make Privoxy process them, adding some certificates of something; again, like Adguard for Windows does with HTTPS.

Thanks!
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 4098
Location: UK, London, just across the river..

PostPosted: Sun Jul 08, 2018 17:52    Post subject: Reply with quote
hmm as far as i know this script has nothing common with https modified headders Smile
this script relys on hots/tmp file block list..
so far its much better than privoxy
for more on the subject https://forum.dd-wrt.com/phpBB2/viewtopic.php?p=1078933

_________________
Atheros
TP-Link WR740Nv1 -----DD-WRT 44538 BS AP,NAT
TP-Link WR740Nv4 -----DD-WRT 44251 BS WAP/Switch
TP-Link WR1043NDv2 ---DD-WRT 45229 BS AP,NAT,AP Isolation,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---DD-WRT 45454 BS AP,NAT,AD Block,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---Gargoyle OS 1.12.0 AP,NAT,QoS,Quotas
Qualcomm/IPQ8065
Netgear R7800 -----DD-WRT 44719 BS AP,NAT,AD-Block,AP&Net Isolation,VLAN's,Firewall,Local DNS,DoT
Broadcom
Netgear R7000 -----DD-WRT 45454 BS AP,Wi-Fi OFF,NAT,AD-Block,Firewall,Local DNS,Forced DNS,DoT,VPN
-----------------------------------------------------------------------------------------------
Stubby for DNS over TLS I DNSCrypt v2 by mac913
tatsuya46
DD-WRT Guru


Joined: 03 Jan 2010
Posts: 7410
Location: YWG, Canada

PostPosted: Sun Jul 08, 2018 20:23    Post subject: Reply with quote
privoxy is a messy scripty thing thats too much for me, and it doesnt "just work" like dns filtering does for all devices. so i use ad and tracking blocking with dnsmasq with my own built list
_________________
LATEST FIRMWARE(S)

BrainSlayer wrote:
we just do it since we do not like any restrictions enforced by stupid cocaine snorting managers

[x86_64] Haswell i3-4150/QCA9984/QCA9882 ------> r45460 std
[QUALCOMM] DIR-862L --------------------------------> r45460 std
[QUALCOMM] WNDR4300 v1 --------------------------> r45460 std
[QUALCOMM] DIR-862L --------------------------------> r45460 std
▲ ACTIVE / INACTIVE ▼
[BROADCOM] DIR-860L A1 ----------------------------> r44901 std


If you use DSLReports please enable hi-res bufferbloat.


Sigh.. why do i exist anyway..
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 4098
Location: UK, London, just across the river..

PostPosted: Mon Jul 09, 2018 6:06    Post subject: Reply with quote
tatsuya46 wrote:
privoxy is a messy scripty thing thats too much for me, and it doesnt "just work" like dns filtering does for all devices. so i use ad and tracking blocking with dnsmasq with my own built list


hmm its interesting what tracking blocking script you use ??

_________________
Atheros
TP-Link WR740Nv1 -----DD-WRT 44538 BS AP,NAT
TP-Link WR740Nv4 -----DD-WRT 44251 BS WAP/Switch
TP-Link WR1043NDv2 ---DD-WRT 45229 BS AP,NAT,AP Isolation,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---DD-WRT 45454 BS AP,NAT,AD Block,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---Gargoyle OS 1.12.0 AP,NAT,QoS,Quotas
Qualcomm/IPQ8065
Netgear R7800 -----DD-WRT 44719 BS AP,NAT,AD-Block,AP&Net Isolation,VLAN's,Firewall,Local DNS,DoT
Broadcom
Netgear R7000 -----DD-WRT 45454 BS AP,Wi-Fi OFF,NAT,AD-Block,Firewall,Local DNS,Forced DNS,DoT,VPN
-----------------------------------------------------------------------------------------------
Stubby for DNS over TLS I DNSCrypt v2 by mac913
JacobSeymour
DD-WRT Novice


Joined: 01 Sep 2018
Posts: 1

PostPosted: Sat Sep 01, 2018 10:50    Post subject: Reply with quote
tatsuya46 wrote:
privoxy is a messy scripty thing thats too much for me evertime, and it doesnt "just work" like dns filtering does for all devices. so find my results posted on this article ad and tracking blocking with dnsmasq with my own built list


Yes, you're right. Privoxy is mediocre at its best. Would you be kind enough to share your own curated list?


Last edited by JacobSeymour on Mon Sep 21, 2020 23:53; edited 1 time in total
underdose
DD-WRT Novice


Joined: 12 Jun 2019
Posts: 20

PostPosted: Sun Jul 21, 2019 12:35    Post subject: Reply with quote
What do you think of Diversion with added support for https?

https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=320563&highlight=

It's a superb ad-blocker for Asus-Merlin and the owner of the code is thinking of porting it to DD-WRT if there's a demand for it. I've used it on my N66W for more than a year and highly recommend it.

Thoughts?
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 1060
Location: Appalachian mountains, USA

PostPosted: Mon Jul 22, 2019 21:53    Post subject: Reply with quote
Alozaros wrote:
add to start up script
wget -qO /tmp/mvps http://winhelp2002.mvps.org/hosts.txt
wget -qO /tmp/someonewhocares http://someonewhocares.org/hosts/zero/hosts
wget -qO /tmp/sbc http://sbc.io/hosts/hosts
stopservice dnsmasq && startservice dnsmasq

Under Setup/Services/DNSMasq/Additional DNSMasq Options:

addn-hosts=/tmp/mvps
addn-hosts=/tmp/someonewhocares
addn-hosts=/tmp/sbc

add to cronjob
0 12 * * * root /tmp/.rc_startup

and yes privoxi is buggy


Well, here's a really late reply! This is an update to Alazaros's code above to set up dnsmasq to return dummy IP address 0.0.0.0 for over 40,000 dubious domain names.

I updated the Alazaros Startup code to this:
Code:

#hosts to block in dnsmasq (Alazaros 7/8/18 post in
#https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=315773)
  touch /tmp/badhosts
  ( (sleep 60 ; \
     wget -qO - http://winhelp2002.mvps.org/hosts.txt ; \
     wget -qO - http://sbc.io/hosts/hosts ; \
#    wget -qO - http://someonewhocares.org/hosts/zero/hosts \
    ) | sed -e 's/\r//' -e '/^0\.0\.0\.0 /!d' -e '/0\.0\.0\.0$/d' \
      | sort -u >/tmp/badhosts \
  ) && stopservice dnsmasq && startservice dnsmasq &

The touch is to ensure the file is there when dnsmasq looks for it early on. This may not be needed. The sleep is to give my two dnscrypt-proxy instances time to get certificates before asking them to look things up. The 60 sec is definitely overkill, but I'm too lazy to test to trim it down. One wget is commented out because it forwards to an https site that wget can't handle. The sed first eliminates the CR characters, since one of the files was apparently generated on Windows, and it eliminates the seemingly pointless line 0.0.0.0 0.0.0.0 as well. Most importantly, and this is the whole point of this report, it eliminates any line that does not begin with 0.0.0.0 so that slipping in a line like blah.di.blah.di goodbank.com, either at the source or in some mitm attack, can't steer your banking to evilbank.com at IP address blah.di.blah.di. The sort eliminates redundant entries; it reduced the file size about 20%.

Just the one line addn-hosts=/tmp/badhosts needs incuding in DNSMasq Additional Options. Doesn't seem to matter where it goes relative to other lines there.

_________________
Five WRT1900ACSv2's on 42926, 44048.
VLANs, VAPs, NAS, client-mode travel router, OpenVPN client (AirVPN), DDNS, wireguard servers, wireguard clients (AzireVPN), two DNSCrypt DNS providers (incl Quad9) via OpenVPN/wireguard clients.
PavelVD
DD-WRT User


Joined: 26 Jul 2019
Posts: 67

PostPosted: Sat Aug 31, 2019 22:19    Post subject: Reply with quote
SurprisedItWorks or Alozaros
I am not using DNSCrypt, but Unbound (Recursive DNS Resolving) along with DNSMaq as suggested here. And your method does not work: If you take some real site from the /tmp/badhosts file, for example - "skgroup.kiev.ua", then it will still be available, albeit with some delay.
You have no idea how to win this?
SurprisedItWorks
DD-WRT Guru


Joined: 04 Aug 2018
Posts: 1060
Location: Appalachian mountains, USA

PostPosted: Sat Aug 31, 2019 23:36    Post subject: Reply with quote
PavelVD wrote:
SurprisedItWorks or Alozaros
I am not using DNSCrypt, but Unbound (Recursive DNS Resolving) along with DNSMaq as suggested here. And your method does not work: If you take some real site from the /tmp/badhosts file, for example - "skgroup.kiev.ua", then it will still be available, albeit with some delay.
You have no idea how to win this?

Actually, I have tested it many times with sites from /tmp/badhosts, and it works fine. Of note, however, is that my router has IPv6 disabled (my vpn provider requires that for its recommended dd-wrt setup), so the fact that some systems will obtain IPv6 addresses for these sites and allow them to be loaded does not affect me. One could modify the sed script to turn each line into two for the same domain, with one line mapping it to 0.0.0.0 for IPv4 and the other mapping it to :: for IPv6, but I have only done the most minimal experiments in that direction.

I have improved the script a bit since then, though the functioning is basically the same, but now with capturing of error information in case one of the sites fails to download. Also I now use curl instead of wget. Here is what I currently have in my startup commands for adblocking:
Code:
#hosts to block in dnsmasq (Alozaros 7/8/18 post in
#https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=315773)
  ( cd /tmp;
     touch badhosts
     sleep 30
     ( ERRA=0 ; ERRB=0 ; ERRC=0
       curl -s http://winhelp2002.mvps.org/hosts.txt \
         2>badhosts.log || ERRA=$?
       curl -s http://sbc.io/hosts/hosts \
         2>>badhosts.log || ERRB=$?
       curl -sk https://someonewhocares.org/hosts/zero/hosts \
         2>>badhosts.log || ERRC=$?
       echo $ERRA $ERRB $ERRC > badhosts.errcodes
     ) | sed 's/\t/ /g; /^0\.0\.0\.0 /!d; s/ *\#.*$//; s/\r//' \
       | sort -u \
       | sed -E '/\.hulu(|ad)\./d' \
       > badhosts
     ) && stopservice dnsmasq && startservice dnsmasq &

The hulu line near the end is to make an exception for the key hulu ad sites without which hulu streaming simply fails. They don't permit you to opt out of ads. One could modify the line to add other exceptions.

The file /tmp/badhosts.errcodes should show 0 0 0 when everything is working, and the file /tmp/badhosts.log should be empty. If /tmp/badhosts.errcodes has any nonzeros, its position will reveal which curl had problems, and /tmp/badhosts.log should clarify the nature of the error. I added these features assuming that eventually one or more of those three lists of "bad" domains will no longer be posted.

ALSO, VERY IMPORTANT! In GUI>Services>Services in Additional Dnsmasq Options, one must add the line addn-hosts=/tmp/badhosts or the script will indeed be useless!

_________________
Five WRT1900ACSv2's on 42926, 44048.
VLANs, VAPs, NAS, client-mode travel router, OpenVPN client (AirVPN), DDNS, wireguard servers, wireguard clients (AzireVPN), two DNSCrypt DNS providers (incl Quad9) via OpenVPN/wireguard clients.
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 4098
Location: UK, London, just across the river..

PostPosted: Sun Sep 01, 2019 7:11    Post subject: Reply with quote
back in the days, when i tried unbound, i noticed those
adblocker scripts, ware not working as intended, but that was the unbound, recursive resolving option from basic set up page...i never tried unbound via entware..
I can also confirm those scripts are working with DNScrypt, and stubby via entware witch i have running on my units too...

_________________
Atheros
TP-Link WR740Nv1 -----DD-WRT 44538 BS AP,NAT
TP-Link WR740Nv4 -----DD-WRT 44251 BS WAP/Switch
TP-Link WR1043NDv2 ---DD-WRT 45229 BS AP,NAT,AP Isolation,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---DD-WRT 45454 BS AP,NAT,AD Block,Firewall,Local DNS,Forced DNS,DoT,VPN,VLAN
TP-Link WR1043NDv2 ---Gargoyle OS 1.12.0 AP,NAT,QoS,Quotas
Qualcomm/IPQ8065
Netgear R7800 -----DD-WRT 44719 BS AP,NAT,AD-Block,AP&Net Isolation,VLAN's,Firewall,Local DNS,DoT
Broadcom
Netgear R7000 -----DD-WRT 45454 BS AP,Wi-Fi OFF,NAT,AD-Block,Firewall,Local DNS,Forced DNS,DoT,VPN
-----------------------------------------------------------------------------------------------
Stubby for DNS over TLS I DNSCrypt v2 by mac913
PavelVD
DD-WRT User


Joined: 26 Jul 2019
Posts: 67

PostPosted: Sun Sep 01, 2019 8:45    Post subject: Reply with quote
Thanks SurprisedItWorks! I like your improvement!
I allowed a slight improvement for myself:
The command "0 12 * * * root /tmp/.rc_startup" from the section "Additional Cron Jobs" restarts services that I do not want to touch, and replaced it with:
Code:
30 02 * * * root /tmp/custom.sh
And your script placed in "Administration--Commands--Custom Script", and changed for yourself:
Code:
#!/bin/sh
#hosts to block in dnsmasq (Alozaros 7/8/18 post in
#https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=315773)
  ( cd /jffs;
     # sleep 30
     touch badhosts
     ( ERRA=0 ; ERRB=0 ; ERRC=0
       curl -s http://winhelp2002.mvps.org/hosts.txt \
         2>badhosts.log || ERRA=$?
       curl -s http://sbc.io/hosts/hosts \
         2>>badhosts.log || ERRB=$?
       curl -sk https://someonewhocares.org/hosts/zero/hosts \
         2>>badhosts.log || ERRC=$?
       echo $ERRA $ERRB $ERRC > badhosts.errcodes
     ) | sed 's/\t/ /g; /^0\.0\.0\.0 /!d; s/ *\#.*$//; s/\r//' \
       | sort -u \
       | sed -E '/\.hulu(|ad)\./d' \
       > badhosts
     ) && stopservice dnsmasq && dnsmasq --conf-file=/tmp/dnsmasq.conf

You need to press the "Run Commads" button once by placing "/tmp/custom.sh" in "Administration - Commands--Command Shell--Commands". (It is clear that you need to do this after rebooting the router after completing the setup.)

The advantage is that the "/jffs/badhosts" file does not disappear when the router is restarted, it is available very quickly and there is no need to wait for the certificate to be updated for DNSCrypt. (Just in case, I inform you that "jffs" is mounted automatically on an external USB flash card.)

Well and, of course, in "Additional Dnsmasq Options" include
Code:
addn-hosts=/jffs/badhosts

Everything works well! Very Happy

The only pity is that there is no white list of sites for which badhosts would not work. Confused

And, yes ... The Unbound malfunctioning error that I wrote about above ... Gone! I did not understand what it was.

Thank you very much!
Goto page 1, 2  Next Display posts from previous:    Page 1 of 2
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> General Questions All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum