Ad-blocking on DD-WRT

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware
Goto page 1, 2, 3 ... 15, 16, 17  Next
Author Message
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 96

PostPosted: Mon Feb 20, 2017 11:33    Post subject: Ad-blocking on DD-WRT Reply with quote
Ad-blocking on DD-WRT
---------------------
Script to generate a MEGA hosts file for DD-WRT

https://github.com/m-parashar/adbhostgen
Copyright 2018 Manish Parashar

DOWNLOAD: https://github.com/m-parashar/adbhostgen/releases/latest

Developed and tested on: NETGEAR R8500 / DD-WRT v3.0-r35030M kongac
Hardware requirements: Minimum 4GB USB drive.
Partitioned, formatted, and mounted as /jffs (1-2GB), swap (256-512MB), and optionally /opt (1-2GB)

NOTE: DO NOT try to run this script on your internal jffs.

Software requirements: DD-WRT (preferably latest), cURL

Installation:
-------------

1a. Download installer.sh and make it executable (chmod +x), then run it in /tmp or /jffs.
It will automatically create /jffs/dnsmasq and extract required files into it. OR

1b. Download and extract adbhostgen.7z or adbhostgen.tar.gz into /jffs/dnsmasq directory.

2. Run "./adbhostgen.sh" in /jffs/dnsmasq without quotes.

3. Use --update or -u command line option to update to the latest version. [only available since 20180315]

Settings:
---------

1. Disable internal flash (JFFS2) if you already have a USB drive mounted as /jffs.





2. Enable DNSMasq and local DNS for LAN and WAN. Add these lines under the additional options section.

Code:
conf-file=/jffs/dnsmasq/mpdomains
addn-hosts=/jffs/dnsmasq/mphosts




3. Enter additional options for dnsmasq if required, for example:

Code:
domain-needed
bogus-priv


4. Under Administration -> Cron, enter this or choose your own schedule:

Code:
0 6 * * 1,4 root /jffs/dnsmasq/adbhostgen.sh




5. Reboot


Usage:
------

Add your blacklist or whitelist domains to myblacklist and mywhitelist files respectively.

Do not save anything in blacklist or whitelist files as they will be overwritten every
time the script is executed.

Status:





--

BLITZ modes and host file sizes:

BLITZ=0 mphosts: 2.7 MB, number of domains blocked: ~103743
BLITZ=1 mphosts: 9.9 MB, number of domains blocked: ~358430
BLITZ=2 mphosts: 24.0 MB, number of domains blocked: ~866434
BLITZ=3 mphosts: 31.9 MB, number of domains blocked: ~1159037

mpdomains: ~3.1MB after processing (enabled in all modes)

BLITZ modes and processing time (on R8500):

BLITZ=0: 1:40 minutes
BLITZ=1: 3:50 minutes
BLITZ=2: 7:00 minutes
BLITZ=3: 10:14 minutes

Changelog:

20180331:
POINT RELEASE with all the bugfixes and new features from the alpha releases below.

20180329a2:
No more "bad name at line n" errors.

20180328a2:
Changed default ad destination IP from 0.0.0.0 to 0.1.2.3

20180327a1:
New options:
-i <ip-address>
--ip=<ip-address>
where <ip-address> is an IPv4 address of your own personal ad hell.

20180326a1:
New options:
-f, blocks Facebook.com and services
-F, blocks Facebook, Instagram, and WhatsApp

20180324a6:
Significant changes and bugfixes. The script should behave better as a cron job now.
A new option -q or --quiet suppresses output to stdout/console, yet retains log entires.
Considering that many users are still on ancient DD-WRT versions with outdated and/or
expired cURL certificates and exotic configuration that often doesn't play well with
this script, it now uses insecure cURL downloads (-k) as default mode of operation.
For experienced users, the new --secure command line parameter restores the secure
TLS/SSL communication with servers. After thinking long and hard and working with
users on solving their problems, offering sane defaults with switches to enable
advanced functionality seems like a good tradeoff. For now.




DOWNLOAD: https://github.com/m-parashar/adbhostgen/releases/latest

See this post for detailed instructions.

Things to whitelist (mywhitelist file or adbhostgen.sh -w <domainname> --wl=<domainname>)

* https://discourse.pi-hole.net/t/commonly-whitelisted-domains/212
* https://github.com/anudeepND/whitelist/blob/master/domains/whitelist.txt

_________________

Router
NETGEAR R7800: DD-WRT v3.0-r41517 std (11/11/19)
NETGEAR R8500: DD-WRT v3.0-r41517 std (11/11/19)

Scripts
Ad-blocking on DD-WRT -- Github


Last edited by Yamaraj on Sun Apr 01, 2018 5:23; edited 61 times in total
Sponsor
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 96

PostPosted: Tue Feb 21, 2017 13:19    Post subject: Reply with quote
rev19: Cleaned up, added custom hosts.
rev22: Significant updates.
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 96

PostPosted: Sat Feb 25, 2017 6:52    Post subject: Reply with quote
rev25: Speed optimizations. The script now gracefully handles blank lines and trailing whitespace in custom_hosts and whitelist.

rev26: Full path introduced to avoid cron failure. [2017/03/04]
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 96

PostPosted: Fri Mar 17, 2017 9:50    Post subject: Reply with quote
rev28: script gracefully handles network downtime failures and DNS resolution issues. Automatically sorts whitelist and custom_hosts, and creates backup of the last hosts file.

rev29: whitelist updated. [2017/03/19]
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 96

PostPosted: Thu May 04, 2017 10:17    Post subject: Reply with quote
rev30: whitelist updated. "custom_hosts" renamed to "blacklist" to avoid ambiguity.

Rock solid performance so far. Any feedback is welcome.
tweakradje
DD-WRT Novice


Joined: 25 Sep 2008
Posts: 34

PostPosted: Fri May 05, 2017 21:23    Post subject: Why so complicated? Reply with quote
I only use these 3 lines in startup for blocking:


sleep 20

# get hosts file with unix EOL
wget -qO- http://www.mvps.org/winhelp2002/hosts.txt|grep "^0.0.0.0" >> /tmp/hosts

# re-read hosts file
killall -HUP dnsmasq

together with dnsmasq as local dns and dhcp. You can use any inet hosts file you like of course. Disadvantage of hosts is it need to be hosts not domains. As extra I use the dnsmasq option field. I list the domains like:


domain-needed
bogus-priv
no-negcache
address=/2mdn.net/0.0.0.0
address=/2o7.net/0.0.0.0
address=/360yield.com/0.0.0.0
address=/ad-center.com/0.0.0.0
address=/ad-inside.com/0.0.0.0
address=/adbrite.com/0.0.0.0
address=/adinfuse.com/0.0.0.0
address=/admob.com/0.0.0.0
address=/admob.mobi/0.0.0.0
...


Can you explain what your method does more?

Cheers

_________________
Netgear R6400
Linksys wrt54G v5
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 96

PostPosted: Mon May 08, 2017 19:02    Post subject: Reply with quote
1. The script merges 18 different hosts files into one for aggressive ad-blocking. This is obviously a custom solution since I use the same hosts file across different systems/networks.

2. Includes a whitelist/blacklist solution for custom needs.

3. Helps you start right away with minimum configuration. It's almost completely automated and portable.

4. Creates a backup of the last hosts file, just in case.

5. It's the best solution for my needs so far.

The script is actually quite straightforward and simple.

Best.
fatalhalt
DD-WRT Novice


Joined: 29 Oct 2015
Posts: 27

PostPosted: Tue May 09, 2017 3:10    Post subject: Reply with quote
Seems like a solid block list. I was only aware of somewhocares.org blocklist, but seeing your script there are a lot more lists out there.

You could silence the first curl command as well with '-s' (the "01. Mother of All Ad Blocks list")

11 megabyte text file is a pretty big list Surprised .
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 96

PostPosted: Sat May 13, 2017 17:09    Post subject: Reply with quote
Thank you!

rev31: Renamed "createmphosts.sh" to "adbhostgen.sh", added silent option to the #01 list, as suggested by fatalhalt, updated the original post with fresh instructions and images.
genlish
DD-WRT Novice


Joined: 14 Jun 2017
Posts: 3

PostPosted: Wed Jun 21, 2017 11:27    Post subject: Re: Ad-blocking on Netgear R8500 Reply with quote
Yamaraj wrote:
System: Netgear R8500

Firmware: Kong's DD-WRT v3.0-r31870M (04/16/17)

Additional configuration: OpenVPN client and an 8GB pendrive partitioned and formatted as swap, /jffs, and /opt running Kong's optware.

Background: Needed a more elegant and basic setup than dealing with the nuances of OpenVPN and Privoxy enabled on the same router. Repurposing RPi3 had something to do with it, too.

Solution: hosts file & dnsmasq

Link: https://gist.github.com/m-parashar/ee38454c27f7a4f4e4ab28249a834ccc

Instructions:

1. Copy/create adbhostgen.sh in /jffs/dnsmasq directory. chmod +x it.

https://gist.github.com/m-parashar/ee38454c27f7a4f4e4ab28249a834ccc/raw/5968ae10fc16a0ea411c601e677e8118032ad4d9/adbhostgen.sh

2. Create/download the file "whitelist" without quotes in /jffs/dnsmasq and populate it with the domains you do not want blocked. ̶E̶n̶s̶u̶r̶e̶ ̶t̶h̶e̶r̶e̶'̶s̶ ̶n̶o̶ ̶t̶r̶a̶i̶l̶i̶n̶g̶ ̶n̶e̶w̶l̶i̶n̶e̶/̶c̶r̶.̶ [rev25]

https://gist.github.com/m-parashar/ee38454c27f7a4f4e4ab28249a834ccc/raw/5968ae10fc16a0ea411c601e677e8118032ad4d9/whitelist

3. Create/download file "blacklist" and populate it with the domains you want to block. [rev30]

https://gist.github.com/m-parashar/ee38454c27f7a4f4e4ab28249a834ccc/raw/5968ae10fc16a0ea411c601e677e8118032ad4d9/blacklist

4. Execute adbhostgen.sh in /jffs/dnsmasq to generate the hosts file. By default the hosts file is quite aggressive and approx 10MB in size, which is manageable enough for Netgear R8500. Comment out the hosts repos as you see fit.

5. Enable DNSMasq and local DNS for LAN and WAN

6. Enter this into the additional options field
Code:
addn-hosts=/jffs/dnsmasq/mphosts
domain-needed
bogus-priv




7. Under Administration -> Cron, enter this or choose your own schedule:
Code:
0 6 * * 1,4 root /jffs/dnsmasq/adbhostgen.sh
30 6 * * 1,4 root restart_dns




8. Reboot

Should work on R7000, R8000 and other >128MB RAM routers too. Any feedback is welcome.



It`s any way to make one computer (specified ip) or a IP range to be excluded from the filter? Like using iptables on router or using a different dns on the computer?
I've tried the solution of chjohans:
http://www.dd-wrt.com/phpBB2/viewtopic.php?p=740330
"Add the following to your "Startup" script:
dnsmasq -S 208.67.222.222 -R -i br0 -p 1054

This starts an dnsmasq instance on port 1054 using the OpenDNS DNS server (208.67.222.222)

Add the following to your "Firewall" script:
iptables -t nat -A PREROUTING -p tcp -i br0 -s 192.168.1.100 --dport 53 -j DNAT --to $(nvram get lan_ipaddr):1054
iptables -t nat -A PREROUTING -p udp -i br0 -s 192.168.1.100 --dport 53 -j DNAT --to $(nvram get lan_ipaddr):1054
"

but the router prioritizes his own host file instead skiping it.
does anyone have the solution?
Thanks.
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 96

PostPosted: Sat Nov 25, 2017 6:36    Post subject: Re: Ad-blocking on Netgear R8500 Reply with quote
genlish wrote:
It`s any way to make one computer (specified ip) or a IP range to be excluded from the filter? Like using iptables on router or using a different dns on the computer?
I've tried the solution of chjohans:
http://www.dd-wrt.com/phpBB2/viewtopic.php?p=740330
"Add the following to your "Startup" script:
dnsmasq -S 208.67.222.222 -R -i br0 -p 1054

This starts an dnsmasq instance on port 1054 using the OpenDNS DNS server (208.67.222.222)

Add the following to your "Firewall" script:
iptables -t nat -A PREROUTING -p tcp -i br0 -s 192.168.1.100 --dport 53 -j DNAT --to $(nvram get lan_ipaddr):1054
iptables -t nat -A PREROUTING -p udp -i br0 -s 192.168.1.100 --dport 53 -j DNAT --to $(nvram get lan_ipaddr):1054
"

but the router prioritizes his own host file instead skiping it.
does anyone have the solution?
Thanks.


You will probably need to install optware and use another instance of dnsmasq to setup a separate DNS for those IPs. I haven't tried that yet. Anyone?


[rev33]: added mpdomains for additional protection. Lists updated.

SIZE:
mpdomains: ~2MiB
mphosts: ~14MiB


Status:




_________________

Router
NETGEAR R7800: DD-WRT v3.0-r41517 std (11/11/19)
NETGEAR R8500: DD-WRT v3.0-r41517 std (11/11/19)

Scripts
Ad-blocking on DD-WRT -- Github


Last edited by Yamaraj on Sat Nov 25, 2017 12:28; edited 1 time in total
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 3065
Location: UK, London, just across the river..

PostPosted: Sat Nov 25, 2017 8:06    Post subject: Reply with quote
personally i find privoxy as a little slow and buggy, although its working it add's a latency on browsing and sometimes blocks not listed sites ...
_________________
Atheros
TP-Link WR740Nv1 ------DD-WRT 33772 BS WAP/Switch (wired)
TP-Link WR1043NDv2 ----DD-WRT 41659 BS (AP,PPPoE,NAT,AD Blocking,AP Isolation,Firewall,Local DNS,Forced DNS,DoT, VPN)
TP-Link WR1043NDv2 ----DD-WRT 41664 BS (AP,NAT,AD Blocking,Firewall,Wi-Fi OFF,Local DNS,Forced DNS,DoT)
TP-Link WR1043NDv2 ----Gargoyle OS 1.11.0 (AP,NAT,QoS,Quotas)
Qualcomm/IPQ8065
2x Netgear R7800 -------DD-WRT 40270M 4.9 Kong (AP,NAT,AD-Blocking,AP&Net Isolation,VLAN's,Firewall,Local DNS,DNSCrypt-proxy v2 x2)
Broadcom
Netgear R7000 -------DD-WRT 40270M Kong (AP,NAT,VLAN,AD-Blocking,Firewall,Local DNS,Forced DNS,DoT)
------------------------------------------------------------------------------------------------
Stubby for DNS over TLS I DNSCrypt v2 via Entware by mac913
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 96

PostPosted: Sat Nov 25, 2017 11:02    Post subject: Reply with quote
Alozaros wrote:
personally i find privoxy as a little slow and buggy, although its working it add's a latency on browsing and sometimes blocks not listed sites ...

Yeah, I played with it for a while and then settled for this solution for the same reasons. This script may not be perfect, but it can be tailored to do exactly what I want.

_________________

Router
NETGEAR R7800: DD-WRT v3.0-r41517 std (11/11/19)
NETGEAR R8500: DD-WRT v3.0-r41517 std (11/11/19)

Scripts
Ad-blocking on DD-WRT -- Github
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 96

PostPosted: Thu Nov 30, 2017 17:15    Post subject: Reply with quote
rev36: Added a secure version of the script which downloads cURL CA cert directly from the author's server and then uses secure transmission for downloading lists. Whitelist now applies to the domain list as well as hosts list.
_________________

Router
NETGEAR R7800: DD-WRT v3.0-r41517 std (11/11/19)
NETGEAR R8500: DD-WRT v3.0-r41517 std (11/11/19)

Scripts
Ad-blocking on DD-WRT -- Github
labo
DD-WRT Guru


Joined: 30 Jan 2015
Posts: 673
Location: Texas, USA

PostPosted: Tue Dec 05, 2017 16:20    Post subject: Reply with quote
Yamaraj,
I tried this and works great!. But seems bit aggressive list. Do you have a lite version of the blacklist?
With this current list I see issues with,
facebook and some deal sites such as Slickdeals.com and some legitimate redirect links.

_________________
1 X R7800: Main gateway( r39855M kongat (05/25/19)
1 X RAX120 WAP Bridge
1 X RAX75 WAP Bridge
1 X R7800 Mesh
2 X R9000 Mesh
1 X R8000: Repeater bridge
1 X R8500: Client bridge (RB doesn't work on R8500)
Goto page 1, 2, 3 ... 15, 16, 17  Next Display posts from previous:    Page 1 of 17
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum