Ad-blocking on DD-WRT

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware
Goto page Previous  1, 2, 3, 4 ... 15, 16, 17  Next
Author Message
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 95

PostPosted: Tue Feb 20, 2018 10:59    Post subject: Re: quick question from the beginner Reply with quote
kkdu wrote:
Thanks Yamaraj and others for testing this.

From the beginners point of reading, the steps isnt clear yet (atleast to me), or i overlooked the critical steps.

I have a router Dir868L with DDWRT v24sp-2-std build 25974. A 8GB USB is formatted as /swap and ready to be plugged-in.

My questions are

1. Do we need to ENABLE or DISABLE jffs in the router. Screenshot shows disabled.
2. If its disabled how to proceed ; do I need to create the /dnsmasq folder in /jffs path. will this work
3. If /jffs is enable, will the /dsmasq, /mphosts, /mpdomains still need to created.
4. will the UUID for mounting the partition be explicitly added or automatically detected.


@kkdu

I'll admit the instructions aren't quite beginner friendly. I hope to update them soon as I can.

1. If you already have a USB drive formatted as /jffs and swap (optionally /opt as well), then you can and should disable internal flash storage, as shown in the screenshot. Those without a pendrive have no choice but to keep it enabled.

2. If your pendrive does not have a /jffs partition, I recommend you create one. Any and all standard Linux tools should work, e.g. fdisk. You can SSH to the router and use fdisk to create /jffs. Google for details on creating a partition using fdisk. Use ext2 or ext3 for /jffs and /opt. Once it is done and your router recognizes the /jffs partition on the pendrive, you can create a directory named "dnsmasq" on partition /jffs.

Code:
cd /jffs
mkdir dnsmasq



3. Whether you use external /jffs or internal, you will need to create the directory "dnsmasq" on partition /jffs and then copy all the files inside the 7z archive to this directory. Alternatively, you could use cURL to download the files directly from Github.

4. Once you create /jffs, swap, (and optionally /opt) on your pendrive and reboot, you will have the UUIDs on Services>USB page if you have enabled USB support. You can, then, copy the UUID of /jffs to the field shown in screenshot.

In short, UUID is automatically created, but you need to explicitly tell the router to map it to a specific mount point.


_________________

Router
Netgear R8500: DD-WRT v3.0-r36840M kongac (09/03/18) / SFE: quarkysg - r34080-linux-4.4.105

Scripts
Ad-blocking on DD-WRT -- Github
Sponsor
kkdu
DD-WRT Novice


Joined: 20 Feb 2018
Posts: 3

PostPosted: Wed Feb 21, 2018 14:55    Post subject: Reply with quote
Thanks Yamaraj.
Installed as instructed and it started to work as expected. Was able to enjoy the difference the way the page loads now.

I would still have couple of questions that need your input.

1. when executing the script #time ./abhostgen.sh the first time, the script runs as expected generating all hosts files and saved to the mphosts.log. I see that there are 157 ad domains blocked. However , when checking the file sizes of mpdomains and mphosts , its reported as 0B and 4kB respectively.

2. I then overwrite these files from the original to get the original sizes of 3MB and 9MB. Again executing the script, this reduces the size of these files. Is there anything im missing

3. Most sites riddled with ads are tested and are blocked successfully. ex. http://www.angelfire.com/alt2/entertainment/ad_block_test.html .However, when trying to run the same site in the Chrome incognito mode, some ads appear again. Test on FireFox - no issue. is this a behavior of Chrome and have you experienced it.

4. Youtube ads most time are killed, however, some come back. Not sure if this is related to the issue 1 where the mpdoamins and mphosts are not updated.

5. Can this link https://www.reddit.com/r/dropgoogle/comments/5tnjxl/block_youtube_ads_2017_hosts_file/ be added to the mphosts to get rid of any Youtube Ads


PS: I am also finishing the manual/documentation for the setup , will pm you once i have all things checked. if okayed, you can choose to publish for all users
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 95

PostPosted: Wed Feb 21, 2018 19:39    Post subject: Reply with quote
@kkdu, glad to hear it.

1 & 2. You needn't time the script when running it. "./adbhostgen.sh" without the quotes is enough. Sounds like an issue with configuration. Check and confirm that the script and other files are in /jffs/dnsmasq directory. You can PM me with screenshots of your router config and an ls -al of /jffs/dnsmasq.

3. You may need to clear Chrome cache. Google have more than a few fallback options for serving ads even if we attempt to disable most of them. Blocking most ads without breaking functionality is what I had in mind when creating this script. As it is, hosts files and dnsmasq have their own limitations. They can't block canvas and other js ads with clever and sneaky servers.

4 & 5. Could be the problem addressed above. I'll look into that list and add servers that are missing in my blocklist. Thank you.

--

VERSION: 20180221
The script now automatically downloads latest whitelist and blacklist files from the official Github repository if not found in /jffs/dnsmasq. What it means: You no longer need to download and extract the 7z archive for the script to work. Just get the script and everything it needs to function will be downloaded automatically. If you mess up your whilelist/blacklist files, just delete them and they will be auto downloaded.

EXPERIMENTAL: a self-updating[1] strain of the official script now exists for you to play with. mpadblock.sh This version will attempt to update itself if there's a new version available on Github.

[1]: self-modifying/self-updating scripts are not POSIX compliant and the desired outcome is not guaranteed.

DOWNLOADS:
1. 7z archive including the script and hosts/domain files
or
2. Script only
or
3. Experimental self-updating version
(You'll need to replace "adbhostgen.sh" with "mpadblock.sh" in instructions for it to work.)

_________________

Router
Netgear R8500: DD-WRT v3.0-r36840M kongac (09/03/18) / SFE: quarkysg - r34080-linux-4.4.105

Scripts
Ad-blocking on DD-WRT -- Github
kkdu
DD-WRT Novice


Joined: 20 Feb 2018
Posts: 3

PostPosted: Thu Feb 22, 2018 9:40    Post subject: Reply with quote
Noted Yamaraj. sent the details via PM ; have a look and advise me
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 95

PostPosted: Thu Feb 22, 2018 18:23    Post subject: Reply with quote
VERSION: 20180222
Added file size information for logging and debugging. Cleaned up code.

Read first post for instructions and download links.

_________________

Router
Netgear R8500: DD-WRT v3.0-r36840M kongac (09/03/18) / SFE: quarkysg - r34080-linux-4.4.105

Scripts
Ad-blocking on DD-WRT -- Github
0Legolas0
DD-WRT Novice


Joined: 14 Jan 2018
Posts: 12

PostPosted: Sun Feb 25, 2018 1:51    Post subject: My travails Reply with quote
Hi Yamaraj
I finally got adblocking working a few minutes ago. I followed your instructions, and had to do a bit of digging around to get it working. I have captured my learnings hoping others will benefit too.

Preparing USB Stick

- Used 32GB USB stick for my purpose
- [Could not get fdisk to work when I login to the router via telnet]
- Downloaded and followed instructions from: http://www.techovity.com/how-to-create-ext2ext3-linux-partition-from-windows/
- Used "MiniTool Partition Wizard" to create below partitions on Win10 PC (Took a few hours!! to complete the format & partioning)
/jffs in ext2 format
/opt in ext2 format
/data in ext2 format
[Linux Swap]
- Services--> USB enabled USB
- Plugged in USB-drive but was not recognized; Router Reboot Fixed it
- Mapped created UUID to /jffs and /opt in Services--> USB (/data had no option!!)

- telnet to router (username: root; passwd: <WebUI Passwd>)
- cd /jffs
- mkdir dnsmasq
- cd dnsmasq
- [Unsuccessful in getting wget and curl to download https://raw.githubusercontent.com/m-parashar/adbhostgen/master/adbhostgen.sh ]
- vi adbhostgen.sh (copy/paste contents of the script file, and saved it)
- chmod +x adbhostgen.sh
- Run the script ...
================================================================================
Code:
root@NetgearAC1750_DDWRT:/jffs/dnsmasq# ./adbhostgen.sh
Network up. Generating the hosts file now...
Downloading / updating cURL cacert for secure communication...
Whitelist and Blacklist files not found. Downloading latest defaults...
BLITZ mode OFF.
StevenBlack list
notracking list
Cameleon list
hosts-file ad_servers list
Disconnect.me list
quidsup/notrack list
MalwareDomains list
Securemecca list
abuse.ch blocklist
adaway list
Easylist & others
Processing whitelist and blacklist files...
Size of /tmp/mphosts.tmp before formatting: 13.2M
Size of /tmp/mpdomains.tmp before formatting: 6.1M
Removing duplicates and formatting the list of domains...
Removing temporary lists...
Size of /jffs/dnsmasq/mphosts after formatting: 9.3M
Size of /jffs/dnsmasq/mpdomains after formatting: 5.0M
335827 ad domains blocked.
Restarting DNS server (dnsmasq)...

================================================================================
Code:
root@NetgearAC1750_DDWRT:/jffs/dnsmasq# ls -l
-rwxr-xr-x    1 root     root         14272 Feb 24 17:13 adbhostgen.sh
-rw-r--r--    1 root     root          4648 Feb 24 17:14 blacklist
-rw-r--r--    1 root     root        223903 Feb 24 17:13 cacert.pem
-rw-r--r--    1 root     root       5198984 Feb 24 17:21 mpdomains
-rw-r--r--    1 root     root       9732132 Feb 24 17:19 mphosts
-rw-r--r--    1 root     root           797 Feb 24 17:21 mphosts.log
-rw-r--r--    1 root     root         15398 Feb 24 17:14 whitelist

================================================================================

- Services --> Services --> DNSMasq
- Enabled DNSMasq and options per OP's Post#1
-
- Administration --> Cron
- options per OP's Post#1

- Reboot
0Legolas0
DD-WRT Novice


Joined: 14 Jan 2018
Posts: 12

PostPosted: Sun Feb 25, 2018 4:06    Post subject: Unblock request Reply with quote
Hi
Unsure why sporting goods store is blocked. Unlucky that their name starts w/ dicks!! Smile

address=/dickssportinggoods.com/0.0.0.0
address=/dickssportinggoods.com/::

I added it to the whitelist but it is not working, do we need to reboot for this to take effect?

Thanks in advance
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 95

PostPosted: Sun Feb 25, 2018 6:47    Post subject: Re: Unblock request Reply with quote
0Legolas0 wrote:
Hi
Unsure why sporting goods store is blocked. Unlucky that their name starts w/ dicks!! :)

address=/dickssportinggoods.com/0.0.0.0
address=/dickssportinggoods.com/::

I added it to the whitelist but it is not working, do we need to reboot for this to take effect?

Thanks in advance


First things first, thanks for writing instructions. I'll update the first post with a link to yours.

Just add dickssportinggoods.com to whitelist and run the script again. It should work.

_________________

Router
Netgear R8500: DD-WRT v3.0-r36840M kongac (09/03/18) / SFE: quarkysg - r34080-linux-4.4.105

Scripts
Ad-blocking on DD-WRT -- Github
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 95

PostPosted: Sun Feb 25, 2018 6:59    Post subject: Re: My travails Reply with quote
0Legolas0 wrote:

- [Unsuccessful in getting wget and curl to download https://raw.githubusercontent.com/m-parashar/adbhostgen/master/adbhostgen.sh ]
- vi adbhostgen.sh (copy/paste contents of the script file, and saved it)


Code:
curl -s -k https://raw.githubusercontent.com/m-parashar/adbhostgen/master/adbhostgen.sh > adbhostgen.sh


Also, you should be using SSH, not telnet.

_________________

Router
Netgear R8500: DD-WRT v3.0-r36840M kongac (09/03/18) / SFE: quarkysg - r34080-linux-4.4.105

Scripts
Ad-blocking on DD-WRT -- Github
0Legolas0
DD-WRT Novice


Joined: 14 Jan 2018
Posts: 12

PostPosted: Wed Feb 28, 2018 16:31    Post subject: Outbrain, Taboola, etc Reply with quote
Hi Yamaraj

I see ads/sponsored content from Outbrain, Taboola, etc still showing up. Any ideas on how to block these type of "sponsored stories" sites?

In other words, help block all these ad-providers Smile

https://www.adpushup.com/blog/7-best-native-ad-networks-for-online-publishers/

Thanks in advance
mac913
DD-WRT Guru


Joined: 02 May 2008
Posts: 1375
Location: Canada

PostPosted: Thu Mar 01, 2018 1:03    Post subject: Reply with quote
Yamaraj wrote:
VERSION: 20180222
Added file size information for logging and debugging. Cleaned up code.

Read first post for instructions and download links.


I've jumped from the customized simple adblocker to a customized version of this script that fits my needs.

TIA for helping to filter the $hit!

_________________
Home Network on Telus PureFibre - Wired GigaBit Backbone
| 2x DNSCrypt-Proxy V2 | DHCPv4/v6 via DNSMasq | Telus-IPv6 |
| USB Storage - Scripts Only | 7x VLANs/BRs | WiFi 5x SSIDs |
| 3x OpenVPN-Clients | QoS | SSH | Entware | YAMon3 | Ad-Blocking |

R7800 Gateway & VPN1 - KONGAT 39855M Release 2019-05-25
R7800 WiFi & VPN2 & VPN3 - KONGAT 39855M Release 2019-05-25

______________________
YAMon 3.4 | Ad-Blocking for DD-WRT | DNSCrypt-Proxy V2
DD-WRT Beta Builds | KONG Builds
CloneVince's copies of Kong's Arm Firmware
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 95

PostPosted: Sun Mar 04, 2018 12:07    Post subject: Re: Outbrain, Taboola, etc Reply with quote
0Legolas0 wrote:
Hi Yamaraj

I see ads/sponsored content from Outbrain, Taboola, etc still showing up. Any ideas on how to block these type of "sponsored stories" sites?

In other words, help block all these ad-providers :)

https://www.adpushup.com/blog/7-best-native-ad-networks-for-online-publishers/

Thanks in advance

Some of these ads are nearly impossible to quash using only hosts and domain level blocking. JS and canvas based sneaky ads often and frequently even dodge browser level ad blockers. However, I have blacklisted a few of their servers in the past two days. I recommend that you update blacklist and whitelist files, or just get the latest 7z archive and update everything.

If you have a list of hosts/domains that I or others have missed, or would prefer to include them in the official distribution, please submit them here: https://github.com/m-parashar/adbhostgen/issues

mac913 wrote:
Yamaraj wrote:
VERSION: 20180222
Added file size information for logging and debugging. Cleaned up code.

Read first post for instructions and download links.


I've jumped from the customized simple adblocker to a customized version of this script that fits my needs.

TIA for helping to filter the $hit!

So glad to hear this little script is making web and browsing experience a little more tolerable.

_________________

Router
Netgear R8500: DD-WRT v3.0-r36840M kongac (09/03/18) / SFE: quarkysg - r34080-linux-4.4.105

Scripts
Ad-blocking on DD-WRT -- Github
Max Power
DD-WRT Novice


Joined: 27 Feb 2018
Posts: 25

PostPosted: Sun Mar 04, 2018 15:15    Post subject: Reply with quote
Yamaraj,

If we add a TLD (e.g. instagram.com) to the whitelist, will all subdomains be whitelisted as well? Or do we have to go through the hosts file to find every mention of *.instagram.com to add it to the whitelist?

Thanks!
mac913
DD-WRT Guru


Joined: 02 May 2008
Posts: 1375
Location: Canada

PostPosted: Sun Mar 04, 2018 20:52    Post subject: Reply with quote
Max Power wrote:
Yamaraj,

If we add a TLD (e.g. instagram.com) to the whitelist, will all subdomains be whitelisted as well? Or do we have to go through the hosts file to find every mention of *.instagram.com to add it to the whitelist?

Thanks!


Add to whitelist...

instagram.com

and run the script again.

_________________
Home Network on Telus PureFibre - Wired GigaBit Backbone
| 2x DNSCrypt-Proxy V2 | DHCPv4/v6 via DNSMasq | Telus-IPv6 |
| USB Storage - Scripts Only | 7x VLANs/BRs | WiFi 5x SSIDs |
| 3x OpenVPN-Clients | QoS | SSH | Entware | YAMon3 | Ad-Blocking |

R7800 Gateway & VPN1 - KONGAT 39855M Release 2019-05-25
R7800 WiFi & VPN2 & VPN3 - KONGAT 39855M Release 2019-05-25

______________________
YAMon 3.4 | Ad-Blocking for DD-WRT | DNSCrypt-Proxy V2
DD-WRT Beta Builds | KONG Builds
CloneVince's copies of Kong's Arm Firmware
mac913
DD-WRT Guru


Joined: 02 May 2008
Posts: 1375
Location: Canada

PostPosted: Sun Mar 04, 2018 21:18    Post subject: Re: Outbrain, Taboola, etc Reply with quote
Yamaraj wrote:
So glad to hear this little script is making web and browsing experience a little more tolerable.

This is a very welcome script and can easily be customized to your needs, THANKS!

_________________
Home Network on Telus PureFibre - Wired GigaBit Backbone
| 2x DNSCrypt-Proxy V2 | DHCPv4/v6 via DNSMasq | Telus-IPv6 |
| USB Storage - Scripts Only | 7x VLANs/BRs | WiFi 5x SSIDs |
| 3x OpenVPN-Clients | QoS | SSH | Entware | YAMon3 | Ad-Blocking |

R7800 Gateway & VPN1 - KONGAT 39855M Release 2019-05-25
R7800 WiFi & VPN2 & VPN3 - KONGAT 39855M Release 2019-05-25

______________________
YAMon 3.4 | Ad-Blocking for DD-WRT | DNSCrypt-Proxy V2
DD-WRT Beta Builds | KONG Builds
CloneVince's copies of Kong's Arm Firmware
Goto page Previous  1, 2, 3, 4 ... 15, 16, 17  Next Display posts from previous:    Page 3 of 17
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum