Ad-blocking on DD-WRT

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware
Goto page Previous  1, 2, 3 ... 13, 14, 15, 16, 17  Next
Author Message
Dr_K
DD-WRT User


Joined: 23 Mar 2018
Posts: 425

PostPosted: Mon Jun 04, 2018 22:45    Post subject: Reply with quote
@Yamaraj

Thank You for all your hard work you share with us
Also Thank You for being on top of issues and being up front with what is changing/going on

The labeling and retaining of older versions works perfect, and is greatly appreciated

Your script has been working flawlessly for my local network
Though had an issue on remote network with fb messenger on v 20180506a2 (which for some, was like the end of the world)
It probably was my fault, using my setup on theirs, running remotely late at night, though the log did not reflect that.
I need to find time to be on site with certain people to test before re-enabling script unfortunately


@totbl

You may be beating a dead horse

Did they finally add back curl to the v3.0 mega build for the Netgear WNDR3400?

My WNDR3400v2 & E1200v2 on build dd-wrt.v3.0-r35531_mega-nv64k (03/26/18 ) doesn't & has not for quite some time

Try typing curl in a ssh shell & see what you get
Code:
root@DD-WRT:~# curl

_________________
Location 1
R6300V2- DD-WRT v3.0-r39345M kongac (04-03-19) Gateway
WNDR3400v1 DD-WRT.v3.0-r35531_mega-nv64k (03/26/18 ) Access Point
WRT160Nv3 DD-WRT ?v3?.0-r35531 mini (03/26/18 ) Access Point
WRT54GSv5 DD-WRT v24-r33555_micro_generic (10/20/17) Repeater
Location 2
R6300V2- DD-WRT v3.0-r39345M kongac (04/03/19) Gateway
R6300V2- DD-WRT v3.0-r39345M kongac (04/03/19) Access Point
WNDR3700v2 DD-WRT v3.0-r35531 std (03/26/18 ) Access Point
E1200 v2 DD-WRT v3.0-r35531 mega-nv64k (03/26/18 ) Gateway(for trivial reasons)
2 devices: SXT 5 ac (mipsbe) RB 6.45.1 (06/27/19) PTP Bridge (0.8km/0.5mi)tx/rx866.6Mbps-1GbpsLAN

Thank You <Kong> & BrainSlayer for ALL that you do also to everyone here that shares their knowledge
Sponsor
totbl
DD-WRT Novice


Joined: 01 Jun 2018
Posts: 27

PostPosted: Tue Jun 05, 2018 2:58    Post subject: Reply with quote
Dr_K wrote:
@Yamaraj

Thank You for all your hard work you share with us
Also Thank You for being on top of issues and being up front with what is changing/going on

The labeling and retaining of older versions works perfect, and is greatly appreciated

Your script has been working flawlessly for my local network
Though had an issue on remote network with fb messenger on v 20180506a2 (which for some, was like the end of the world)
It probably was my fault, using my setup on theirs, running remotely late at night, though the log did not reflect that.
I need to find time to be on site with certain people to test before re-enabling script unfortunately


@totbl

You may be beating a dead horse

Did they finally add back curl to the v3.0 mega build for the Netgear WNDR3400?

My WNDR3400v2 & E1200v2 on build dd-wrt.v3.0-r35531_mega-nv64k (03/26/18 ) doesn't & has not for quite some time

Try typing curl in a ssh shell & see what you get
Code:
root@DD-WRT:~# curl


Hey, sorry dude. I actually got it mounted. The issue was the latest MEGA BUILD for k3x didn't have the ext drivers.
I swapped my firmware over to the latest k26 build and that has them. So that was def. the issue.

Thanks for looking out, tho. Hopefully my journey can help others. It'd be cool if there was a place to see what each build supports; if there is, I didn't come across it. Once again, I appreciate everything, folks. Smile
totbl
DD-WRT Novice


Joined: 01 Jun 2018
Posts: 27

PostPosted: Tue Jun 05, 2018 19:07    Post subject: Reply with quote
All right, so I'm finally trying to set this up and I'm up to step 2 where it tells me:
Code:

root@DD-WRT:/jffs/dnsmasq# ./adbhostgen.sh
>>> WARNING: cURL not found
>>> ERROR: ABORTING

Not sure why I'm getting that error. So far, this has been a job and a half to install this thing lol
I'm running into roadblock after roadblock.

I was able to get the installer to install as well, but not thru command. It's much easier to use a GUI interface, in my opinion. I used WINSCP so I can get a visual confirmation on everything.

Edit* Huh...according to the dude above I may not have cURL installed? wth lol
Is there like a definitive version of DD-WRT that has all the features? I'm literally using the BIG version of the latest build. I thought that was as feature-laden as it got.

I'm lost honestly.
Wickiman
DD-WRT Novice


Joined: 11 Jun 2015
Posts: 31
Location: Germany/Mexico/China

PostPosted: Fri Jun 15, 2018 14:01    Post subject: Reply with quote
Hello Yamaraj,

since my last message I haven't made any update.

I tried to stop the ad-blocker due to the world cup since ad-blockers are blocked and you cannot see the stream.

It looks like this when I try to pause the script.

root@Embassaden:/jffs/dnsmasq# ./adbhostgen.sh -?
Network up. Generating the hosts file now...
BLITZ mode OFF.
StevenBlack list
notracking list
Cameleon list
...

I need to update, correct?
there is no other way to pause the script?

Thanks
Wickiman

_________________
Netgear AC1900/R7000 DD-WRT v3.0-r37015M (09/23/18 ) kongac
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 95

PostPosted: Fri Jun 15, 2018 14:55    Post subject: Reply with quote
totbl wrote:
Edit* Huh...according to the dude above I may not have cURL installed? wth lol
Is there like a definitive version of DD-WRT that has all the features? I'm literally using the BIG version of the latest build. I thought that was as feature-laden as it got.

I'm lost honestly.

You need curl installed for the script to work. The static version we tried wasn't successful, so I recommend you look into entware/optware.

Wickiman wrote:
Hello Yamaraj,

since my last message I haven't made any update.

I tried to stop the ad-blocker due to the world cup since ad-blockers are blocked and you cannot see the stream.

It looks like this when I try to pause the script.

root@Embassaden:/jffs/dnsmasq# ./adbhostgen.sh -?
Network up. Generating the hosts file now...
BLITZ mode OFF.
StevenBlack list
notracking list
Cameleon list
...

I need to update, correct?
there is no other way to pause the script?

Thanks
Wickiman

Looks like you are still using an ancient and slightly buggy version of it. Please update immediately.

_________________

Router
Netgear R8500: DD-WRT v3.0-r36840M kongac (09/03/18) / SFE: quarkysg - r34080-linux-4.4.105

Scripts
Ad-blocking on DD-WRT -- Github
chorton
DD-WRT Novice


Joined: 06 Jul 2006
Posts: 12

PostPosted: Sun Jul 22, 2018 3:47    Post subject: Reply with quote
I don't know if it's just me but I updated to the latest stable version of Kong for for R7000, and when I run adbhosten.sh it shows:

# VERSION: 20180601a2
ping: bad address 'google.com'
# NETWORK: DOWN | MODE: OFFLINE


It worked fine prior to upgrading firmwware. Has anyone else had this issue?
Charcoal
DD-WRT Novice


Joined: 26 Oct 2017
Posts: 9

PostPosted: Mon Jul 23, 2018 21:18    Post subject: Reply with quote
It doesn't work for me with the latest Kong build (36325M) as well.
When processing the final files, it shows "segmentation fault" in Putty and the number of ad domains blocked is "0".
Back to build 36070M and everything is working perfectly fine.
redson
DD-WRT Novice


Joined: 14 May 2018
Posts: 10

PostPosted: Tue Jul 24, 2018 21:02    Post subject: Reply with quote
Thanks so much for the script! Just got this up and running and it's incredibly useful.

I'm still trying to figure out whitelisting for a few sites, and more generally how to debug in case it comes up again. This site at sfgate works fine on desktop safari & chrome on my mac desktop (high sierra 10.13.6), but doesn't work through mobile safari on iOS.

If I click on that link, the browser hangs for ~4 minutes before loading the page contents.



after enabling dnsmasq logging, this is what I see for the four minutes before the page loads:
Code:
root@router:~# tail -f /jffs/dnsmasq/dnsmasq.log | grep 0.1.2.3
Jul 24 13:52:41 dnsmasq[10165]: config treg.hearstnp.com is 0.1.2.3
Jul 24 13:52:41 dnsmasq[10165]: /jffs/dnsmasq/mphosts ak.sail-horizon.com is 0.1.2.3
Jul 24 13:52:41 dnsmasq[10165]: /jffs/dnsmasq/mphosts cdn.blueconic.net is 0.1.2.3
Jul 24 13:52:42 dnsmasq[10165]: /jffs/dnsmasq/mphosts g.3gl.net is 0.1.2.3
Jul 24 13:52:42 dnsmasq[10165]: config aps.hearstnp.com is 0.1.2.3
Jul 24 13:53:34 dnsmasq[10165]: /jffs/dnsmasq/mphosts e.crashlytics.com is 0.1.2.3
Jul 24 13:53:49 dnsmasq[10165]: /jffs/dnsmasq/mphosts js.adsrvr.org is 0.1.2.3
Jul 24 13:53:49 dnsmasq[10165]: /jffs/dnsmasq/mphosts secure-us.imrworldwide.com is 0.1.2.3
Jul 24 13:53:49 dnsmasq[10165]: /jffs/dnsmasq/mphosts dart.l.doubleclick.net is 0.1.2.3
Jul 24 13:53:49 dnsmasq[10165]: /jffs/dnsmasq/mphosts stats.l.doubleclick.net is 0.1.2.3
Jul 24 13:53:50 dnsmasq[10165]: /jffs/dnsmasq/mphosts snap.adsrvr.org is 0.1.2.3
Jul 24 13:53:50 dnsmasq[10165]: config so.rlcdn.com is 0.1.2.3
Jul 24 13:53:50 dnsmasq[10165]: /jffs/dnsmasq/mphosts x.skimresources.com is 0.1.2.3
Jul 24 13:53:50 dnsmasq[10165]: config r.dlx.addthis.com is 0.1.2.3
Jul 24 13:53:50 dnsmasq[10165]: /jffs/dnsmasq/mphosts pixel.tapad.com is 0.1.2.3
Jul 24 13:53:50 dnsmasq[10165]: config sync.graph.bluecava.com is 0.1.2.3
Jul 24 13:53:56 dnsmasq[10165]: config treg.hearstnp.com is 0.1.2.3
Jul 24 13:53:57 dnsmasq[10165]: /jffs/dnsmasq/mphosts ak.sail-horizon.com is 0.1.2.3
Jul 24 13:53:57 dnsmasq[10165]: /jffs/dnsmasq/mphosts cdn.blueconic.net is 0.1.2.3
Jul 24 13:54:38 dnsmasq[10165]: /jffs/dnsmasq/mphosts e.crashlytics.com is 0.1.2.3
Jul 24 13:54:44 dnsmasq[10165]: /jffs/dnsmasq/mphosts stats.l.doubleclick.net is 0.1.2.3
Jul 24 13:54:44 dnsmasq[10165]: /jffs/dnsmasq/mphosts s.btstatic.com is 0.1.2.3
Jul 24 13:55:00 dnsmasq[10165]: /jffs/dnsmasq/mphosts e.crashlytics.com is 0.1.2.3
Jul 24 13:55:12 dnsmasq[10165]: /jffs/dnsmasq/mphosts nexus.ensighten.com is 0.1.2.3
Jul 24 13:55:12 dnsmasq[10165]: /jffs/dnsmasq/mphosts widget.perfectmarket.com is 0.1.2.3
Jul 24 13:55:12 dnsmasq[10165]: /jffs/dnsmasq/mphosts gum.criteo.com is 0.1.2.3
Jul 24 13:55:12 dnsmasq[10165]: /jffs/dnsmasq/mphosts sb.scorecardresearch.com is 0.1.2.3
Jul 24 13:56:27 dnsmasq[10165]: /jffs/dnsmasq/mphosts trc.taboola.com is 0.1.2.3
Jul 24 13:56:27 dnsmasq[10165]: config api.viafoura.com is 0.1.2.3
Jul 24 13:56:27 dnsmasq[10165]: /jffs/dnsmasq/mphosts www.googletagmanager.com is 0.1.2.3


note that even when the content loads, the progress bar indicates it's still downloading.

Would you recommend white-listing all of these domains to get the page to work? Is there a way to drill down into which domains are causing problems on the phone? Thanks!

Configuration
    * R7000
    * Kong build 36070M
    * script on 32GB pen drive (/jffs & /opt on 4GB ext4 partitions with journaling disabled)

logging enabled for dnsmasq:
Code:
log-queries 
log-facility=/jffs/dnsmasq/dnsmasq.log 
log-async 
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 95

PostPosted: Wed Jul 25, 2018 21:53    Post subject: Reply with quote
redson wrote:
Thanks so much for the script! Just got this up and running and it's incredibly useful.

I'm still trying to figure out whitelisting for a few sites, and more generally how to debug in case it comes up again. This site at sfgate works fine on desktop safari & chrome on my mac desktop (high sierra 10.13.6), but doesn't work through mobile safari on iOS.

...

The best way to find domains to whitelist is to install DNSthingy Assistant Chrome/<browser> extension and carefully curating the list of domains before whitelisting them. SFgate works fine on my systems, but that may be due to whitelisted domains in my personal mywhitelist file.

_________________

Router
Netgear R8500: DD-WRT v3.0-r36840M kongac (09/03/18) / SFE: quarkysg - r34080-linux-4.4.105

Scripts
Ad-blocking on DD-WRT -- Github
redson
DD-WRT Novice


Joined: 14 May 2018
Posts: 10

PostPosted: Wed Jul 25, 2018 22:02    Post subject: Reply with quote
Quote:
The best way to find domains to whitelist is to install DNSthingy Assistant Chrome/<browser> extension and carefully curating the list of domains before whitelisting them. SFgate works fine on my systems, but that may be due to whitelisted domains in my personal mywhitelist file.


Thanks for the response! I appreciate the repsonse.

I've been digging into this non-stop, my current theory is the problem is actually related to https timeouts as discussed here. Am having trouble with ip6tables, will report back if I'm successful.
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 95

PostPosted: Wed Jul 25, 2018 22:13    Post subject: Reply with quote
chorton wrote:
I don't know if it's just me but I updated to the latest stable version of Kong for for R7000, and when I run adbhosten.sh it shows:

# VERSION: 20180601a2
ping: bad address 'google.com'
# NETWORK: DOWN | MODE: OFFLINE


It worked fine prior to upgrading firmwware. Has anyone else had this issue?

Charcoal wrote:
It doesn't work for me with the latest Kong build (36325M) as well.
When processing the final files, it shows "segmentation fault" in Putty and the number of ad domains blocked is "0".
Back to build 36070M and everything is working perfectly fine.

Yes, the latest Kong build 36325M isn't playing nice with the script. Likely due to the recent BusyBox fixes/bugs. Last known good build is 36070M, and I recommend you stay on it until the bugs are sorted out.

For reference:
Code:
LC_ALL=C cat $tmphosts | sed -r 's/^[[:blank:]]*//; s/[[:blank:]]*$//; /^$/d; /^\s*$/d' | cat tmpbl - | grep -Fvwf tmpwl | sort -u | awk -v "IP=$ADHOLEIP" '{sub(/\r$/,""); print IP" "$0}' > $mphosts
is throwing the segfault. In particular, the grep -Fvwf tmpwl part. I haven't had enough time to look into BusyBox 1.29.0, but the recent updates might have something to do with it.

In the meantime, I have updated the script. Note that it still does not work with 36325M.


ROLLING UPDATE: 20180725a1

1. Gracefully handles network unavailability and OFFLINE mode.
2. Easter egg removed, script now counts mpdomains too, better parsing.


How to update if you are on 20180315 or later:
SSH into router
cd /jffs/dnsmasq
./adbhostgen.sh -u

If you are still using an older version, download https://raw.githubusercontent.com/m-parashar/adbhostgen/master/adbhostgen.sh into /jffs/dnsmasq and chmod +x it before running.

_________________

Router
Netgear R8500: DD-WRT v3.0-r36840M kongac (09/03/18) / SFE: quarkysg - r34080-linux-4.4.105

Scripts
Ad-blocking on DD-WRT -- Github
redson
DD-WRT Novice


Joined: 14 May 2018
Posts: 10

PostPosted: Thu Jul 26, 2018 17:56    Post subject: Reply with quote
It seems in 20180725a1, offline mode doesn't actually do anything? I would expect it to add a specified domain to the whitelist without re-downloading & processing lists.

instead it looks like it bails when it realizes it's offline:

Code:
root@router:/jffs/dnsmasq# ./adbhostgen.sh  -0 -o --wl=tags.tiqcdn.com
======================================================
|                adbhostgen for DD-WRT               |
|      https://github.com/m-parashar/adbhostgen      |
|           Copyright 2018 Manish Parashar           |
======================================================
             Thu Jul 26 10:53:57 PDT 2018
# VERSION: 20180725a1
# NETWORK: DOWN | MODE: OFFLINE
root@router:/jffs/dnsmasq#
Darias
DD-WRT Novice


Joined: 17 Sep 2016
Posts: 2

PostPosted: Thu Jul 26, 2018 18:44    Post subject: Reply with quote
Please check dead host at
https://github.com/dead-hosts
Replace alive this url
https://adzhosts.hizo.fr/hosts/adzhosts-android.txt
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt

My log

Code:
> Processing blacklist/whitelist files
> Processing final mphosts/mpdomains files
Segmentation fault
# Size of /jffs/h: 0
# Size of /jffs/d: 3.3M
# Number of ad hosts blocked: approx 0
# Number of ad domains blocked: approx 103306
> Restarting DNS server (dnsmasq)
> Deleting: /jffs/tmp
# Total time: 9:25 minutes
# DONE


Using "sort -u" to sort bigfile => "Segmentation fault"
Replace this code to fix it
Code:
awk '{if ($1 in a) next; a[$1]=$0; print}'


Invallid domain do not fix yet
Code:

0.1.2.3 mailto:info@mypornbible.com
0.1.2.3 st.adxxx.o0.0.0.0
0.1.2.3 addons.sync.maxthon.
0.1.2.3 adspayformymortgage.win.
0.1.2.3 os.uptodowncdn.com.
0.1.2.3 www.forgotmyemailpassword.com.
0.1.2.3 poolnoodle.tech.
0.1.2.3 abbp1.website.
0.1.2.3 abbp1.space.
0.1.2.3 hashbag.cc:8688
0.1.2.3 girlsandxxx1.comf4a
0.1.2.3 igrid.org^
0.1.2.3 www.just-anchor.com?
0.1.2.3 Found</title>

Please search with keyword:
Code:
( ) < > ? & : # @ com.*$ \.$ ^\.

To find invallid domain and fix it



h.log.txt
 Description:
Log

Download
 Filename:  h.log.txt
 Filesize:  4.96 KB
 Downloaded:  142 Time(s)

Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 95

PostPosted: Fri Jul 27, 2018 5:55    Post subject: Reply with quote
redson wrote:
It seems in 20180725a1, offline mode doesn't actually do anything? I would expect it to add a specified domain to the whitelist without re-downloading & processing lists.

instead it looks like it bails when it realizes it's offline:

Code:
root@router:/jffs/dnsmasq# ./adbhostgen.sh  -0 -o --wl=tags.tiqcdn.com
======================================================
|                adbhostgen for DD-WRT               |
|      https://github.com/m-parashar/adbhostgen      |
|           Copyright 2018 Manish Parashar           |
======================================================
             Thu Jul 26 10:53:57 PDT 2018
# VERSION: 20180725a1
# NETWORK: DOWN | MODE: OFFLINE
root@router:/jffs/dnsmasq#

That was the idea, but the last update broke it. Should be fixed now. Update and let me know if it still doesn't work.

_________________

Router
Netgear R8500: DD-WRT v3.0-r36840M kongac (09/03/18) / SFE: quarkysg - r34080-linux-4.4.105

Scripts
Ad-blocking on DD-WRT -- Github
Yamaraj
DD-WRT User


Joined: 25 Jan 2017
Posts: 95

PostPosted: Fri Jul 27, 2018 6:04    Post subject: Reply with quote
Darias wrote:
Replace alive this url
https://adzhosts.hizo.fr/hosts/adzhosts-android.txt
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt

Using "sort -u" to sort bigfile => "Segmentation fault"
Replace this code to fix it
Code:
awk '{if ($1 in a) next; a[$1]=$0; print}'


Invallid domain do not fix yet
Code:

0.1.2.3 mailto:info@mypornbible.com
0.1.2.3 st.adxxx.o0.0.0.0
0.1.2.3 addons.sync.maxthon.
0.1.2.3 adspayformymortgage.win.
0.1.2.3 os.uptodowncdn.com.
0.1.2.3 www.forgotmyemailpassword.com.
0.1.2.3 poolnoodle.tech.
0.1.2.3 abbp1.website.
0.1.2.3 abbp1.space.
0.1.2.3 hashbag.cc:8688
0.1.2.3 girlsandxxx1.comf4a
0.1.2.3 igrid.org^
0.1.2.3 www.just-anchor.com?
0.1.2.3 Found</title>

Please search with keyword:
Code:
( ) < > ? & : # @ com.*$ \.$ ^\.

To find invallid domain and fix it

Fixed:
1. Dead links.
2. Segfault. The update works fine in Kong build r36400M. At least on R8500. If it doesn't on yours, replace "sort -u" with "awk '!a[$0]++'".
3. I have introduced more filters to weed out some of these invalid domains. The idea is to format and adapt existing lists to get the job done with minimal code base. Some of these lists often introduce incomplete or invalid domains and fix it in subsequent updates. I will keep an eye on these and make suitable changes to take care of it.

_________________

Router
Netgear R8500: DD-WRT v3.0-r36840M kongac (09/03/18) / SFE: quarkysg - r34080-linux-4.4.105

Scripts
Ad-blocking on DD-WRT -- Github
Goto page Previous  1, 2, 3 ... 13, 14, 15, 16, 17  Next Display posts from previous:    Page 14 of 17
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum