QOS Traffic Monitor w/ Reporting by Day, Host and IP

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Contributions Upload
Goto page 1, 2  Next
Author Message
Howt
DD-WRT Novice


Joined: 26 Aug 2012
Posts: 6

PostPosted: Sun Aug 26, 2012 5:19    Post subject: QOS Traffic Monitor w/ Reporting by Day, Host and IP Reply with quote
About a week back, I found alexatkinuk's post detailing his ip_conntrack with QoS and bandwidth use reporting.
http://www.dd-wrt.com/phpBB2/viewtopic.php?t=84442&postdays=0&postorder=asc&start=0

I love the thing. So much so that I decided to enhance it. My thanks to the original author.

What's different?

This enhancement - QOS Traffic Monitor - provides for persistent periodic bandwidth reporting by day, host, IP address, or combination thereof. It does not display anything within DD-WRT. Rather, it provides JavaScript data for use in HTML reports.
    Daily reporting in addition to point in time.
    Less write wear on flash memory - 25 writes/day v. 86,400 (defaults).
    Less CPU intensive (simply runs less often).
    No data loss upon GUI/Telnet/SSH reboot.
    HTML reports with Javascript library for further development.
    Hosts defined by DHCP static leases and/or external file.
Everything in the original work by alexatkinuk is retained including the slick MyPage display incorporated into the DD-WRT menus. On the downside, the MyPage display is less responsive. The objective of QOS-TM is geared more towards reporting.

How to install?

A USB drive, JFFS2, or SD/MMC memory is required. A script performs the actual installation. Then, using the DD-WRT GUI, a Startup script, Shutdown script, and Cron Job are defined.

The read-me text file provides the details and can be downloaded here:
http://dl.dropbox.com/u/6624068/QOS_TM_READ_ME.txt

If you would like to examine the setup package (not required as it can be downloaded from within a Telnet/SSH session) is available here:
http://dl.dropbox.com/u/6624068/traff_setup.tar.gz

For accurate identification of hosts, it is strongly suggested that Static Leases and Use JFFS2 for client lease DB be employed. Both are in the DHCP Server section of the Services | Services page. Hosts can also be defined externally in the file traffic_extra_hosts.js.

Confused?

A picture is worth 1,000 words. Here is a screenshot from a demo HTML page. It provides a glimpse of the data arrays made available by QOS-TM. Three "canned" reports are also included and shown in the next post.



In a nutshell, arrays of Javascript data are generated which include:
Code:
USERS[ mac ] = [ ip, hostname, up-bytes, down-bytes ];
DAYS[ date ] = [ up-bytes, down-bytes ];
IPS[ ip ]    = [ mac, up-bytes, down-bytes ];
RAW[ 0... ]  = [ router-file, date, mac, ip, up-bytes, down-bytes ];
RANGE[ 0...] = [ start date, end date ];

As well, a collection of JavaScript functions make coding your own HTML reports a fairly simple task. See the read-me.

Operation.

Once installed, it runs unattended. If MTD reports suffice for your needs, nothing else need be done. The only maintenance required is deleting older data files when no longer required.

The full extent of the possibilities are detailed in the read-me. It is suggested reading.

Important Note: QOS-TM uses data file modification times to assign bandwidth to the correct dates. Thus, you should not modify the files traffic_YYYY-MM-DD.asp or traffic_YYYY-MM-DD_HH:MM:SS.asp. If you do, then you can use the touch command to restore the correct modification time.

Effort has been put into making the installation and use as easy as possible. It's important that we be nice to the newbs. Very Happy

I hope this proves useful for everyone. Let me know if any problems arise... but I'm sure you will.

_________________
Asus RT-N16 DDWRT v24-sp2 (11/21/10) big Build 15778


Last edited by Howt on Sat Sep 01, 2012 19:09; edited 3 times in total
Sponsor
Howt
DD-WRT Novice


Joined: 26 Aug 2012
Posts: 6

PostPosted: Mon Aug 27, 2012 20:23    Post subject: Reply with quote
Good to go!

The issues mentioned previously in this post have been corrected and everything is good to go as detailed in the original post above.

These are the three reports included with QOS-TM. For info on the report options and tools for creating your own reports, see REPORTING.txt in the setup distribution file.

Summary report -
By MAC by Day -
By IP by Day -

My apologies for any inconvenience.

_________________
Asus RT-N16 DDWRT v24-sp2 (11/21/10) big Build 15778


Last edited by Howt on Tue Aug 28, 2012 23:54; edited 1 time in total
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 131
Location: Sheffield, UK

PostPosted: Tue Aug 28, 2012 20:07    Post subject: Reply with quote
Thanks, but I suspect the bits of my scripts you are using are the same bits I "borrowed" from wrtbwmonitor. So I'm not sure if I deserve any credit.

Also I just wanted to leave a word of warning. I have found that using iptables to do bandwidth reporting on a per host basis eats a LOT of CPU power as traffic increases.

While I have not been using my DD-WRT script for a while due to switching to OpenWRT, I still use the same iptables rules to capture the data on there.

I have found that on my Buffalo router it would completely max out SIRQs when doing over 40Mbit of traffic, with or without QoS turned on.

So now I am switching to an Atom instead and will be doing something similar to the script on here, except I will just periodically offload the current traffic stats to my NAS over MySQL.

So, if you have problems with the router lagging, DNS failing, then it might be that your router can't handle the traffic monitoring. I was struggling with it for a week or so before it occurred to me to check removing the iptables rules for traffic monitoring.

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/
Howt
DD-WRT Novice


Joined: 26 Aug 2012
Posts: 6

PostPosted: Wed Aug 29, 2012 0:52    Post subject: Reply with quote
Thanks for the heads up.

My default installation has traffic_monitor.sh set to sleep 10 seconds between executions versus the original 1 second. For my purposes, it could go as high as 60 seconds since I am more interested in the reports than real-time feedback via the MyPage display... however cool it is Very Happy ... and it is.

That delay is stored in an NVRAM variable making changes a tad easier. The NVRAM variable is qos_delay. If QOS-TM is running, it can be shut down with the script traff_off.sh, the NVRAM value set and committed, and the script restarted with traff_on.sh. Data loss would be negligible.

Closely related, data from those executions is saved to regular memory (/tmp/data/traffic.asp) rather than risking wear on the flash memory potentially damaging it. Every so many iterations, it is copied to persistent flash memory. That too is stored in an NVRAM variable qos_cycles. A restart is also required should this value change.

This results in 25 writes/day to flash memory versus the original 86,400.

Of greater interest to me is how often the data is saved to flash memory. That is, should the router reboot unexpectedly, what is the maximum potential data loss? That is calculated by qos_delay times qos_cycles. Using the default settings: 10 seconds x 360 = 3600 seconds = 1 hour. The average loss would be 30 minutes.

NOTE: Running on a UPS, I don't recall the router rebooting unexpectedly in... well ever really. I experience months of consistent up time and eventually will have cause to reboot it myself.

Rebooting is a negligible data loss issue if performed from the DD-WRT GUI (or reboot from Telnet.SSH). In that case, the Shutdown and Startup scripts insure that no more than qos_delay (10 seconds) of data is lost.

The trade-offs become... MyPage refresh time v. data safety v. flash memory wear v. CPU demand.

I did notice an increase in CPU use with the original 1 second sleep time. Thus, the increase to 10 seconds. I haven't seen excessive CPU use since but... I have a relatively speedy Asus RT-N16 router (Broadcom BCM4718 CPU @ 480Mhz), traffic through it hasn't been particularly heavy, and my maximum download speed is "only" 30Mbit.

As we discussed off-board, your Atom CPU solution is vastly preferable. But, for a residential application tracking just a handful of hosts, this has proven more than sufficient thus far.

Thanks.

_________________
Asus RT-N16 DDWRT v24-sp2 (11/21/10) big Build 15778


Last edited by Howt on Fri Aug 31, 2012 23:38; edited 2 times in total
alexatkinuk
DD-WRT User


Joined: 07 Dec 2010
Posts: 131
Location: Sheffield, UK

PostPosted: Wed Aug 29, 2012 1:27    Post subject: Reply with quote
The problem I encountered wasn't related to the monitoring script at all, it happens even when its not running.

The problem seems to be that passing the packets through the RRDIPT chain exhausted the software interrupt ability of the routers CPU.

I'm guessing this is due to the fact its not just counting the traffic, its actually routing it around memory to be able to count it and that uses CPU power. Of course, I might be wrong as iptables is pretty much voodoo magic to me.

_________________
2xWZR-HP-G300NH(B) (B0 B0) DD-WRT v24-sp2 (06/14/11) std 17201
One antenna swapped for an RP-SMA connector and 14dB external Yagi.
http://csdprojects.co.uk/ddwrt/
mikeones
DD-WRT Novice


Joined: 02 May 2009
Posts: 7

PostPosted: Fri Aug 31, 2012 1:59    Post subject: Reply with quote
Hi, this traffic monitor add-on is awesome! Great work and IMO its a much cleaner solution compared to the other per host traffic monitoring solutions I have tried on wrt.

I have been able to get your work going without much hassle and MyPage.asp is being populated with host data, but when I access the reporting page at http://ip/user/traffic_summary.html, the page is blank except for the summary text title page.

The version I am running is http://dl.dropbox.com/u/6624068/traff_setup.tar.gz and my browser says v0.13 in the tab. Is this the latest version? Any idea why the summary page would be blank?

I noticed these three js files are included.
Code:
root@e3000:~# ls /jffs/www/js/
qos_conntrack.js        traffic_extra_hosts.js  traffic_process.js


The html file for the summary page lists a js file that is not in the above directory... could the missing file be causing my issue?
Code:
  <script type="text/javascript" src="js/traffic.js"></script>
  <script type="text/javascript" src="js/traffic_extra_hosts.js"></script>
  <script type="text/javascript" src="js/traffic_process.js"></script>
Howt
DD-WRT Novice


Joined: 26 Aug 2012
Posts: 6

PostPosted: Fri Aug 31, 2012 18:47    Post subject: Reply with quote
Thanks. I'm glad you like it. I believe the problem is .../js/traffic.js missing.

That file is created at 11:59pm by the default cron job. My guess is that you simply haven't had that occur yet.

You can force the issue by running the script manually.
Code:
cd /opt  (or, whatever mount point)
./traff_2js.sh

It may be more convenient to enter the command from the DD-WRT GUI... "/opt/traff_2js.sh". See also: READ_ME.txt and REPORTING.txt.

Let me know if that resolves the issue. Otherwise, it's one dumb-arse thing or another and we'll figure it out.

"v0.13" indicates the version of alexatkinuk's original monitor upon which this is built. See also:
http://www.dd-wrt.co.in/phpBB2/viewtopic.php?p=518497&highlight=&sid=f1ba6aab98c8d7600598c8ba372ce836

_________________
Asus RT-N16 DDWRT v24-sp2 (11/21/10) big Build 15778
Howt
DD-WRT Novice


Joined: 26 Aug 2012
Posts: 6

PostPosted: Sun Sep 02, 2012 13:36    Post subject: Corrected Script Reply with quote
The script traff_2js.sh has been updated to correct an error calculating bandwidth. The setup package (see first post) has been corrected. If you have already installed QOS-TM, it may be easier to download the one script which has changed.
http://dl.dropbox.com/u/6624068/traff_2js.sh

EXPLAIN: The error was in calculating bandwidth for the first day of a reporting period. The script assumed the router always reboots at the start of a period and would mark the data as such. Obviously, that is wrong.

_________________
Asus RT-N16 DDWRT v24-sp2 (11/21/10) big Build 15778
wollyka
DD-WRT Novice


Joined: 16 Oct 2010
Posts: 3

PostPosted: Sat Oct 13, 2012 17:20    Post subject: Reply with quote
Hi
Thanks for the script but i am having problems running it.
I have a Linksys E4200v1 with DD-WRT v24-sp2 (07/20/12) big - build 19519
I followed the instructions but i am always getting a blank page with going to http://ip/user/traffic_summary.html. I only get the title.
Also i noticed that My_Page is not showing the connections but it is stuck on calculating...

I tried the original script by alexatkinuk and is working fine.. But i need periodic data monitoring with few data loss that your script provides.
Any help is appreciated
Thanks

Edit: the js error i am getting when running the traffic_summary.html report is:
Code:
ReferenceError: hosts is not defined
Source File: http://192.168.2.200/user/js/traffic_process.js
Line: 28
jozwikjp
DD-WRT Novice


Joined: 25 Jun 2008
Posts: 5

PostPosted: Wed Oct 17, 2012 16:29    Post subject: Reply with quote
Hello,

I have installed the script and have manually ran the traff_2js.sh script and verified that the process is running.

But I am getting various errors saying "bad number"
here is the output. Could this be a space somewhere or something.
Code:

root@DD-WRT:/jffs# sh traff_2js.sh
sh: bad number
root@DD-WRT:/jffs# sh traff_info.sh

=======================================================
=== QOS Traffic Reporting: Status information.      ===
=======================================================
RUNNING as process 4154.
sh: jffs: bad number
Valid mount point found: /jffs
Data directory found: /jffs/traff
Web directory found: /jffs/www
Javascript directory found: /jffs/www/js
CSS directory found: /jffs/www/css
MyPage directory found: /jffs/MyPage
NVRAM current settings:
   qos_mount="jffs"  [optional argument 1]
   wshaper_uplink=10000  [optional argument 2]
   wshaper_downlink=50000  [optional argument 3]
   mypage_scripts="/jffs/MyPage/qos_conntrack.sh"
   qos_cycles=360  [will be set to 360 cycles]
   qos_delay=10  [will be set to 10 seconds]

root@DD-WRT:/jffs# sh traff_info.sh


ebagnoli
DD-WRT Novice


Joined: 14 Jan 2013
Posts: 1

PostPosted: Tue Jan 15, 2013 15:30    Post subject: Is it possible on HDD Reply with quote
Hi all:

I have tried to follow the tutorial and installed the script in /mnt.

But I have an HDD mounted and not an USB stick. Is it possible to load the script on HDD and what is the syntax?

Tks
tmphuc
DD-WRT Novice


Joined: 16 Feb 2013
Posts: 1

PostPosted: Sat Feb 16, 2013 3:44    Post subject: Fixed Reply with quote
wollyka wrote:
Hi
Thanks for the script but i am having problems running it.
I have a Linksys E4200v1 with DD-WRT v24-sp2 (07/20/12) big - build 19519
I followed the instructions but i am always getting a blank page with going to http://ip/user/traffic_summary.html. I only get the title.
Also i noticed that My_Page is not showing the connections but it is stuck on calculating...

I tried the original script by alexatkinuk and is working fine.. But i need periodic data monitoring with few data loss that your script provides.
Any help is appreciated
Thanks

Edit: the js error i am getting when running the traffic_summary.html report is:
Code:
ReferenceError: hosts is not defined
Source File: http://192.168.2.200/user/js/traffic_process.js
Line: 28



Got the same issue. After a while reading the code, I come up with this solution for you:

In DD-WRT, go to Services tab and try to add at least 1 Static DHCP Lease.

traff_2js.sh got a bug at line 'nvram get static_leases'. This will return null if no static lease assigned so it makes the js file has syntax error.

I've made a small change in the qos_conntrack.sh file so it will links to these html pages: https://www.dropbox.com/s/xd3wvdowao1jgc4/qos_conntrack.sh

To make the Logging page works, try ssh into the router and run:
ln -s /var/log/message /www/user/log.html

Also add this to Wirewall Command (Administrations tab --> Command):
echo "<pre>" | cat - /var/log/messages > /tmp/out && mv /tmp/out /var/log/messages

Hope this help.
staktrace
DD-WRT Novice


Joined: 07 Dec 2013
Posts: 1

PostPosted: Sat Dec 07, 2013 11:58    Post subject: Reply with quote
Thanks a lot for this, it's something I've been wanting for a while!

Also, just so you know, there is a typo in the traff_install.sh script that caused it to fail for me. The following line:

chmod +x ./$MyPage/*.sh

has an extra $ and really should be this:

chmod +x ./MyPage/*.sh
z7ths
DD-WRT Novice


Joined: 06 Feb 2018
Posts: 1

PostPosted: Tue Feb 06, 2018 15:29    Post subject: Reply with quote
All the DropBox links are not working.

does anybody have a copy of this script and could maybe post it / attach it / upload it?

Thank you!
KittyChampion
DD-WRT Novice


Joined: 19 Sep 2017
Posts: 23

PostPosted: Fri Feb 09, 2018 14:52    Post subject: Reply with quote
z7ths wrote:
All the DropBox links are not working.

does anybody have a copy of this script and could maybe post it / attach it / upload it?

Thank you!


Use YAMon
Goto page 1, 2  Next Display posts from previous:    Page 1 of 2
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Contributions Upload All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum