Per-user Live Down/Upload Rate Monitoring [No PC needed!]

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Contributions Upload
Goto page Previous  1, 2, 3, 4, 5, 6 ... 10, 11, 12  Next
Author Message
aanev
DD-WRT Novice


Joined: 12 Jan 2009
Posts: 7

PostPosted: Tue Oct 05, 2010 10:16    Post subject: Reply with quote
I saw many changes were talked here. Can someone please upload fixed accurate version of wrtbwmon that use 'cat' instead of using 'sort -n'.Thanks.
Sponsor
RobotBanana
DD-WRT Novice


Joined: 15 Jul 2010
Posts: 22

PostPosted: Tue Oct 05, 2010 15:40    Post subject: Reply with quote
FezJay wrote:
Im not sure, but it could have somthing to do with the temp files that are created (maybe I should delete them to).


As far as I know, the temp files are completely overwritten with the amount of data transferred since the last pass-through of the script. Conveniently, this allows you to delete/move last month's usage.db, but not lose any of the bandwidth info that passes through the router while you're doing so. But it also means that other than the very first write to the new usage.db, you shouldn't get any residual bandwidth from last month being put into your current month's usage.db.

How much data is measured for it to go 150mb over, and what are you comparing this "overage" to that makes you deem it an inaccuracy? It is possible that the 150mb is overhead. I can't think of any other explanation? Confused If it is indeed overhead, then you probably DO want that to be in your bandwidth count since your ISP will include it on your bill anyway. Certain things, like Torrents, use a crazy amount of overhead. The bandwidth measurement wouldn't be anywhere close to the size of the file(s) you were downloading.
FezJay
DD-WRT User


Joined: 11 Sep 2010
Posts: 178

PostPosted: Wed Oct 06, 2010 1:49    Post subject: Reply with quote
Hey

Firstly, from your answer, I should just leave the traffic files alone and continue to remove the usage.db file only?

Secondly, I should have mentioned what I am comparing it against.

Basically, after I call the custom reset function I created (backs up the /tmp/usage.db and then removes it), I also reset the WAN Traffic monitor data (set it back to 0).

Then I compare the wrtbwmon total usage for users (down and up) against the WAN usage (down and up).

I expect about 3 to 5 MB diff for the down and about 1-2 for the Up, as its a fairly complicated script and reset can mean that some data is lost.

But I expect this differece to not keep on rising.

Ill post a copy of my script that I am using.

Just a quick question in regards to this.

Say the router has been up and running for 5 mins (but wrtbwmon is not running), and then I manually start the wrtmon to start monitoring, does that mean I will lose the data for those first 5 minutes, and wrtbwmon will basically just start collecting data from when I started it or does the iptables record the data so wrtbwmon does not lose any information?

Cheers

A
FezJay
DD-WRT User


Joined: 11 Sep 2010
Posts: 178

PostPosted: Wed Oct 06, 2010 9:21    Post subject: Reply with quote
Hey

Just wondering what the two temp files are for in regards to the system.

traffic_55.tmp and traffic_66.tmp

The traffic_55 file gets updates via the wrtbwmon read command, and then the traffic_66 file gets updates when the wrtbwmon update command is executed.

Its also in the update command where they both seem to be used.

I can only guess that the traffic_55 one is for downloading and uploading speeds, while the traffic_66 is for the actual downloading and uploading usage?

Is that correct?

Cheers

A
RobotBanana
DD-WRT Novice


Joined: 15 Jul 2010
Posts: 22

PostPosted: Thu Oct 07, 2010 1:18    Post subject: Reply with quote
FezJay wrote:
I can only guess that the traffic_55 one is for downloading and uploading speeds, while the traffic_66 is for the actual downloading and uploading usage?

Is that correct?


Not too sure. Hopefully Wolf Kodi can chime in.

One of the traffic files is present in Emmanuel Brucy's original script (which doesn't do speed measurements), so you might be on to something.
fatusik
DD-WRT Novice


Joined: 08 Sep 2010
Posts: 7

PostPosted: Sat Oct 09, 2010 12:40    Post subject: Reply with quote
I'm using N16 with dd-wrt.v24-15396_NEWD-2_K2.6_mini_usb

Slightly modified the interface. Looks like Status/LAN window.
There is only one tab in mypage, so don't need to make submenus from topmenu, only one page in Status.



Put my wrtbwmon and usage.sh on ftp and use script:
Code:

sleep 10
wget ftp://blablabla.com/wrtbwmon -O /tmp/wrtbwmon && chmod +x /tmp/wrtbwmon
wget ftp://blablabla.com/usage.sh -O /tmp/usage.sh && chmod +x /tmp/usage.sh
wget ftp://blablabla.com/usage.db -O /tmp/usage.db

while :
do
/tmp/wrtbwmon setup
/tmp/wrtbwmon read
sleep 9
/tmp/wrtbwmon update /tmp/usage.db
/tmp/wrtbwmon publish /tmp/usage.db /tmp/usage.html
done &

while :
do
sleep 900
cd /tmp/ && ftpput -u user -p pass blablabla.com/usage.db usage.db
done


Works great.

P.S. And don't forget to change nvram.
Code:

nvram set mypage_scripts="/tmp/usage.sh"
nvram commit
FezJay
DD-WRT User


Joined: 11 Sep 2010
Posts: 178

PostPosted: Wed Oct 13, 2010 0:29    Post subject: Reply with quote
Hey

I have been looking pretty closely at the wrtbwmon and I have come across a very interesting issue.

Its to do with the setup command

Basically this step checks to see if a iptables rule has been set up and that its always at rule position 1. If it exists, but its not number 1, then delete it and reassign it.

This will mean that it will get executed first when new traffic comes in (its the first rule)

Once the rule is verified, we then go through all the currently connected IP's (/proc/etc/arp file) and assign the IPs to the rule (Only br0 IPs are added as the grep 'br0' command filters the vlan one out).

Issues: The original version for deleting the rule is iptables -D RRPDT (something like that).

Unfortunately, this does not work on some versions of DD-WRT (or if it worked at all).

If it doesnt work, you get multiple entries for RRPDT in the 'iptables -L FORWARD list'. This is bad, as what this causes it double up or quadruple up (so on) to occur on usage data.

The fix suggested in this topic is to find the line number iptables -L FORWARD --line-numbers | grep RRPDT | awk 'print{1}' and then delete the rule by line number.

This is good in theory, except when the rules move up or down before the delete takes place (everything takes time, no matter how small), which has happened to me...10 times. I now have 10 entries (after 5 hours) for RRPDT in my FORWARD chain rules. I downloaded 10MBs and it was reported as 120MB or something lol.

I hate to think what actual rule I was deleting lol.

Fortunately its easy to detect the cause of the issue. If you usage is really unexpectedly high, its getting counted more than once and you MUST check the iptables for multiple rules.

The FIX:

It appears that the -D must be exactly as the -I

iptables -I FORWARD -j "$CHAIN_RULE" #Create RULE

iptables -D FORWARD -j "$CHAIN_RULE" #Deletes RULE

Simple as that Smile
bpsmicro
DD-WRT User


Joined: 07 Oct 2010
Posts: 180

PostPosted: Wed Oct 13, 2010 15:48    Post subject: Reply with quote
I finally got wrtbwmon working on my Buffalo WZR-HP-G300NH. Like many, I suffered from the lack of a 'sort' command.
The fact I know next than diddly about Linux slowed me down a bit too. Lots of learning curve. :-)

Anyway, I'm left with a couple of questions:

1) There's a reference to a MAC-to-Name file which associates MAC addresses with machine names (or whatever name you want). The LAN Status page already has a table that includes machine names & MACs. Is there some sort of script (or perhaps somebody would like to write one) that can somehow hook into this "already known" info and generate the file wrtbwmon uses?

2) On the missing 'sort' thing (and again, remember that I know almost nothing about Linux): is there not some way for the script to check if sort is available on the router, and just use cat instead? Rather than making people hand-edit wrtbwmon to change one line?

3) In the table, is it correct to assume that "total down" & "total up" values are "since last router reboot"? Or is it basically "forever"? Somebody earlier mentioned some sort of "by month" upgrade, which would be super-cool. In my specific case though, if this were to become available someday, I'd be hoping for some sort of "definition" of when a month begins, to match my ISP account "month" (ie. my "month" starts at midnight of the 15th of each calendar month).

Thanks to all who are working on this. This particular "feature" is one of the primary reasons I made the leap into the world of DD-WRT in the first place (iptables-based firewall being another).

Brad.
FezJay
DD-WRT User


Joined: 11 Sep 2010
Posts: 178

PostPosted: Wed Oct 13, 2010 23:07    Post subject: Reply with quote
Hi bpsmicro

I can only answer a few of ur questions.

1. I am assuming you are reffering to the mac to static IP address under "SERVICES".

In this case, yes the current wrtbwmon will use that file, but only if you do not supply/pass your own on when calling the wrtbwmonn and you are using DNSMasq.

If the real names are not appearing on the output (so its still showing MAC etc), try restarting ur router.

If that doesnt work still you have to make sure you have set it up properly from the web ui.

Under "SETUP =>Network Address Server Settings (DHCP)" Please make sure you have selected the 'Use DNSMasq for DHCP' and 'Use DNSMasq for DNS'.

Then under services enable all the DNSMasq options and add the following to the additional DNSMasq options:

domain=lan
local=/lan/
expand-hosts

Remeber to enter 'lan' into the "lan domain", which is just above the section where you add mac-ips.

2. I guess it is possible.
sort -t file || cat file

3. It keeps going buddy.
But you can get it to do what you are asking pretty easily.

The easiest way is to set a cron job to to do the following

1 0 15 * * cp /tmp/usage/ /backup/history/usageDATE.db && rm -f rm /backup/usage.db && /tmp/usage.db

Basically, 1 minute past midnight on the 15 of every month, copy the current usage file to a history folder, and then remove the backed up one, and the one currently in tmp.

The only problem is. if one of the steps fails. Say the rm from /tmp/ the process fails.

You may not have been able to remove the file as its in use. Very unluckly to happen (and unlucky) but it can occur)

One way (easiest) to control the timing is to put the wole thing into cron jobs etc. Then run things at specific minutes of each hour. So you know at 59 minuts past the hour the publish will finish.

* * * * * wrtbwmon setup
1-59/1 * * * * wrtbwmon upate
1-59/1 * * * * wrtbwmon publish #This only does a read, so its pretty safe to do this
5-55/10 * * * * wrtbwmon backup
0 0 15 * * reset logic

The better way is to write it part of the script

e.g. wrtbwmon publish 15. The 15 represents ur reset day. It doesnt have to be publish either, upate will be fine.

So then you would go somthing like:

currentDay=`date +%d`
currentHour=`date +%H`

if [ $currentDay -eq $param_num ] && [ $currentHour -eq 0 ]; then

if [ -f /backup/history/usage`date +%d-%m-%y`.db]; th
en
cp -f /tmp/usage.db /backup/history/usage`date +%d-%m-%y`.db
rm /backup/usage.db
rm /tmp/usage.db
fi

fi


So what this will do is check to see if the current day is 15 ($param_num) and if we are into midnight 0.

If we are, then we check to make sure we have not already created a history backup (we have already done a reset), if that file does not exist, we can do out reset logic.

This was just written just now so its not tested and can contain syntax errors, but it gives you an idea.
bpsmicro
DD-WRT User


Joined: 07 Oct 2010
Posts: 180

PostPosted: Thu Oct 14, 2010 0:20    Post subject: Reply with quote
Thanks for that. The last bit slightly exceeds my comprehension of Linux commands, but I understand enough that I could probably play with it a while. I'm always game to learn new things. :-)

For the first question, what I was actually referring to was the info in the Status->LAN page (keep in mind I'm running the Buffalo-branded DD-WRT so some of my screens may be slightly different).
My page has an "Active Clients" table, which has machine names in the first column, assigned IP in the second, and MAC in the third.
There's also another table on that page called "DHCP Clients", which also has name, IP, MAC columns.

Having said that, I only have about a dozen "active" machines on my network, so it's not like I couldn't make a hand-rolled MAC->Name file easily enough.

Brad.
FezJay
DD-WRT User


Joined: 11 Sep 2010
Posts: 178

PostPosted: Thu Oct 14, 2010 1:59    Post subject: Reply with quote
Hey

Just a general question about iptables and rules.

wrtbwmon creates a rule

This rule is inserted into the forward chain.

Then the rule is assigned against any connected IP that is found in the /proc/net/arp file.

This should and does happen very often.

This has been proven to work and it seems to be doing it's job.

I have one concern:

When I go iptables -L FORWARD -vnx

I get the following:
22335 12617810 BWMON 0 -- * * 0.0.0.0/0 0.0.0.0/0

What I have bolded is the total bytes that have come through this rule.

Now, any person would think well, if we have this total on the rule, then the SUM of all clients below it (IPs that is has been assigned against) should add up or get very close to it, right?

I understand there is overhead and timing issues between when the packets get read and a new IP gets assigned the rule, but its only 5 to 10 seconds (depending on how you have it set up).

However, I am seeing a larger difference between the sum of the clients and the actual rule total. Its now at 23MBs.

The thing is, the client usage updating is working correctly. The right stuff is getting added to the right MAC etc and going up according to the bytes value.

So, what could be happening?

I only have ideas, but I bet someone with a bit of knowledge around iptables and how the rules have been setup will be able to inform us.

1. Could it be a ghost client? Very unlikely as even the MAC address would be shown on the usage output.

2. Its just overhead thats not getting assigned to the IP's. Like incoming/outgoing traffic that has been blocked by the firewall maybe

3. Becasue the rule is an 'all in' rule, it records traffic over every device that is using the internet. This could mean system calls etc which are not considered clients could be executing web requests etc.

Really I am not too sure.

All I know it the recorded total bytes by the wrtbwmon RULE is very close to the total bytes shown by the WEB UI's WAN monitor. So this proves that the wrtbwmon is recording information correctly.

If anyone is wanting to be clever about this. You could write into the publish script a few lines that pull out the total recorded bytes by the wrtbwmon RULE and display that as the total.

Then total up all the clients usage, and basically work out the amount of data that is not being assigned to any MAC/Client.
FezJay
DD-WRT User


Joined: 11 Sep 2010
Posts: 178

PostPosted: Fri Oct 22, 2010 12:31    Post subject: Reply with quote
Hey

I have created a monitor solution based roughly on the wrtbwmon solution.

If you are interested in helping me test it, please see this link

http://www.dd-wrt.com/phpBB2/viewtopic.php?p=499293#499293

A

_________________
dd-wrt eko/V24-K26 15508
Belkin F7D4301
8MB Flash
64MB RAM

Optware (jffs/mnt) 4GB Flash Drive
qualif
DD-WRT Novice


Joined: 06 Jul 2008
Posts: 46

PostPosted: Sat Nov 06, 2010 11:00    Post subject: Reply with quote
ok good script
qualif
DD-WRT Novice


Joined: 06 Jul 2008
Posts: 46

PostPosted: Sun Nov 07, 2010 10:42    Post subject: Reply with quote
mojso wrote:
Would be good to have a new page for each month. something like Traffic by Month, but only for a month not every day


+1
Sash
DD-WRT Guru


Joined: 20 Sep 2006
Posts: 17638
Location: Hesse/Germany

PostPosted: Wed Dec 01, 2010 11:07    Post subject: Reply with quote
FYI: all of these scripts doenst work on routerd networks since they use only arp info on setting up the iptables rules.
_________________
Forum Guidelines...How to get help
&
Forum Rules
&
RTFM/STFW
&
Throw some buzzwords into the WIKI search Exclamation
_________________
I'm NOT rude, just offer pure facts!
_________________
Atheros (TP-Link & Clones, etc ) debrick service in EU
_________________
Guide on HowTo be Safe, Secure and Protect Your Online Anonymity!
Goto page Previous  1, 2, 3, 4, 5, 6 ... 10, 11, 12  Next Display posts from previous:    Page 5 of 12
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Contributions Upload All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum