YAMon 3.4 - usage monitoring for your router

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> General Questions
Goto page Previous  1, 2, 3 ... 26, 27, 28, 29, 30, 31  Next
Author Message
VTecheira
DD-WRT User


Joined: 17 Sep 2014
Posts: 54

PostPosted: Mon Feb 25, 2019 10:38    Post subject: Reply with quote
I think it should be in your forum inbox unless you deleted the messages but I'll post below for posterity.

Note for those looking to leverage this, it was based on the 3.4.5 code base. This is an extract of messages I sent to Al a while back, with some additional notes and commentary. I also included my initial script to validate the functionality, since it may help folks understand exactly what's happening.



Quote:

Ok.. the code is a bit crude but it seems to be workable in my very limited testing.

Couple of things as I mentioned before, I think you can improve wireless bridge by checking static leases against bridged IPs. You're code could probably handle it already if you just auto-add all static leases to users.js.

So for my code to work it assumes the users has already set up appropriately the SSH with key to the bridge router.

The basic pseudo logic is this.

1. Get list of bridge macs
2. SSH to the gateway and ping each of them (this is necessary to make sure they haven't been flushed from the cache when we do the look up)
3. get and parse the arp results
4. Try to put in with the rest of the YAMon code
5. Anything left over send through the normal bridge logic.

I add the following to the config file
Code:

_keyFile="/tmp/root/.ssh/ssh_host_rsa_key"   # SSH Key file locations  -- VT
_bridgeUser="root"            # Remote SSH User  -- VT
_bridgeGate="192.168.X.XX"         # Wireless Bridge ip address  -- VT


Attached is code I added. It's a bit crude but it seems to work in limited testing. I'm sure you can clean it up a lot easier since you understand the YAMon code better than I do.

Couple of caveats.
* It assumes whoever's doing this will have already set up the SSH key to the remote machine.
* When the router restarts the knownhosts file gets reset. Currently I use a script to copy the knownhosts file back otherwise you have to manually reaccept host key every time. DD-WRT has an option '-y' to autoaccept, but it's not universal and seems like a big security risk.


I've attached my edited YAMon file here. The changes I made were to the checkBridgeMac function under checkIPs. My code snippets start at line 925 and end at line 1037.


Another caveat not in the original, but I just thought of since I saw a previous poster had four wireless bridges, this code assumes only one bridge, so it would need to be modified to support multiple wireless bridges, but that should be relatively easy.

Note I've been running this code for about 3 months in my local environment and it seems to be working well.



remotearp.txt
 Description:
my original code snippet for testing the functionality.

Download
 Filename:  remotearp.txt
 Filesize:  2.33 KB
 Downloaded:  173 Time(s)


yamon3.4.5.vt.txt
 Description:
The modified YAMon file

Download
 Filename:  yamon3.4.5.vt.txt
 Filesize:  59.71 KB
 Downloaded:  160 Time(s)

Sponsor
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 1981
Location: Ottawa Canada

PostPosted: Tue Feb 26, 2019 21:35    Post subject: Reply with quote
VTecheira wrote:
I think it should be in your forum inbox unless you deleted the messages but I'll post below for posterity.

Ahh... found it. I was looking in my inbox not the forum inbox... oops!

VTecheira wrote:
<snip>
Couple of things as I mentioned before, I think you can improve wireless bridge by checking static leases against bridged IPs. You're code could probably handle it already if you just auto-add all static leases to users.js.
<snip>

I now auto-add the static leases but if I recall correctly the code only checks on start-up ATM so changes new static leases or changes to existing entries don't get updated automatically. I figured the overhead of constantly checking was unnecessary given the small numbers of static leases and/or the infrequency of changes.

I'll look at getting this added to v3.4.8

Al
hyelton
DD-WRT Novice


Joined: 12 Apr 2007
Posts: 45

PostPosted: Wed Mar 06, 2019 23:12    Post subject: Reply with quote
Bit of a different question for this.

Is it possible to use this without NAT/Firewall/DHCP? Pretty much just having it in the middle between the modem and router? I'm looking for something that I can monitor daily/monthly/live traffic usage but continuing to keep current router without having a double NAT
VTecheira
DD-WRT User


Joined: 17 Sep 2014
Posts: 54

PostPosted: Fri Mar 08, 2019 12:01    Post subject: Reply with quote
hyelton wrote:
Is it possible to use this without NAT/Firewall/DHCP? Pretty much just having it in the middle between the modem and router?


I'll let Al weigh in but I'm fairly certain the answer to this is no. YAMon monitors data by capturing the info in iptables. So unless your router some allows you to somehow "export and replay" the traffic patterns in a way that seems native to YAMon, I don't think it will be possible.

As a corollary, YAMon doesn't need DHCP and NAT. It monitors the router traffic with iptables chain and aggregates per identified MAC/IP. It then tries to pretty up the data by naming it based on what you have defined in DHCP but also uses several other techniques including arp to identify the devices generating WAN traffic

What router firmware do you have? YAMon also works with Lede, tomato and a few others.
veekay
DD-WRT Novice


Joined: 09 Mar 2009
Posts: 33

PostPosted: Sat Mar 09, 2019 17:19    Post subject: Reply with quote
I am trying to reset the monthly date, but nothing seems to stick.

I changed it in the config.file and the config3.4.js, but whenever I restart Yamon the date doesn't change (stuck on the 22nd, want the 1st).
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 1981
Location: Ottawa Canada

PostPosted: Mon Mar 11, 2019 12:52    Post subject: Reply with quote
VTecheira wrote:
hyelton wrote:
Is it possible to use this without NAT/Firewall/DHCP? Pretty much just having it in the middle between the modem and router?


I'll let Al weigh in but I'm fairly certain the answer to this is no. YAMon monitors data by capturing the info in iptables. So unless your router some allows you to somehow "export and replay" the traffic patterns in a way that seems native to YAMon, I don't think it will be possible.

As a corollary, YAMon doesn't need DHCP and NAT. It monitors the router traffic with iptables chain and aggregates per identified MAC/IP. It then tries to pretty up the data by naming it based on what you have defined in DHCP but also uses several other techniques including arp to identify the devices generating WAN traffic

What router firmware do you have? YAMon also works with Lede, tomato and a few others.


Vince - not sure how I can expand upon your answer. You've explained the situation as well (if not better) than I can.

Al
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 1981
Location: Ottawa Canada

PostPosted: Mon Mar 11, 2019 13:14    Post subject: Reply with quote
veekay wrote:
I am trying to reset the monthly date, but nothing seems to stick.

I changed it in the config.file and the config3.4.js, but whenever I restart Yamon the date doesn't change (stuck on the 22nd, want the 1st).


VeeKay - where are you seeing the problem? I presume in the reports... right?

After you restart,
* does the value of `_ispBillingDay` in your config.file revert to `22`? (I really hope not!)
* does `_ispBillingDay` pick up the new value in config3.4.js? (Again, I really hope not)
* have you tried to force a refresh in the reports with a <ctrl>+F5? (Without this, your reports could be working off of a cached version of the JS files)

What changes were you expecting to see but did not?

I've tried changing the value of `_ispBillingDay` in config3.4.js and I see the change in the reports right away...

Al
veekay
DD-WRT Novice


Joined: 09 Mar 2009
Posts: 33

PostPosted: Mon Mar 11, 2019 22:57    Post subject: Reply with quote
Edit - Date fixed, but new problem.

After the date change all of my stats are wrong. I deleted the monthly files and ran the h2m.sh to recreate them. It seemed like everything worked, but now when I look at my monthly stats the data is still wrong and each time I change months my totals double.

What is the easiest way to fix the stats from 2-19 until today? I have my hourlys and backups available.

Oh, it also seems to think today is the end of the month as it is creating an hourly file for 3-31
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 1981
Location: Ottawa Canada

PostPosted: Wed Mar 13, 2019 14:27    Post subject: Reply with quote
veekay wrote:
Edit - Date fixed, but new problem.

After the date change all of my stats are wrong. I deleted the monthly files and ran the h2m.sh to recreate them. It seemed like everything worked, but now when I look at my monthly stats the data is still wrong and each time I change months my totals double.

What is the easiest way to fix the stats from 2-19 until today? I have my hourlys and backups available.

Oh, it also seems to think today is the end of the month as it is creating an hourly file for 3-31


I have not actually fully gone through this process myself... but a number of manual steps come to mind. I have not automated these steps because this sort of change does not occur frequently.

On your router:
0. run shutdown.sh
1. change the value in config.file
2. backup your YAMon install directory
3. assuming that data is organized by year and month (_organizeData='2' in config.file), you will have to manually move daily data files into the proper billing interval directories * (see below)
Also ,remove (or rename) the existing monthly totals files
4. run h2m.sh to regenerate the monthly totals (you will have to do this for each of the existing billing intervals)
5. run restart.sh

In the reports:
0. open the Settings tab and click `Export Settings`
1. open the Summary tab and click the trash can icon in each row of the historic usage table (on the right)... this is the fastest way to remove those old entries. Those totals will be recalculated when you next view the report for that interval.

I think that covers it all...


* from #3 above, remember that YAMon organizes data files by billing interval rather than calendar month and this is dependent upon the value of `_ispBillingDay` in your config.file.
- if _ispBillingDay=1 then the billing interval and calendar month are the same thing
- for any other value, the interval directory will contain data files spanning two calendar months - e.g., in my case _ispBillingDay=5, so the `/2019/03/ folder has data files from Mar 5th to Apr 4th

Hope this helps!

Al
veekay
DD-WRT Novice


Joined: 09 Mar 2009
Posts: 33

PostPosted: Wed Mar 13, 2019 14:43    Post subject: Reply with quote
al_c wrote:


I have not actually fully gone through this process myself... but a number of manual steps come to mind. I have not automated these steps because this sort of change does not occur frequently.
Al


That is basically what I tried, minus the last step to delete the reports. The problem with data doubling still happens though. I don't think it is permanent, since when I reload it goes back to normal.

Here is what I mean, from the monthly breakdown:

Fri 01 Mar 2019 13.0 GB 125.3 GB 138.4 GB 138.4 GB

Go back to Feb and then to March again and the numbers become:

Fri 01 Mar 2019 26.0 GB 250.7 GB 276.7 GB 276.7 GB

And it doubles each time.

My summary is still correct, so it seems to only affect the monthly totals.

Since I have less than a month of data I think I may just restart from scratch.
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 1981
Location: Ottawa Canada

PostPosted: Wed Mar 13, 2019 15:03    Post subject: Reply with quote
veekay wrote:
al_c wrote:


I have not actually fully gone through this process myself... but a number of manual steps come to mind. I have not automated these steps because this sort of change does not occur frequently.
Al


That is basically what I tried, minus the last step to delete the reports. The problem with data doubling still happens though. I don't think it is permanent, since when I reload it goes back to normal.

Here is what I mean, from the monthly breakdown:

Fri 01 Mar 2019 13.0 GB 125.3 GB 138.4 GB 138.4 GB

Go back to Feb and then to March again and the numbers become:

Fri 01 Mar 2019 26.0 GB 250.7 GB 276.7 GB 276.7 GB

And it doubles each time.

My summary is still correct, so it seems to only affect the monthly totals.

Since I have less than a month of data I think I may just restart from scratch.


Is anyone else seeing this? I'm not?!?

I open the reports, go to the monthly breakdown tab then change to Feb and then change back... those are the steps right?

I tried in Firefox and Chrome... what browser are you using?
veekay
DD-WRT Novice


Joined: 09 Mar 2009
Posts: 33

PostPosted: Thu Mar 14, 2019 2:24    Post subject: Reply with quote
al_c wrote:


I tried in Firefox and Chrome... what browser are you using?


This was in both. I just reset and started over and it isn't doing it. Things just really screwed up with that date change and redoing the totals.
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 1981
Location: Ottawa Canada

PostPosted: Sun Mar 17, 2019 11:55    Post subject: Reply with quote
veekay wrote:
al_c wrote:


I tried in Firefox and Chrome... what browser are you using?


This was in both. I just reset and started over and it isn't doing it. Things just really screwed up with that date change and redoing the totals.


I guess the word of advice is to properly select that value during the set-up process....
veekay
DD-WRT Novice


Joined: 09 Mar 2009
Posts: 33

PostPosted: Sun Mar 17, 2019 17:13    Post subject: Reply with quote
al_c wrote:
veekay wrote:
al_c wrote:


I tried in Firefox and Chrome... what browser are you using?


This was in both. I just reset and started over and it isn't doing it. Things just really screwed up with that date change and redoing the totals.


I guess the word of advice is to properly select that value during the set-up process....


It was the proper value for my billing, but I didn't like how it made the calendars display.
scope2
DD-WRT User


Joined: 12 Jul 2017
Posts: 83

PostPosted: Wed Mar 20, 2019 20:04    Post subject: Reply with quote
Im looking to start using Yamon again after nearly a years absence.. I have incomplete data for the period when I last used it, how do I clear the logs/stats down so that I restart with a fresh clean sheet?
Goto page Previous  1, 2, 3 ... 26, 27, 28, 29, 30, 31  Next Display posts from previous:    Page 27 of 31
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> General Questions All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum