YAMon 3.4 - usage monitoring for your router

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> General Questions
Goto page Previous  1, 2, 3 ... 33, 34, 35, 36  Next
Author Message
MomenMamdouh
DD-WRT User


Joined: 19 Sep 2017
Posts: 60
Location: Egypt

PostPosted: Fri Aug 09, 2019 18:20    Post subject: Reply with quote
al_c wrote:
scope2 wrote:
I just updated to latest version ( 3.4.8 ) and something is not right:

From Live tab:

1 Current Device
Time
Group Device Downloads Uploads
Total KB/s Total KB/s
18:27:24 # devices: 1 # connections: 1100 0.0 0.0
Unknown No Matching MAC 0.0 0.0
18:26:54 # devices: 1 # connections: 1122 0.0 0.0
Unknown No Matching MAC 0.0 0.0
18:25:28 # devices: 1 # connections: 1372 0.0 0.0
Unknown No Matching MAC 0.0 0.0
18:24:59 # devices: 1 # connections: 1178 0.0 0.0
Unknown No Matching MAC 0.0 0.0
18:24:28 # devices: 1 # connections: 920 0.0 0.0
Unknown No Matching MAC 0.0 0.0

only 1 device? I usually have a handful access the internet at any given time.


Ack!!! I'm *definitely* not seeing this at my end.

Can someone please send log (and perhaps live data) files?

I *am* seeing some bad behaviour when devices jump between my main and guest networks... manifesting as extraneous incomplete entries in users.js. I am working on a fix to that.

Al
TIA.


i sent to you another log file, please check your PM

_________________
Routers:
TP-Link Archer C7 v2-(EU)
TP-Link WR840n v3
TP-Link WR740n v5
Sponsor
bullshooter
DD-WRT Novice


Joined: 17 Apr 2018
Posts: 2

PostPosted: Wed Aug 14, 2019 13:36    Post subject: Not all devices logging Reply with quote
If this is already been discussed, I missed it. I just re-installed YAMon (3.4.Cool after not running it for a while. I've noticed that while most of my devices showed up on the Summary page, data is only actually being logged for a few of my devices.

The log file shows a LOT of lines similar to the following for those devices that aren't logging:

22:01:29 2 !!! No matching entry in _currentUsers for SomeDevice (SomeDevice)?!? - adding 0 to unknown mac (_totalLostBytes=0 / 0)

Any thoughts?
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2029
Location: Ottawa Canada

PostPosted: Fri Aug 16, 2019 11:54    Post subject: Re: Not all devices logging Reply with quote
bullshooter wrote:
If this is already been discussed, I missed it. I just re-installed YAMon (3.4.Cool after not running it for a while. I've noticed that while most of my devices showed up on the Summary page, data is only actually being logged for a few of my devices.

The log file shows a LOT of lines similar to the following for those devices that aren't logging:

22:01:29 2 !!! No matching entry in _currentUsers for SomeDevice (SomeDevice)?!? - adding 0 to unknown mac (_totalLostBytes=0 / 0)

Any thoughts?
sorry for the grief... Please revert to 3.4.7

Al
ddaniel51
DD-WRT Guru


Joined: 19 Feb 2013
Posts: 1323

PostPosted: Sat Aug 17, 2019 18:47    Post subject: Reply with quote
Is it just my imagination or do I hear Al muttering and grumbling in the background?

Meanwhile 3.4.7 is just cruising along.

_________________
Segment 1 R9000 10Gb LAN, 1Gb ISP BS
Test Bed R9000 10Gb LAN
Wired AP 1 R9000 10Gb LAN
Wired AP 2 R9000 1Gb LAN
Wired AP 3 R9000 1Gb LAN
Test Bed R7800 1Gb LAN OpenWRT Kong
www.desipro.de/openwrt/ (Kong's)
YAMon usage-
www.monitoring.com/manualInstall.php
scope2
DD-WRT User


Joined: 12 Jul 2017
Posts: 87

PostPosted: Sat Aug 24, 2019 21:11    Post subject: Reply with quote
No biggie, but for some reason my Monthly Usage Breakdown tab is showing the icon that I have the router has rebooted every day for the last 2 weeks, but router is showing this uptime:

Quote:
22:07:39 up 80 days, 2:47, load average: 0.77, 0.83, 0.78


Could it be YAMon restarting every day? I havent got a crontab job for this - so not sure how it is restarting every day?

Also, it reports this error:
Quote:
Error #2 reading the hourly data file: `data3/2019/06/2019-07-06-hourly_data.js` Not Found


I have the "2019-07-06-hourly_data.js" file, but it is in the 07 folder, not 06 as reported above?
ddaniel51
DD-WRT Guru


Joined: 19 Feb 2013
Posts: 1323

PostPosted: Sun Aug 25, 2019 21:05    Post subject: Reply with quote
At midnight Yamon closes the current log and hourly data files and creates new ones for the next 24 hr period. It does not reboot.

On the ISP Billing date it creates a new monthly data file.

Changing the ISP Billing date can also shift where Yamon creates and looks for daily/hourly files.

_________________
Segment 1 R9000 10Gb LAN, 1Gb ISP BS
Test Bed R9000 10Gb LAN
Wired AP 1 R9000 10Gb LAN
Wired AP 2 R9000 1Gb LAN
Wired AP 3 R9000 1Gb LAN
Test Bed R7800 1Gb LAN OpenWRT Kong
www.desipro.de/openwrt/ (Kong's)
YAMon usage-
www.monitoring.com/manualInstall.php
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2029
Location: Ottawa Canada

PostPosted: Tue Aug 27, 2019 13:28    Post subject: Reply with quote
ddaniel51 wrote:
Is it just my imagination or do I hear Al muttering and grumbling in the background?

Meanwhile 3.4.7 is just cruising along.

Evil or Very Mad
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2029
Location: Ottawa Canada

PostPosted: Tue Aug 27, 2019 13:29    Post subject: Reply with quote
Well, version 3.4.8 truly was a bust... Things worked really well on my system at home (honest!) but not so well in the wild (the joys of firmware discrepancies!!!)

If you've updated to 3.4.8, you may get most of your traffic going to 'unknown mac'. Please revert to version 3.4.7 which you can do by re-running install.sh.

I'm really sorry for the grief.

On a related front... I started working on YAMon at least six years ago and the code and features have developed in a really haphazard way. I'm going to take the next couple of weeks to go back to the drawing board to re-think things.

Now's the time to ask if you want particular features added (or fixed)...

Al
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2029
Location: Ottawa Canada

PostPosted: Tue Aug 27, 2019 13:48    Post subject: Reply with quote
scope2 wrote:
No biggie, but for some reason my Monthly Usage Breakdown tab is showing the icon that I have the router has rebooted every day for the last 2 weeks, but router is showing this uptime:

Quote:
22:07:39 up 80 days, 2:47, load average: 0.77, 0.83, 0.78


Could it be YAMon restarting every day? I havent got a crontab job for this - so not sure how it is restarting every day?

Also, it reports this error:
Quote:
Error #2 reading the hourly data file: `data3/2019/06/2019-07-06-hourly_data.js` Not Found


I have the "2019-07-06-hourly_data.js" file, but it is in the 07 folder, not 06 as reported above?

Can you send screenshots and your monthly data file to me at question@usage-monitoring.com?

RE: data file locations... things depend upon two parameters in your config.file
1. _organizeData, and
2. _ispBillingDay

The default value for _organizeData is '2'... meaning that data files are grouped into folders by year and billing interval.
BTW, if _organizeData=1, files grouped just by year (is anyone using this?); and if _organizeData=0, all of the files go into the same folder (which IMHO gets messy)

I stress that when _organizeData='2', files are grouped by year and billing interval.
This is where the value of _ispBillingDay comes into play... if _ispBillingDay=1 then the billing interval is a month. For any other value, the billing interval folder will contain data files from two months - e.g., for my ISP, _ispBillingDay=5 meaning that in the /data/2019/08 folder, you will find the daily data files from Aug 5-->Sep 4. if _ispBillingDay=23 the interval folders would contain files from the Aug 23-->Sep 22.

Now that I'm thinking about this, I'm not sure what happens in Feb if _ispBillingDay=29 or higher?!? Does anyone have this in their setup?

Is this clear as mud now?

Al
DeadDjembe
DD-WRT Novice


Joined: 04 Apr 2017
Posts: 3

PostPosted: Thu Aug 29, 2019 13:42    Post subject: Reply with quote
Hi all,

I've been having an issue with YAMon detecting many different mac addresses for my server. My server is running UNRAID 6.7. The mac address appears to randomly change many times throughout the day. DD-WRT assigns my server a static IP based on the server's mac address - the server IP address is constant. Any thoughts on what would cause this?


al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2029
Location: Ottawa Canada

PostPosted: Thu Aug 29, 2019 16:59    Post subject: Reply with quote
DeadDjembe wrote:
Hi all,

I've been having an issue with YAMon detecting many different mac addresses for my server. My server is running UNRAID 6.7. The mac address appears to randomly change many times throughout the day. DD-WRT assigns my server a static IP based on the server's mac address - the server IP address is constant. Any thoughts on what would cause this?




YAMon presumes that the MAC address for a device is unique and static (which is the case on most networks).

I know that it is possible in Windows to randomize hardware addresses (and I presume with other OSs) - e.g., see your WiFi settings dialog. It sounds to me like you have enabled that option and not surprisingly, it has broken YAMon.

It might be possible to alter the way that YAMon works (i.e., map the IP back to a static MAC). But I have not played with that setting so I don't know how much work it would require. I expect that a lot of refactoring will be needed. It is fine to say that the IP uniquely identifies the server but is supposed to happen if/when that IP changes?!? Can you set a static IP if the hardware address is randomized? (I don't think that is how the static IP dialog works in DD-WRT).

Sorry for not having a better answer ATM.

Al
scope2
DD-WRT User


Joined: 12 Jul 2017
Posts: 87

PostPosted: Fri Aug 30, 2019 10:31    Post subject: Reply with quote
al_c wrote:
scope2 wrote:
No biggie, but for some reason my Monthly Usage Breakdown tab is showing the icon that I have the router has rebooted every day for the last 2 weeks, but router is showing this uptime:

Quote:
22:07:39 up 80 days, 2:47, load average: 0.77, 0.83, 0.78


Could it be YAMon restarting every day? I havent got a crontab job for this - so not sure how it is restarting every day?

Also, it reports this error:
Quote:
Error #2 reading the hourly data file: `data3/2019/06/2019-07-06-hourly_data.js` Not Found


I have the "2019-07-06-hourly_data.js" file, but it is in the 07 folder, not 06 as reported above?

Can you send screenshots and your monthly data file to me at question@usage-monitoring.com?

RE: data file locations... things depend upon two parameters in your config.file
1. _organizeData, and
2. _ispBillingDay

The default value for _organizeData is '2'... meaning that data files are grouped into folders by year and billing interval.
BTW, if _organizeData=1, files grouped just by year (is anyone using this?); and if _organizeData=0, all of the files go into the same folder (which IMHO gets messy)

I stress that when _organizeData='2', files are grouped by year and billing interval.
This is where the value of _ispBillingDay comes into play... if _ispBillingDay=1 then the billing interval is a month. For any other value, the billing interval folder will contain data files from two months - e.g., for my ISP, _ispBillingDay=5 meaning that in the /data/2019/08 folder, you will find the daily data files from Aug 5-->Sep 4. if _ispBillingDay=23 the interval folders would contain files from the Aug 23-->Sep 22.

Now that I'm thinking about this, I'm not sure what happens in Feb if _ispBillingDay=29 or higher?!? Does anyone have this in their setup?

Is this clear as mud now?

Al


The error seems to have gone away now.. I have the following config settings:

_ispBillingDay='1'
_organizeData='2'

I do seem to remember that the _ispBillingDay was set to 5 (for some reason - I wouldnt have knowingly done that) when I upgraded to 3.4.8 (before downgrading to 3.4.7) - so it might have been as a result of that.?

Anyway, it seems to be behaving now.. I will monitor. Smile
DeadDjembe
DD-WRT Novice


Joined: 04 Apr 2017
Posts: 3

PostPosted: Fri Aug 30, 2019 18:38    Post subject: Reply with quote
al_c wrote:
DeadDjembe wrote:
Hi all,

I've been having an issue with YAMon detecting many different mac addresses for my server. My server is running UNRAID 6.7. The mac address appears to randomly change many times throughout the day. DD-WRT assigns my server a static IP based on the server's mac address - the server IP address is constant. Any thoughts on what would cause this?




YAMon presumes that the MAC address for a device is unique and static (which is the case on most networks).

I know that it is possible in Windows to randomize hardware addresses (and I presume with other OSs) - e.g., see your WiFi settings dialog. It sounds to me like you have enabled that option and not surprisingly, it has broken YAMon.

It might be possible to alter the way that YAMon works (i.e., map the IP back to a static MAC). But I have not played with that setting so I don't know how much work it would require. I expect that a lot of refactoring will be needed. It is fine to say that the IP uniquely identifies the server but is supposed to happen if/when that IP changes?!? Can you set a static IP if the hardware address is randomized? (I don't think that is how the static IP dialog works in DD-WRT).

Sorry for not having a better answer ATM.

Al


Thanks for the reply. I've checked a few settings and dug deeper into the issue. What appears to be happening, I am running a number of Docker Containers on the server. When the network type on the container is set to "Host", the container is supposed to share the IP with the Host and is allowed to utilize any ports not used by the Host. There is no isolation between the host machine and the container. For some reason, this causes the container to generate it's own MAC address, but maintain the IP address of the server/server, thus it appears that multiple MAC addresses have the same IP address in YAMon. This does not happen if the network type is set to "Bridge".
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2029
Location: Ottawa Canada

PostPosted: Thu Sep 05, 2019 0:19    Post subject: Reply with quote
DeadDjembe wrote:
al_c wrote:
DeadDjembe wrote:
Hi all,

I've been having an issue with YAMon detecting many different mac addresses for my server. My server is running UNRAID 6.7. The mac address appears to randomly change many times throughout the day. DD-WRT assigns my server a static IP based on the server's mac address - the server IP address is constant. Any thoughts on what would cause this?




YAMon presumes that the MAC address for a device is unique and static (which is the case on most networks).

I know that it is possible in Windows to randomize hardware addresses (and I presume with other OSs) - e.g., see your WiFi settings dialog. It sounds to me like you have enabled that option and not surprisingly, it has broken YAMon.

It might be possible to alter the way that YAMon works (i.e., map the IP back to a static MAC). But I have not played with that setting so I don't know how much work it would require. I expect that a lot of refactoring will be needed. It is fine to say that the IP uniquely identifies the server but is supposed to happen if/when that IP changes?!? Can you set a static IP if the hardware address is randomized? (I don't think that is how the static IP dialog works in DD-WRT).

Sorry for not having a better answer ATM.

Al


Thanks for the reply. I've checked a few settings and dug deeper into the issue. What appears to be happening, I am running a number of Docker Containers on the server. When the network type on the container is set to "Host", the container is supposed to share the IP with the Host and is allowed to utilize any ports not used by the Host. There is no isolation between the host machine and the container. For some reason, this causes the container to generate it's own MAC address, but maintain the IP address of the server/server, thus it appears that multiple MAC addresses have the same IP address in YAMon. This does not happen if the network type is set to "Bridge".
So you have changed the setting to 'bridge'? Are things running as expected now?

Al
al_c
DD-WRT Guru


Joined: 13 Apr 2013
Posts: 2029
Location: Ottawa Canada

PostPosted: Wed Sep 11, 2019 16:56    Post subject: A sneak peak into YAMon v4... Reply with quote
As I alluded to in a previous update, after 6+ years of making tweaks, modifications & fixes to the original YAMon code, I'm throwing it all out and starting over from scratch.

I've learned a lot about bash script over the past several years (and continue to do so every day) so it easier rebuild the code than go back and fix the confusing mess it had become.

Here's an update of recent progress

The biggest change is that YAMon 4 is going to be cron based. The nice thing about cron is that I don't have to worry about timing loops and detecting new hours, new days, etc. On the downside, you cannot schedule things any faster than once per minute.

Another nice thing is that the cron job entries will automatically restart if your server reboots. There could, of course, be some firmware variant specific foibles but we'll burn those bridges when we get to them.

I've already started building and testing a variety of separate tasks:
Code:
# m h    dom mon dow  command
#YAMon jobs: (updated 2019-09-11)
0 0 5 * * /opt/YAMon4/new-billing-interval.sh
0 0 * * * /opt/YAMon4/new-day.sh
15 2 * * * /opt/YAMon4/in-unlimited.sh start
0 8 * * * /opt/YAMon4/in-unlimited.sh end
0 * * * * /opt/YAMon4/new-hour.sh
* * * * * /opt/YAMon4/update-live-data.sh
* * * * * /opt/YAMon4/check-network.sh
*/5 * * * * /opt/YAMon4/update-reports.sh
59 * * * * /opt/YAMon4/end-of-hour.sh
59 23 * * * /opt/YAMon4/end-of-day.sh


For those not familiar with cron jobs, the fields in the lines above represent from left to right:
1 - minutes
2 - hour
3 - day of month
4 - month
5 - day of week
6 - code to execute

So, the `new-billing-interval` script is launched at midnight (00:00) on the 5 day of every month
The `in-unlimited.sh start` script will run at 2:15AM every night (every day and every day of the week)
The `update-reports` script runs every 5 minutes (as denoted by the */5 and based upon the parameter in config.file).
etc.

For those of you asking about `blocking` access to certain users at certain times, fields 1, 2, & 5 are particularly intriguing... Once again, the biggest issue will be defining a UI/communication strategy to allow implementation of the blocking rules.

YAMon 4 is running in parallel with YAMon v3 and ahs completely separate iptables entries and data storage.

The format of the new data files *will* change but that will have no bearing on the history you've already accumulated. Early on, I decided to build the data files around JavaScript functions rather than data structures. For v4, all I have to do is define new JS functions which maintain the same data structures and the reports will continue to work with the old and new files (at least in theory... ha ha ha).

Much of the data is now being accumulated in /tmp and infrequently written to /opt... (thereby reducing wear and tear on your USB drive). I also plan to implement a lot more of the functionality (esp. around users.js) on the reports side of things rather than the router.

So far I'm generating the live data file and have gotten a good start on the hourly usage (but IPv4 only ATM)

Next steps
- compare hourly results with those that I'm getting from v3
- roll the hourly results into the billing interval file
- re-implement adding new devices
- replicate the IPv4 functionality for IPv6
- rebuild the setup script
- increase communication between the reports and the scripts (via the database integration option in the reports)
- build the blocking interface and functionality
- test, test, test and then some more testing

And then more testing

Let me know if you have any thoughts...

Al
Goto page Previous  1, 2, 3 ... 33, 34, 35, 36  Next Display posts from previous:    Page 34 of 36
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> General Questions All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum