Posted: Sat May 18, 2024 6:12 Post subject: 4am daily freeze - how to get more info
Hi,
Every morning at almost precisely 4am every day my router starts processing very slowly (affecting all attached clients) it doesn't stop entirely but it just incredibly slow.
I enabled SNMP and external Syslog:
* In syslog, nothing except NTP update (which happens every hour)
* On SNMP, I see both CPU cores jump to between 10-15% usage and stay there, never dropping back to the normal idle of about 2.5%.
So i assume that some process has kicked off and is looping but I don't know what.
Any guidance on how I can try and get more log info/detail at 4am to try and work out what's going on? As a work around I've set a scheduled reboot for 0430 which solves it, but I want to solve the real problem.
Asus RT-AC68U C1
DD-WRT v3.0-r55460 std (03/25/24)
Linux 4.4.302-st49 #11249 SMP Thu Mar 21 07:12:03 +06 2024 armv7l
Device is in dump AP mode - with 5 SSIDs with VLAN trunking back to my OPNSense Firewall.
Any guidance/assistance/direction would be appreciated.
Thanks
Cameron.
Joined: 16 Nov 2015 Posts: 6899 Location: UK, London, just across the river..
Posted: Sat May 18, 2024 7:53 Post subject:
the standard is ...update to a newer build (current 56409), reset and manually rebuild your settings..
are you going out of nvram or your PSU (Power adapter) plays bad...?? _________________ Atheros
TP-Link WR740Nv1 ---DD-WRT 58184 WAP
TP-Link WR1043NDv2 -DD-WRT 61848 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 61915 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x2VLAN,Vanilla
Netgear R7800 --DD-WRT 61915 Gateway/DNSCryptv2,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 61848 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Dynalink DL-WRX36-DDWRT 61745
Broadcom
Netgear R7000 --DD-WRT 61745 Gateway/DNScrypt-proxy2/AD-Block,IPset Firewall,Forced DNS,x4VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
Joined: 26 Mar 2013 Posts: 1884 Location: Hung Hom, Hong Kong
Posted: Sat May 18, 2024 10:17 Post subject: Re: 4am daily freeze - how to get more info
camsaway wrote:
Every morning at almost precisely 4am every day my router starts processing very slowly (affecting all attached clients) it doesn't stop entirely but it just incredibly slow.
....
Asus RT-AC68U C1
DD-WRT v3.0-r55460 std (03/25/24)
Linux 4.4.302-st49 #11249 SMP Thu Mar 21 07:12:03 +06 2024 armv7l
Device is in dump AP mode - with 5 SSIDs with VLAN trunking back to my OPNSense Firewall.
Is it just the wifi? Is the LAN working fine?
If it's just about the wifi, are you living in a house or a tall building? For the later, maybe it's channel conflict. You can try changing wireless channel.
Another possibility is re-key interval. Some old gadgets might not work well with it. You can try setting a longer re-key interval in wireless settings. Or you can just always forget the connection to DD-WRT and reconnect at that "4 a.m.". _________________ Router: Asus RT-N18U (rev. A1)
Drink, Blink, Stretch! Live long and prosper! May the Force and farces be with you!
Ok, well I've flashed to the latest firmware - let's start there.
I guess next try will be to reset to factory default and redo the config.
Then final option - maybe skip the scheduled restart and ssh into the device and run top while it's constrained.
In answer to the question, i'm not using LAN ports other than to trunk back from the AP to the FW.
(My other networks, eg. wired LAN connection to the FW are all working fine. It's only this DDWRT device that is having a problem).
Joined: 26 Mar 2013 Posts: 1884 Location: Hung Hom, Hong Kong
Posted: Mon May 20, 2024 10:34 Post subject:
camsaway wrote:
In answer to the question, i'm not using LAN ports other than to trunk back from the AP to the FW.
(My other networks, eg. wired LAN connection to the FW are all working fine. It's only this DDWRT device that is having a problem).
When something is so predictable and reproducible, it might have nothing to do with DD-WRT. Anyway, no harm trying options. But BEWARE that you might be misleading or fooling YOURSELF! _________________ Router: Asus RT-N18U (rev. A1)
Drink, Blink, Stretch! Live long and prosper! May the Force and farces be with you!
Ok - so the plot thickens. I've redone the config, and tried an alternative device and I still get the issue.
But here's something weird. When I run TOP I don't see a process using the CPU, and the GUI doesn't show the CPU load - so maybe it's not actually CPU load.
Can anyone help me/direct me how to determine what these OIDs represent?
.1.3.6.1.2.1.25.3.3.1.2.196608
.1.3.6.1.2.1.25.3.3.1.2.196609
It should be the two cores of the processesor from what I can find, but apparently it's not
.iso.org.dod.internet.mgmt.mib-2.host.hrDevice.hrProcessorTable.hrProcessorEntry
Are there other types of processor inside the router it could be representing?
Alternatively - back to my original question - is there anyway to generate more verbose logs that get sent to syslog server so I can try and determine what's happening at exactly 4am each day.
Joined: 16 Nov 2015 Posts: 6899 Location: UK, London, just across the river..
Posted: Sun May 26, 2024 19:13 Post subject:
serial log is the best...and tells everything..
you can do external log (kernel log activated) have a look at services>syslog section..
either sent to a USB or external PC with syslog running...but in both cases it wont be that efficient as serial log..but still better than nothing no idea if it will show the problem...
freezes are usually, bad settings or hardware failure bound... _________________ Atheros
TP-Link WR740Nv1 ---DD-WRT 58184 WAP
TP-Link WR1043NDv2 -DD-WRT 61848 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 61915 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x2VLAN,Vanilla
Netgear R7800 --DD-WRT 61915 Gateway/DNSCryptv2,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 61848 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Dynalink DL-WRX36-DDWRT 61745
Broadcom
Netgear R7000 --DD-WRT 61745 Gateway/DNScrypt-proxy2/AD-Block,IPset Firewall,Forced DNS,x4VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
Or it's something related to ISP or the OPNSense box if it is routinely at 4am... _________________ "The woods are lovely, dark and deep,
But I have promises to keep,
And miles to go before I sleep,
And miles to go before I sleep." - Robert Frost
"I am one of the noticeable ones - notice me" - Dale Frances McKenzie Bozzio
So I've partially figured it out. It was happening at exactly the same time that another WAP on the network was rebooting. Stopped the other nightly reboot and the problem has gone away (1 day so far, but looks promising)
But I am curious to why this is happening.
My setup:
* OPNSense N105 Mini-PC is the Hub,
* igc0 = WAN connection to ISP/Fibre
* igc1 = ZTE MFC286C running OpenWRT
* igc2 = This device running DDWRT (ASUS RT-AC66U B1) - And soon to be a second one as well.
* igc3 = Switched LAN
* I have 4 bridged VLANs configured on OPNSense and the WAPs on both 2.4G and 5G (100 Wired, 200 Wifi, 300 IOT, 400 Guest)
There is no multipathing, but is it possible that a MAC storm is happening due to wireless devices jumping between access points when the one reboots? Do I need to enable STP/RSTP on the VLANs?
...update to a newer build (current 56409), reset and manually rebuild your settings.. 
no idea if it will show the problem...