4am daily freeze - how to get more info

Post new topic   Reply to topic    DD-WRT Forum Index -> Broadcom SoC based Hardware
Author Message
camsaway
DD-WRT Novice


Joined: 17 Apr 2024
Posts: 10

PostPosted: Sat May 18, 2024 6:12    Post subject: 4am daily freeze - how to get more info Reply with quote
Hi,

Every morning at almost precisely 4am every day my router starts processing very slowly (affecting all attached clients) it doesn't stop entirely but it just incredibly slow.

I enabled SNMP and external Syslog:
* In syslog, nothing except NTP update (which happens every hour)
* On SNMP, I see both CPU cores jump to between 10-15% usage and stay there, never dropping back to the normal idle of about 2.5%.

So i assume that some process has kicked off and is looping but I don't know what.

Any guidance on how I can try and get more log info/detail at 4am to try and work out what's going on? As a work around I've set a scheduled reboot for 0430 which solves it, but I want to solve the real problem.

Asus RT-AC68U C1
DD-WRT v3.0-r55460 std (03/25/24)
Linux 4.4.302-st49 #11249 SMP Thu Mar 21 07:12:03 +06 2024 armv7l

Device is in dump AP mode - with 5 SSIDs with VLAN trunking back to my OPNSense Firewall.

Any guidance/assistance/direction would be appreciated.
Thanks
Cameron.
Sponsor
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 6519
Location: UK, London, just across the river..

PostPosted: Sat May 18, 2024 7:53    Post subject: Reply with quote
the standard is Cool ...update to a newer build (current 56409), reset and manually rebuild your settings.. Laughing
are you going out of nvram or your PSU (Power adapter) plays bad...??

_________________
Atheros
TP-Link WR740Nv1 ---DD-WRT 57200 WAP
TP-Link WR1043NDv2 -DD-WRT 56941 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 56941 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 57200 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 57200 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
mwchang
DD-WRT Guru


Joined: 26 Mar 2013
Posts: 1889
Location: Hung Hom, Hong Kong

PostPosted: Sat May 18, 2024 10:17    Post subject: Re: 4am daily freeze - how to get more info Reply with quote
camsaway wrote:
Every morning at almost precisely 4am every day my router starts processing very slowly (affecting all attached clients) it doesn't stop entirely but it just incredibly slow.
....
Asus RT-AC68U C1
DD-WRT v3.0-r55460 std (03/25/24)
Linux 4.4.302-st49 #11249 SMP Thu Mar 21 07:12:03 +06 2024 armv7l

Device is in dump AP mode - with 5 SSIDs with VLAN trunking back to my OPNSense Firewall.

Is it just the wifi? Is the LAN working fine?

If it's just about the wifi, are you living in a house or a tall building? For the later, maybe it's channel conflict. You can try changing wireless channel.

Another possibility is re-key interval. Some old gadgets might not work well with it. You can try setting a longer re-key interval in wireless settings. Or you can just always forget the connection to DD-WRT and reconnect at that "4 a.m.". Smile

_________________
Router: Asus RT-N18U (rev. A1)

Drink, Blink, Stretch! Live long and prosper! May the Force and farces be with you!

Facebook: https://www.facebook.com/changmanwai
Website: https://sites.google.com/site/changmw
SETI@Home profile: http://setiathome.berkeley.edu/view_profile.php?userid=211832
GitHub: https://github.com/changmw/changmw
camsaway
DD-WRT Novice


Joined: 17 Apr 2024
Posts: 10

PostPosted: Sat May 18, 2024 11:21    Post subject: Reply with quote
Ok, well I've flashed to the latest firmware - let's start there.
I guess next try will be to reset to factory default and redo the config.
Then final option - maybe skip the scheduled restart and ssh into the device and run top while it's constrained.

In answer to the question, i'm not using LAN ports other than to trunk back from the AP to the FW.
(My other networks, eg. wired LAN connection to the FW are all working fine. It's only this DDWRT device that is having a problem).
mwchang
DD-WRT Guru


Joined: 26 Mar 2013
Posts: 1889
Location: Hung Hom, Hong Kong

PostPosted: Mon May 20, 2024 10:34    Post subject: Reply with quote
camsaway wrote:
In answer to the question, i'm not using LAN ports other than to trunk back from the AP to the FW.
(My other networks, eg. wired LAN connection to the FW are all working fine. It's only this DDWRT device that is having a problem).

When something is so predictable and reproducible, it might have nothing to do with DD-WRT. Anyway, no harm trying options. But BEWARE that you might be misleading or fooling YOURSELF! Smile

_________________
Router: Asus RT-N18U (rev. A1)

Drink, Blink, Stretch! Live long and prosper! May the Force and farces be with you!

Facebook: https://www.facebook.com/changmanwai
Website: https://sites.google.com/site/changmw
SETI@Home profile: http://setiathome.berkeley.edu/view_profile.php?userid=211832
GitHub: https://github.com/changmw/changmw
camsaway
DD-WRT Novice


Joined: 17 Apr 2024
Posts: 10

PostPosted: Sun May 26, 2024 12:56    Post subject: Reply with quote
Ok - so the plot thickens. I've redone the config, and tried an alternative device and I still get the issue.

But here's something weird. When I run TOP I don't see a process using the CPU, and the GUI doesn't show the CPU load - so maybe it's not actually CPU load.

Can anyone help me/direct me how to determine what these OIDs represent?
.1.3.6.1.2.1.25.3.3.1.2.196608
.1.3.6.1.2.1.25.3.3.1.2.196609

It should be the two cores of the processesor from what I can find, but apparently it's not
.iso.org.dod.internet.mgmt.mib-2.host.hrDevice.hrProcessorTable.hrProcessorEntry

Are there other types of processor inside the router it could be representing?

Alternatively - back to my original question - is there anyway to generate more verbose logs that get sent to syslog server so I can try and determine what's happening at exactly 4am each day.
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 6519
Location: UK, London, just across the river..

PostPosted: Sun May 26, 2024 19:13    Post subject: Reply with quote
serial log is the best...and tells everything..
you can do external log (kernel log activated) have a look at services>syslog section..
either sent to a USB or external PC with syslog running...but in both cases it wont be that efficient as serial log..but still better than nothing Razz no idea if it will show the problem...

freezes are usually, bad settings or hardware failure bound... Cool

_________________
Atheros
TP-Link WR740Nv1 ---DD-WRT 57200 WAP
TP-Link WR1043NDv2 -DD-WRT 56941 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 56941 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 57200 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 57200 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
dale_gribble39
DD-WRT Guru


Joined: 11 Jun 2022
Posts: 2054

PostPosted: Sun May 26, 2024 20:58    Post subject: Reply with quote
Or it's something related to ISP or the OPNSense box if it is routinely at 4am...
_________________
"The woods are lovely, dark and deep,
But I have promises to keep,
And miles to go before I sleep,
And miles to go before I sleep." - Robert Frost

"I am one of the noticeable ones - notice me" - Dale Frances McKenzie Bozzio

<fact>code knows no gender</fact>

This is me, knowing I've ruffled your feathers, and not giving a ****
Some people are still hard-headed.

--------------------------------------
Mac Pro (Mid 2012) - Two 2.4GHz 6-Core Intel Xeon E5645 processors 64GB 1333MHz DDR3 ECC SDRAM OpenSUSE Leap 15.5
bushant
DD-WRT Guru


Joined: 18 Nov 2015
Posts: 2073

PostPosted: Sun May 26, 2024 23:27    Post subject: Reply with quote
camsaway wrote:
tried an alternative device and I still get the issue.

What does this mean?
You replaced your Asus RT-AC68U C1 with another router?
Another Asus RT-AC68U C1?
Was it DD-WRT?
With same firmware?

From console (telnet/ssh) with syslog and klog enabled
Code:
nvram set console_debug=1
nvram commit

will give more output.

dale_gribble39 wrote:
Or it's something related to ISP or the OPNSense box if it is routinely at 4am...

+1 👍

_________________
Forum Guide Lines (with helpful pointers about how to research your router, where and what firmware to download, where and how to post and many other helpful tips!)
How to get help the right way

Before asking for help - Read the forum guidelines AND Upgrade DD-WRT!
Adblock by eibgrad (1.1M blocked) + Blocklist Collection
camsaway
DD-WRT Novice


Joined: 17 Apr 2024
Posts: 10

PostPosted: Mon May 27, 2024 4:33    Post subject: Reply with quote
Thanks. I'll give that a go.

And yes, replaced with an identical RT-AC66U-B1 (running 68U firmware as per the router page)
camsaway
DD-WRT Novice


Joined: 17 Apr 2024
Posts: 10

PostPosted: Tue May 28, 2024 16:30    Post subject: Reply with quote
So I've partially figured it out. It was happening at exactly the same time that another WAP on the network was rebooting. Stopped the other nightly reboot and the problem has gone away (1 day so far, but looks promising)

But I am curious to why this is happening.

My setup:
* OPNSense N105 Mini-PC is the Hub,
* igc0 = WAN connection to ISP/Fibre
* igc1 = ZTE MFC286C running OpenWRT
* igc2 = This device running DDWRT (ASUS RT-AC66U B1) - And soon to be a second one as well.
* igc3 = Switched LAN
* I have 4 bridged VLANs configured on OPNSense and the WAPs on both 2.4G and 5G (100 Wired, 200 Wifi, 300 IOT, 400 Guest)

There is no multipathing, but is it possible that a MAC storm is happening due to wireless devices jumping between access points when the one reboots? Do I need to enable STP/RSTP on the VLANs?

Any suggestions welcomed.
Display posts from previous:    Page 1 of 1
Post new topic   Reply to topic    DD-WRT Forum Index -> Broadcom SoC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum