[tkmds]

Hey Folks,

I'm connected as follows Laptop -> Wireless to -> Asus Router #1 -> Wired to -> Asus Router Internet -> WWW.

When browsing various pages on my laptop, the 'Asus Internet (192.168.0.6)' routers sirq spikes, then at times the router is inaccessible or shows a ping of 3+ seconds. The sirq shoots up to:

30% - 50%+ sirq

All my routers are on firmware r49392. Wondering, what could be the issue here and how to find more information on this?

Cheers,
TK

[Hapi12021]

Software interrupt requests (sIRQ) happen when the kernel is busy handling devices or pseudo devices.

My guess is that somehow you are inspecting the traffic on your internal network with the firewall or another packet filter, when it should be disabled for a strictly internal-only AP, or something like using RIP where a static route would be necessary, instead.

You could also have a bridging loop on your network and tripping STP, if enabled, or getting an old-school packet storm.

If radvd is chewing up CPU then look to devices that aren’t behaving properly on IPv6 or possibly having inadvertently blocked some core IPv6 neighbor protocol or port.

[tkmds]

Thank you very much for the feedback. This is an external facing AP hence have been running a F/W on this unit. STP is disabled. Just switched over to CTF for the Shortcut Forwarding Engine but that had negligible effect. Do have IPV6 enabled however.

Trying to capture some tcpdump results at the time of the high SIRQ usages when router also reboots from time to time.

Captured additional logs. The time of 04:14 GMT is when the router hits 100% SIRQ:

[tkmds]

A few messages around shutdown:

[tkmds]

"or getting an old-school packet storm."

Possibly, based on some tcpdumps taken when sirq is high:

[Mile-Lile]

I saw similar behaviour on my r8000 with brcmfmac (but not with dhd)...

can you post output of:

[egc]

First follow @Hapi12021's and @Mile-Lile's advice

But other things to try, do not use SFE (shortcut forwarding engine on setup page) but if you need the bandwidth use CTF + FA, if you do not need the bandwidth do not use any acceleration.

Not sure what yo are running on the router but if you have a lot of applications running you might need a SWAP file

[tkmds]

[tkmds]

[Mile-Lile]

FA is some kind of NAT acceleration and is brcm specific... On some router models you don't have option enable/disable (e.g. R8000) because it is already enabled (not sure) by default (it is usually on Index.asp at SFE section)... you can enable it with nvram variable but it will cause router to bootloop... outputs and screenshot you posted looks normal...

I have had disabled SFE, CTF&FA but still had crashes (brcmfmac)... Then when I disabled everything I use (ipv6, WG, QoS, NextDNS CLI) I didn't had crashes... I had 10 days router uptime... but as soon as I enable for example ipv6 after some time brcmfmac starts crashing... output of crash doesn't tell much... BS told me cause is unknown... and is very hard to find it... But I noticed similar symptoms like you do (during these 10 days of uptime)... CPU spikes goes to 2.5... with just basic functions... With normal build (non-experimental with dhd driver) I use bunch of apps on router, and router is rock solid... but wifi is much better on brcmfmac...

[tkmds]

When SFE is disabled, DNS resolution is toast, except for internal DNS resolution to internal DNS hosts.

For example can't nslookup any domain. There's no result. Guess this would explain why SIRQ is very high. All DNS requests are going through this service which would explain why simple browing resolution is slow and causes the router to reboot. Now I'm trying to find how to enable DNS resolution without enabling SFE or CTF. Even typing something like nslookup SITE EXTERNAL-DNS fails.

Thank you for the write up Mile-Lile. Flow Acceleration (FA) was already off:

# nvram show|grep -Ei ctf_fa_cap
size: 55203 bytes (75869 left)
ctf_fa_cap=0


Cheers,

[tkmds]

Looks like these settings across my routers seeming improved things a bit. Time will tell:

Shortcut Forwarding Engine: CTF
STP: Disabled

MTU:    Manual    1452

# nvram show|grep -Ei ctf_fa_cap=1
size: 55223 bytes (75849 left)
ctf_fa_cap=1

NOTE: The last option may or may not be ideal for your router, depending if Flow Acceleration (FA) module is included in your setup.

https://microdevsys.com/wp/dd-wrt-network-performance-tuning/

[tkmds]

Looks like I'll need DNS caching on my DNS servers. (facepalm) In the span of 5 minutes, 4K DNS queries occurred. This doesn't look right nor would it be appropriate for the system to do this many DNS queries.

[Alozaros]

hmm, looking at the thread and your heavy router use...do have USB as a storage as this is very CPU heavy if more than one client is using it at the same time...than you may need to increase the USB buffer size..may be..

also speaking of DNS cashing..yep its a good to have it, if there a many client at the same time...you can even set it to 2000-3000-5000 as default it 1500 (an this is lines(hits) not kb or bites) although 1500 is more than enough...

also to improve DNS if you use DNSmasq... you can increase the maximum number of concurrent DNS queries
the default value is 150, but you can set it to 300 or 500 if so...
add this line to advanced DNSmasq rules

dns-forward-max=500

for an edge routers with heavy use:
and lots of DNS requests you can fiddle with few dnsmasq options too
on heavy load, systems those are the settings i use in general

domain-needed
no-ping
no-resov
server=9.9.9.9
no-negcache
cache-size=3000
filterwin2k
dhcp-option=43,01:04:00:00:00:02
dns-loop-detect
quiet-dhcp

you can also fiddle with:
max-cache-ttl=
min-cache-ttl=
max-ttl=

https://dnsmasq.org/docs/dnsmasq-man.html

[tkmds]

Thanks @Alozaros!

I have DNSmasquerade disabled. Haven't been using it for some time. I rely on 6 internal DNS servers, 3x2 clusters. I've enabled caching on one of them, set it to 64MB with a 2 day expiry. That DNS server was set to 0KB w/ 1 day expiry, hence apparently disabled I guess. Page loading and lookup is way faster. So this helps. Will monitor.

Can DNSmasquerade help with external DNS caching in any way? Tried enabling it but it didn't work. I'll have to spend more time to figure out how to get DNSmasquerade working w/ DNS servers sitting on VM's on the internal network.

I do have IPV6 enabled as well on the 4 DDWRT routers. I have all 4 with same FW version. Thinking this is contributing heavily but can't prove it yet. Disabled non OSPF services like BGP to reduce memory and CPU consumption which should help a bit.

For USG buffer size, thinking you mean "Drive Read-Ahead Buffer"? It's currently set to 256.

Regardless, on a secondary router, I ended up getting OOM with version 49392, which leads to reboots. Attached the logs of the OOM.

Can't pin point the culprit yet.

Cheers,