Okay, let me change the settings and enable that, then I will report back the results. Thank you guys for the assistance. I jumped on the internet upgrade without even thinking as I thought these routers were compatible.
Posted: Mon Jun 11, 2018 0:32 Post subject: Netgear R6300v2 Router Report - DD-WRT v3.0-r36070M kongac
Router Model: Netgear R6300V2
Firmware Version: DD-WRT v3.0-r36070M kongac (05/31/18)
Kernel Version: Linux 4.4.134 #568 SMP Thu May 31 11:02:32 CEST 2018 armv7l
Status: Came up just fine. Open VPN (PIA) is fine. Speed is as expected.
Reset: No
Errors: None
Upgraded from: ddup --flash-latest using KiTTY Portable
Router Model: Netgear R6300V2
Firmware Version: DD-WRT v3.0-r35550M kongac (03/28/18)
Kernel Version: Linux 4.4.124 #548 SMP Wed Mar 28 09:52:34 CEST 2018 armv7l
CPU Model: Broadcom BCM4708
CPU Cores: 2
CPU Features: EDSP
CPU Clock: 800 MHz
Load Average: 2% 0.06, 0.05, 0.00
Temperatures: CPU 64.1 °C / WL0 45.1 °C
DHCP Server: Enabled - Running
Samba: Disabled
WRT-radauth: Disabled
WRT-rflow: Disabled
MAC-upd: Disabled
CIFS Automount: Disabled
USB Support: Disabled
WL0 Radio: Radio is On
Mode: AP
Network: Mixed
Channel: 1
TX Power: Auto
Rate: 78 Mbps
Encryption - Interface wl0: Enabled, WPA2 Personal
WL1: Disabled
I have an r7000 and with this build the ipsec strongswan server is broken (again after waiting a lot for a fix). Connects successfully but no access to lan/wan. This time doesn't seem a dns problem because even when entering an ip it doesn't work. Works fine on previous build though. Also I noticed that sometimes, like when adding custom ports forwarding, the firewall rules for accepting ipsec connections from outside are discarded and is needed to manually press apply settings on vpn services page or manually open udp ports 500/4500.
Posted: Tue Jun 12, 2018 1:33 Post subject: Re: New Kong's Test Build v3.0-r36070M kongac (31/05/2018)
[quote="unixpunk"]
<Kong> wrote:
unixpunk wrote:
Router: Netgear R6400 v1(.0.31)
Firmware: New Kong's Test Build v3.0-r36070M kongac (31/05/2018)
Kernel: Linux 4.4.134 #568 SMP Thu May 31 11:02:32 CEST 2018 armv7l DD-WRT
Status: Freezes and somehow brings down entire network, even across multiple switches, really amazing actually considering my machine saw no extra/out of the ordinary packets in promisc w/wireshark yet the lan light was flashing at seizure-levels...
Reset: Yes
Errors: #1 - After a random amount of time httpd runs out of memory somehow and brings down the entire system and network. This has happened to me consistently on EVERY dd-wrt build I've tried so far. Have serial access, happy to debug/reproduce, etc. Happens with http and/or https enabled, only option I haven't tried yet is neither. Amount of time it takes is always random and different.
[Edit, added] #2 - Router always attempts tftp boot (boot wait) even if option is disabled, this adds 30 seconds to boot up time, easy.
******Serial output from boot
CFE for Foxconn Router R6400 version: v1.0.31
Build Date: Tue Apr 14 17:28:19 CST 2015
Init Arena
Init Devs.
Boot up from NAND flash...
Bootcode Boot partition size = 524288(0x80000)
DDR Clock: 533 MHz
Info: DDR frequency set from clkfreq=800,*533*
et0: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 6.37.15.1 (r407936)
et1: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 6.37.15.1 (r407936)
CPU type 0x0: 800MHz
Tot mem: 262144 KBytes
This is not httpd problem, this is a nas (network authentication daemon) problem. I have seen this behavior once on my own router, nas will broadcast like crazy, which can also bring down other devices with bridges.
The httpd memory print out is most likely just a result of the overload, that these broadcast cause.
I have no idea how this is triggered as I was never able to reproduce it, just turning off/on the router fixed it and the same config has been running for month without the issue.
I recommend you fully clear the config with nvram erase if not done before configuring the router.
I had been struggling for weeks to figure out a problem almost identical to this on my R8500.
It started back in march (not sure if it was a firmware change) and I can get it to trigger consistently within 4-24 hours. Once it starts, it seems to triggers more frequently. I also occasionally noticed a high-pitched noise out of the R8500 in the times after it had crashed and rebooted on its own (with no power cycle).
Rebuilt it from scratch after erasing the nvram, shut off all unnecessary services, and the OOM issue continued.
I noticed a flood of DHCP discover/offers, sometimes within a second of the OOM getting triggered. These came from my Logitech Harmony Hub device primarily.
After setting a static entry in the DNS table that was outside of my normal DHCP range with a lease time, the DHCP floods went away.
However the OOM crashes continued.
After reading Kong's comment about the NAS daemon, I focused on the wireless network.
I had a AP/Bridge (different ssid using wl2 as the backhaul) and I removed that from the network and it made no difference
Finally, I moved my Harmony hub (& Nest) to the wireless network on the other side of my bridge and it has been stable ever since (so far around 32 hours)
Posted: Tue Jun 12, 2018 2:47 Post subject: Re: New Kong's Test Build v3.0-r36070M kongac (31/05/2018)
[quote="kiva113"]
unixpunk wrote:
<Kong> wrote:
unixpunk wrote:
Router: Netgear R6400 v1(.0.31)
Firmware: New Kong's Test Build v3.0-r36070M kongac (31/05/2018)
Kernel: Linux 4.4.134 #568 SMP Thu May 31 11:02:32 CEST 2018 armv7l DD-WRT
Status: Freezes and somehow brings down entire network, even across multiple switches, really amazing actually considering my machine saw no extra/out of the ordinary packets in promisc w/wireshark yet the lan light was flashing at seizure-levels...
Reset: Yes
Errors: #1 - After a random amount of time httpd runs out of memory somehow and brings down the entire system and network. This has happened to me consistently on EVERY dd-wrt build I've tried so far. Have serial access, happy to debug/reproduce, etc. Happens with http and/or https enabled, only option I haven't tried yet is neither. Amount of time it takes is always random and different.
[Edit, added] #2 - Router always attempts tftp boot (boot wait) even if option is disabled, this adds 30 seconds to boot up time, easy.
******Serial output from boot
CFE for Foxconn Router R6400 version: v1.0.31
Build Date: Tue Apr 14 17:28:19 CST 2015
Init Arena
Init Devs.
Boot up from NAND flash...
Bootcode Boot partition size = 524288(0x80000)
DDR Clock: 533 MHz
Info: DDR frequency set from clkfreq=800,*533*
et0: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 6.37.15.1 (r407936)
et1: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 6.37.15.1 (r407936)
CPU type 0x0: 800MHz
Tot mem: 262144 KBytes
This is not httpd problem, this is a nas (network authentication daemon) problem. I have seen this behavior once on my own router, nas will broadcast like crazy, which can also bring down other devices with bridges.
The httpd memory print out is most likely just a result of the overload, that these broadcast cause.
I have no idea how this is triggered as I was never able to reproduce it, just turning off/on the router fixed it and the same config has been running for month without the issue.
I recommend you fully clear the config with nvram erase if not done before configuring the router.
I had been struggling for weeks to figure out a problem almost identical to this on my R8500.
It started back in march (not sure if it was a firmware change) and I can get it to trigger consistently within 4-24 hours. Once it starts, it seems to triggers more frequently. I also occasionally noticed a high-pitched noise out of the R8500 in the times after it had crashed and rebooted on its own (with no power cycle).
Rebuilt it from scratch after erasing the nvram, shut off all unnecessary services, and the OOM issue continued.
I noticed a flood of DHCP discover/offers, sometimes within a second of the OOM getting triggered. These came from my Logitech Harmony Hub device primarily.
After setting a static entry in the DNS table that was outside of my normal DHCP range with a lease time, the DHCP floods went away.
However the OOM crashes continued.
After reading Kong's comment about the NAS daemon, I focused on the wireless network.
I had a AP/Bridge (different ssid using wl2 as the backhaul) and I removed that from the network and it made no difference
Finally, I moved my Harmony hub (& Nest) to the wireless network on the other side of my bridge and it has been stable ever since (so far around 32 hours)
My suggestion, look to see if you have a specific wireless device causing the issues.
Thanks for the info! I mostly have my wifi scheduled off (using cron and 'wl radio off' and I get the crash even then. Maybe because nas is still running at that point. Interesting find on the Logitech. I don't have that device but I will research to see what network protocol/apps it supports and maybe find some comparable device here, i.e., Avahi, etc.
I've also tried several different things short of disabling all wifi altogether. 1 hour dhcp leases, turning off various services, etc. I'll see tonight, I expect it to barf again.
Posted: Tue Jun 12, 2018 14:26 Post subject: Re: New Kong's Test Build v3.0-r36070M kongac (31/05/2018)
[quote="unixpunk"]
kiva113 wrote:
unixpunk wrote:
<Kong> wrote:
unixpunk wrote:
Router: Netgear R6400 v1(.0.31)
Firmware: New Kong's Test Build v3.0-r36070M kongac (31/05/2018)
Kernel: Linux 4.4.134 #568 SMP Thu May 31 11:02:32 CEST 2018 armv7l DD-WRT
Status: Freezes and somehow brings down entire network, even across multiple switches, really amazing actually considering my machine saw no extra/out of the ordinary packets in promisc w/wireshark yet the lan light was flashing at seizure-levels...
Reset: Yes
Errors: #1 - After a random amount of time httpd runs out of memory somehow and brings down the entire system and network. This has happened to me consistently on EVERY dd-wrt build I've tried so far. Have serial access, happy to debug/reproduce, etc. Happens with http and/or https enabled, only option I haven't tried yet is neither. Amount of time it takes is always random and different.
[Edit, added] #2 - Router always attempts tftp boot (boot wait) even if option is disabled, this adds 30 seconds to boot up time, easy.
******Serial output from boot
CFE for Foxconn Router R6400 version: v1.0.31
Build Date: Tue Apr 14 17:28:19 CST 2015
Init Arena
Init Devs.
Boot up from NAND flash...
Bootcode Boot partition size = 524288(0x80000)
DDR Clock: 533 MHz
Info: DDR frequency set from clkfreq=800,*533*
et0: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 6.37.15.1 (r407936)
et1: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 6.37.15.1 (r407936)
CPU type 0x0: 800MHz
Tot mem: 262144 KBytes
This is not httpd problem, this is a nas (network authentication daemon) problem. I have seen this behavior once on my own router, nas will broadcast like crazy, which can also bring down other devices with bridges.
The httpd memory print out is most likely just a result of the overload, that these broadcast cause.
I have no idea how this is triggered as I was never able to reproduce it, just turning off/on the router fixed it and the same config has been running for month without the issue.
I recommend you fully clear the config with nvram erase if not done before configuring the router.
I had been struggling for weeks to figure out a problem almost identical to this on my R8500.
It started back in march (not sure if it was a firmware change) and I can get it to trigger consistently within 4-24 hours. Once it starts, it seems to triggers more frequently. I also occasionally noticed a high-pitched noise out of the R8500 in the times after it had crashed and rebooted on its own (with no power cycle).
Rebuilt it from scratch after erasing the nvram, shut off all unnecessary services, and the OOM issue continued.
I noticed a flood of DHCP discover/offers, sometimes within a second of the OOM getting triggered. These came from my Logitech Harmony Hub device primarily.
After setting a static entry in the DNS table that was outside of my normal DHCP range with a lease time, the DHCP floods went away.
However the OOM crashes continued.
After reading Kong's comment about the NAS daemon, I focused on the wireless network.
I had a AP/Bridge (different ssid using wl2 as the backhaul) and I removed that from the network and it made no difference
Finally, I moved my Harmony hub (& Nest) to the wireless network on the other side of my bridge and it has been stable ever since (so far around 32 hours)
My suggestion, look to see if you have a specific wireless device causing the issues.
Thanks for the info! I mostly have my wifi scheduled off (using cron and 'wl radio off' and I get the crash even then. Maybe because nas is still running at that point. Interesting find on the Logitech. I don't have that device but I will research to see what network protocol/apps it supports and maybe find some comparable device here, i.e., Avahi, etc.
I've also tried several different things short of disabling all wifi altogether. 1 hour dhcp leases, turning off various services, etc. I'll see tonight, I expect it to barf again.
Update, it seems that this is really only currently happening when the radio is off via 'wl radio off' command. I only use 2.4ghz (WPA2 Personal,AES), so 5ghz is disabled in the UI. I use cron to schedule turning wifi on and off because I need different schedules for different days and last I checked the radio scheduler doesn't offer that.
So if anyone is willing to test, run the command 'wl radio off' and wait up to 2 hours or so and see if things are hosed up.
I think I might be able to find a workaround where I just kill the nas process when i turn off the radio and then restart it when turning the radio back on. Or maybe there's a better way to turn off the wifi which kills nas automatically as well?
Router Model: Asus RT-5300
Firmware Version: DD-WRT v3.0-r36070M kongac (05/31/2018)
Kernel Version: Linux 4.4.134 #568 SMP Thu May 31 11:02:32 CEST 2018 armv7l
Reset: Yes
Status: Lasted about two days before radio issues. Back to DD-WRT v3.0-r35030M kongac (02/19/18). For my setup, no other recent versions have worked as well.
WL0 - 2.4ghz - ap
wl1 - 5ghz - ap
wl2 - 5ghz - client to hotspot
Kong and BrainSlayer, thank you!
UPDATE - Gave it another go. Seen too much packet loss after a couple days before. This time with bluetooth coexistence enabled. Seems to be working. Will continue to test.
UPDATE#2 - Have gone back to 2/19/2018.
¡BBB!
Health is Wealth _________________ Asus RT-AC5300, Netgear R8500, Netgear R8000, Netgear R6100, Linksys E2500, Linksys E2000, Netgear R7000, Netgear WNDR4500, Netgear WNDR3800, Linksys WRT54Gv4, Linksys WRT54Gv1.1
Last edited by realbbb on Thu Jul 19, 2018 17:56; edited 3 times in total
Posted: Wed Jun 13, 2018 3:16 Post subject: Re: New Kong's Test Build v3.0-r36070M kongac (31/05/2018)
[quote="unixpunk"]
unixpunk wrote:
kiva113 wrote:
unixpunk wrote:
<Kong> wrote:
unixpunk wrote:
Router: Netgear R6400 v1(.0.31)
Firmware: New Kong's Test Build v3.0-r36070M kongac (31/05/2018)
Kernel: Linux 4.4.134 #568 SMP Thu May 31 11:02:32 CEST 2018 armv7l DD-WRT
Status: Freezes and somehow brings down entire network, even across multiple switches, really amazing actually considering my machine saw no extra/out of the ordinary packets in promisc w/wireshark yet the lan light was flashing at seizure-levels...
Reset: Yes
Errors: #1 - After a random amount of time httpd runs out of memory somehow and brings down the entire system and network. This has happened to me consistently on EVERY dd-wrt build I've tried so far. Have serial access, happy to debug/reproduce, etc. Happens with http and/or https enabled, only option I haven't tried yet is neither. Amount of time it takes is always random and different.
[Edit, added] #2 - Router always attempts tftp boot (boot wait) even if option is disabled, this adds 30 seconds to boot up time, easy.
******Serial output from boot
CFE for Foxconn Router R6400 version: v1.0.31
Build Date: Tue Apr 14 17:28:19 CST 2015
Init Arena
Init Devs.
Boot up from NAND flash...
Bootcode Boot partition size = 524288(0x80000)
DDR Clock: 533 MHz
Info: DDR frequency set from clkfreq=800,*533*
et0: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 6.37.15.1 (r407936)
et1: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 6.37.15.1 (r407936)
CPU type 0x0: 800MHz
Tot mem: 262144 KBytes
This is not httpd problem, this is a nas (network authentication daemon) problem. I have seen this behavior once on my own router, nas will broadcast like crazy, which can also bring down other devices with bridges.
The httpd memory print out is most likely just a result of the overload, that these broadcast cause.
I have no idea how this is triggered as I was never able to reproduce it, just turning off/on the router fixed it and the same config has been running for month without the issue.
I recommend you fully clear the config with nvram erase if not done before configuring the router.
I had been struggling for weeks to figure out a problem almost identical to this on my R8500.
It started back in march (not sure if it was a firmware change) and I can get it to trigger consistently within 4-24 hours. Once it starts, it seems to triggers more frequently. I also occasionally noticed a high-pitched noise out of the R8500 in the times after it had crashed and rebooted on its own (with no power cycle).
Rebuilt it from scratch after erasing the nvram, shut off all unnecessary services, and the OOM issue continued.
I noticed a flood of DHCP discover/offers, sometimes within a second of the OOM getting triggered. These came from my Logitech Harmony Hub device primarily.
After setting a static entry in the DNS table that was outside of my normal DHCP range with a lease time, the DHCP floods went away.
However the OOM crashes continued.
After reading Kong's comment about the NAS daemon, I focused on the wireless network.
I had a AP/Bridge (different ssid using wl2 as the backhaul) and I removed that from the network and it made no difference
Finally, I moved my Harmony hub (& Nest) to the wireless network on the other side of my bridge and it has been stable ever since (so far around 32 hours)
My suggestion, look to see if you have a specific wireless device causing the issues.
Thanks for the info! I mostly have my wifi scheduled off (using cron and 'wl radio off' and I get the crash even then. Maybe because nas is still running at that point. Interesting find on the Logitech. I don't have that device but I will research to see what network protocol/apps it supports and maybe find some comparable device here, i.e., Avahi, etc.
I've also tried several different things short of disabling all wifi altogether. 1 hour dhcp leases, turning off various services, etc. I'll see tonight, I expect it to barf again.
Update, it seems that this is really only currently happening when the radio is off via 'wl radio off' command. I only use 2.4ghz (WPA2 Personal,AES), so 5ghz is disabled in the UI. I use cron to schedule turning wifi on and off because I need different schedules for different days and last I checked the radio scheduler doesn't offer that.
So if anyone is willing to test, run the command 'wl radio off' and wait up to 2 hours or so and see if things are hosed up.
I think I might be able to find a workaround where I just kill the nas process when i turn off the radio and then restart it when turning the radio back on. Or maybe there's a better way to turn off the wifi which kills nas automatically as well?
Here is my attempted workaround. I'll know by morning if it helped or not. First attempt, probably a more elegant way I've yet to discover. Instead of just wl radio on or wl radio off:
/usr/sbin/wl radio off && ps | grep nas | grep -v grep | awk '{ $1=$2=$3=$4=""; print $0 }' | sed 's/ //' >/tmp/nas.save && chmod 600 /tmp/nas.save && kill `cat /tmp/nas.wl*lan.pid`
/usr/sbin/wl radio on && sh /tmp/nas.save && rm -f /tmp/nas.save
Posted: Wed Jun 13, 2018 3:40 Post subject: Re: New Kong's Test Build v3.0-r36070M kongac (31/05/2018)
[quote="unixpunk"]
unixpunk wrote:
unixpunk wrote:
kiva113 wrote:
unixpunk wrote:
<Kong> wrote:
unixpunk wrote:
Router: Netgear R6400 v1(.0.31)
Firmware: New Kong's Test Build v3.0-r36070M kongac (31/05/2018)
Kernel: Linux 4.4.134 #568 SMP Thu May 31 11:02:32 CEST 2018 armv7l DD-WRT
Status: Freezes and somehow brings down entire network, even across multiple switches, really amazing actually considering my machine saw no extra/out of the ordinary packets in promisc w/wireshark yet the lan light was flashing at seizure-levels...
Reset: Yes
Errors: #1 - After a random amount of time httpd runs out of memory somehow and brings down the entire system and network. This has happened to me consistently on EVERY dd-wrt build I've tried so far. Have serial access, happy to debug/reproduce, etc. Happens with http and/or https enabled, only option I haven't tried yet is neither. Amount of time it takes is always random and different.
[Edit, added] #2 - Router always attempts tftp boot (boot wait) even if option is disabled, this adds 30 seconds to boot up time, easy.
******Serial output from boot
CFE for Foxconn Router R6400 version: v1.0.31
Build Date: Tue Apr 14 17:28:19 CST 2015
Init Arena
Init Devs.
Boot up from NAND flash...
Bootcode Boot partition size = 524288(0x80000)
DDR Clock: 533 MHz
Info: DDR frequency set from clkfreq=800,*533*
et0: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 6.37.15.1 (r407936)
et1: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 6.37.15.1 (r407936)
CPU type 0x0: 800MHz
Tot mem: 262144 KBytes
[TRUNCATED TO SAVE SPACE]
This is not httpd problem, this is a nas (network authentication daemon) problem. I have seen this behavior once on my own router, nas will broadcast like crazy, which can also bring down other devices with bridges.
The httpd memory print out is most likely just a result of the overload, that these broadcast cause.
I have no idea how this is triggered as I was never able to reproduce it, just turning off/on the router fixed it and the same config has been running for month without the issue.
I recommend you fully clear the config with nvram erase if not done before configuring the router.
I had been struggling for weeks to figure out a problem almost identical to this on my R8500.
It started back in march (not sure if it was a firmware change) and I can get it to trigger consistently within 4-24 hours. Once it starts, it seems to triggers more frequently. I also occasionally noticed a high-pitched noise out of the R8500 in the times after it had crashed and rebooted on its own (with no power cycle).
Rebuilt it from scratch after erasing the nvram, shut off all unnecessary services, and the OOM issue continued.
I noticed a flood of DHCP discover/offers, sometimes within a second of the OOM getting triggered. These came from my Logitech Harmony Hub device primarily.
After setting a static entry in the DNS table that was outside of my normal DHCP range with a lease time, the DHCP floods went away.
However the OOM crashes continued.
After reading Kong's comment about the NAS daemon, I focused on the wireless network.
I had a AP/Bridge (different ssid using wl2 as the backhaul) and I removed that from the network and it made no difference
Finally, I moved my Harmony hub (& Nest) to the wireless network on the other side of my bridge and it has been stable ever since (so far around 32 hours)
My suggestion, look to see if you have a specific wireless device causing the issues.
Thanks for the info! I mostly have my wifi scheduled off (using cron and 'wl radio off' and I get the crash even then. Maybe because nas is still running at that point. Interesting find on the Logitech. I don't have that device but I will research to see what network protocol/apps it supports and maybe find some comparable device here, i.e., Avahi, etc.
I've also tried several different things short of disabling all wifi altogether. 1 hour dhcp leases, turning off various services, etc. I'll see tonight, I expect it to barf again.
Update, it seems that this is really only currently happening when the radio is off via 'wl radio off' command. I only use 2.4ghz (WPA2 Personal,AES), so 5ghz is disabled in the UI. I use cron to schedule turning wifi on and off because I need different schedules for different days and last I checked the radio scheduler doesn't offer that.
So if anyone is willing to test, run the command 'wl radio off' and wait up to 2 hours or so and see if things are hosed up.
I think I might be able to find a workaround where I just kill the nas process when i turn off the radio and then restart it when turning the radio back on. Or maybe there's a better way to turn off the wifi which kills nas automatically as well?
Here is my attempted workaround. I'll know by morning if it helped or not. First attempt, probably a more elegant way I've yet to discover. Instead of just wl radio on or wl radio off:
/usr/sbin/wl radio off && ps | grep nas | grep -v grep | awk '{ $1=$2=$3=$4=""; print $0 }' | sed 's/ //' >/tmp/nas.save && chmod 600 /tmp/nas.save && kill `cat /tmp/nas.wl*lan.pid`
/usr/sbin/wl radio on && sh /tmp/nas.save && rm -f /tmp/nas.save
Thanks all for the eyes on this!
I checked in /tmp/cron.d/cron_jobs and it looks like the webUI cron field can't properly parse commands with single-quotes. It causes it to replace them with backslashes...This means it won't work anyway...trying to escape with \ doesn't work either. I ended up having to put the crontab into a file and then use nvram set cron_jobs="`cat /tmp/crontabs'" [edit added] and updated /tmp/cron.d/cron_jobs. We'll see in the morning.
Posted: Wed Jun 13, 2018 13:50 Post subject: Re: New Kong's Test Build v3.0-r36070M kongac (31/05/2018)
[quote="unixpunk"]
unixpunk wrote:
unixpunk wrote:
unixpunk wrote:
kiva113 wrote:
unixpunk wrote:
<Kong> wrote:
unixpunk wrote:
Router: Netgear R6400 v1(.0.31)
Firmware: New Kong's Test Build v3.0-r36070M kongac (31/05/2018)
Kernel: Linux 4.4.134 #568 SMP Thu May 31 11:02:32 CEST 2018 armv7l DD-WRT
Status: Freezes and somehow brings down entire network, even across multiple switches, really amazing actually considering my machine saw no extra/out of the ordinary packets in promisc w/wireshark yet the lan light was flashing at seizure-levels...
Reset: Yes
Errors: #1 - After a random amount of time httpd runs out of memory somehow and brings down the entire system and network. This has happened to me consistently on EVERY dd-wrt build I've tried so far. Have serial access, happy to debug/reproduce, etc. Happens with http and/or https enabled, only option I haven't tried yet is neither. Amount of time it takes is always random and different.
[Edit, added] #2 - Router always attempts tftp boot (boot wait) even if option is disabled, this adds 30 seconds to boot up time, easy.
******Serial output from boot
CFE for Foxconn Router R6400 version: v1.0.31
Build Date: Tue Apr 14 17:28:19 CST 2015
Init Arena
Init Devs.
Boot up from NAND flash...
Bootcode Boot partition size = 524288(0x80000)
DDR Clock: 533 MHz
Info: DDR frequency set from clkfreq=800,*533*
et0: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 6.37.15.1 (r407936)
et1: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 6.37.15.1 (r407936)
CPU type 0x0: 800MHz
Tot mem: 262144 KBytes
[TRUNCATED TO SAVE SPACE]
This is not httpd problem, this is a nas (network authentication daemon) problem. I have seen this behavior once on my own router, nas will broadcast like crazy, which can also bring down other devices with bridges.
The httpd memory print out is most likely just a result of the overload, that these broadcast cause.
I have no idea how this is triggered as I was never able to reproduce it, just turning off/on the router fixed it and the same config has been running for month without the issue.
I recommend you fully clear the config with nvram erase if not done before configuring the router.
I had been struggling for weeks to figure out a problem almost identical to this on my R8500.
It started back in march (not sure if it was a firmware change) and I can get it to trigger consistently within 4-24 hours. Once it starts, it seems to triggers more frequently. I also occasionally noticed a high-pitched noise out of the R8500 in the times after it had crashed and rebooted on its own (with no power cycle).
Rebuilt it from scratch after erasing the nvram, shut off all unnecessary services, and the OOM issue continued.
I noticed a flood of DHCP discover/offers, sometimes within a second of the OOM getting triggered. These came from my Logitech Harmony Hub device primarily.
After setting a static entry in the DNS table that was outside of my normal DHCP range with a lease time, the DHCP floods went away.
However the OOM crashes continued.
After reading Kong's comment about the NAS daemon, I focused on the wireless network.
I had a AP/Bridge (different ssid using wl2 as the backhaul) and I removed that from the network and it made no difference
Finally, I moved my Harmony hub (& Nest) to the wireless network on the other side of my bridge and it has been stable ever since (so far around 32 hours)
My suggestion, look to see if you have a specific wireless device causing the issues.
Thanks for the info! I mostly have my wifi scheduled off (using cron and 'wl radio off' and I get the crash even then. Maybe because nas is still running at that point. Interesting find on the Logitech. I don't have that device but I will research to see what network protocol/apps it supports and maybe find some comparable device here, i.e., Avahi, etc.
I've also tried several different things short of disabling all wifi altogether. 1 hour dhcp leases, turning off various services, etc. I'll see tonight, I expect it to barf again.
Update, it seems that this is really only currently happening when the radio is off via 'wl radio off' command. I only use 2.4ghz (WPA2 Personal,AES), so 5ghz is disabled in the UI. I use cron to schedule turning wifi on and off because I need different schedules for different days and last I checked the radio scheduler doesn't offer that.
So if anyone is willing to test, run the command 'wl radio off' and wait up to 2 hours or so and see if things are hosed up.
I think I might be able to find a workaround where I just kill the nas process when i turn off the radio and then restart it when turning the radio back on. Or maybe there's a better way to turn off the wifi which kills nas automatically as well?
Here is my attempted workaround. I'll know by morning if it helped or not. First attempt, probably a more elegant way I've yet to discover. Instead of just wl radio on or wl radio off:
/usr/sbin/wl radio off && ps | grep nas | grep -v grep | awk '{ $1=$2=$3=$4=""; print $0 }' | sed 's/ //' >/tmp/nas.save && chmod 600 /tmp/nas.save && kill `cat /tmp/nas.wl*lan.pid`
/usr/sbin/wl radio on && sh /tmp/nas.save && rm -f /tmp/nas.save
Thanks all for the eyes on this!
I checked in /tmp/cron.d/cron_jobs and it looks like the webUI cron field can't properly parse commands with single-quotes. It causes it to replace them with backslashes...This means it won't work anyway...trying to escape with \ doesn't work either. I ended up having to put the crontab into a file and then use nvram set cron_jobs="`cat /tmp/crontabs'" [edit added] and updated /tmp/cron.d/cron_jobs. We'll see in the morning.
No change here even with nas process killed...are we sure this is related to nas or am I killing the wrong process here?
Any additional advice/direction is greatly appreciated! Thanks all!
Do not use wl radio on/off, radio off does not take care of stopping the related services and interfaces, depending on config and usage this causes memory to be consumed without ever freed.
Do not use wl radio on/off, radio off does not take care of stopping the related services and interfaces, depending on config and usage this causes memory to be consumed without ever freed.
Thanks for the advice! I assume your duplicate of startservice was accidental in that one would be stopservice, assuming no need to run it twice. I will get this tested soon and report back. [edit] Or maybe I'm wrong and i need to replace off with on to do the opposite, will test here either way.
I'll look around for a feature request section to post in, but if radio scheduling supported a grid/matrix with no only hours of the day, but also days of the week, this would eliminate my need to use cron for this. Thanks for all your work on this! (BrainSlayer too!) Been running dd-wrt for like a decade on WRT54G's...just now upgrading since it can't keep up with my internet speed anymore.