Joined: 31 Jul 2021 Posts: 2146 Location: All over YOUR webs
Posted: Fri Jul 01, 2022 17:59 Post subject:
You can open a terminal to DD-WRT login and run nvram set console_debug=1 && nvram commit && reboot this enables verbose kernel messages.
Best really are serial logs, these will show whats going on behind the curtains.
However you are running as seriously outdated DD-WRT version my recommendation is upgrade, nvram reset and reconfigure from scratch , many issues fixed and security patches amongst at few.
Posted: Fri Jul 01, 2022 18:08 Post subject: Re: Random restarts andDetailed rsyslog (ddwrt kernel) messa
tkmds wrote:
I have 3 AC-RT88U running DDWRT 47000+ firmware. After a recent power outage, all 3 randomly reboot outside regular reboot cycles I defined.
Have you examined the devices and their AC power adapters and associated surge suppressors for signs of damage? _________________ "The woods are lovely, dark and deep,
But I have promises to keep,
And miles to go before I sleep,
And miles to go before I sleep." - Robert Frost
"I am one of the noticeable ones - notice me" - Dale Frances McKenzie Bozzio
Router 3 is most stable so far. Though, given your comment, I think I'll try the latest here as well and see how that fares in the next few days.
I think I figured out the culprit though I'm not 100% yet. Each router is configured to reboot at 2AM each night but is also configured to source the time from a single internal NTP server.
This internal NTP server is currently offline. Meaning the date on each router is pretty much random. So the cron that kicks off the job now kicks off against the time of the current set date and time, which is already off.
Idea behind the internal NTP sever is not to have 100's of devices off my network ping an external source, but just one, then the rest of the 99 devices can use the internal one for time sync.
Part of being a good 'netizen' I suppose, and not overload external NTP servers too much.
"Have you examined the devices and their AC power adapters and associated surge suppressors for signs of damage?"
Did examine them. Aside from some more frequent WiFi disconnects, which could be attributed to the reboots, everything else seems to work ok.
I think I figured out the culprit though I'm not 100% yet. Each router is configured to reboot at 2AM each night but is also configured to source the time from a single internal NTP server.
This internal NTP server is currently offline. Meaning the date on each router is pretty much random. So the cron that kicks off the job now kicks off against the time of the current set date and time, which is already off.
I suspect that aside from updating firmware, this is the crux of your problems; there are things that rely on NTP to be in sync, such as your scheduled reboot. _________________ "The woods are lovely, dark and deep,
But I have promises to keep,
And miles to go before I sleep,
And miles to go before I sleep." - Robert Frost
"I am one of the noticeable ones - notice me" - Dale Frances McKenzie Bozzio
Joined: 31 Jul 2021 Posts: 2146 Location: All over YOUR webs
Posted: Sat Jul 02, 2022 12:17 Post subject:
So you know, NTP is essential on consumer routers since they dont have a real time clock only software clock and depending on CPU load this will skew more on higher loads.
Everything then that depends on cron will be all over the place, firewall, some services and logging, all affected by bad local time.
You can on recent current dd-wrt release, tweak the NTP update interval, by default its one hour, but 48 hours will be a never mind (little skew then if any) and better your Netizenship status.
As for console_debug, no it shouldn't help anything wifi related best it will do is print verbose kernel messages it will be a great deal of entries as you will notice,
You should be running same builds across routers, some wifi drivers were updated recently, since this is broadcom its also closed source wifi drivers.
Alternatively you can set up your own ntp server running on pretty much anything (like a RaspberryPi, although something with internal battery would be better), or with one of those public NTP servers have the ddwrt router be your NTP server for rest of your network.
I run my own server, and use time.is to check the deviation from atomic clock.
The routers with the 49000+ firmware didn't exhibit the same reboot after NTP was set. The routers running the older firmware did reboot outside their scheduled reboot cycle. All are running OSPF for my multiple VLAN's defined off 2 Cisco switches. Works great. All VLAN's are visible. So I need OSPF.
dmesg messages:
[ DD-WRT v3.0-r49139 std (06/10/22) - WRT3200 ]
# dmesg|grep -Ei "warn|error|fail|denied|invalid|inacc"
[ 0.000000] mvebu_mbus: [Firmware Warn]: deprecated mbus-mvebu Device Tree, suspend/resume will not work
[ 0.049412] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 0.049419] pci 0000:00:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 12.868015] ubi1 warning: ubi_eba_init: cannot reserve enough PEBs for bad PEB handling, reserved 32, need 40
[ 12.943064] UBIFS error (pid: 1087): cannot open "ubi1:ddwrt", error -19
[ 54.108999] mwifiex_sdio mmc0:0001:1: CMD_RESP: cmd 0x4d error, result=0x1
[ 54.116849] mwifiex_sdio mmc0:0001:1: set mac address failed: ret=-1
[ DD-WRT v3.0-r48646 std (04/12/22) - AC-RT1300 - Router 1 ] - AC RT1300 not AC RT88U
DD-WRT v3.0-r48646, DD-WRT v3.0-r47153 - Seems most stable for WiFi however the NTP setting to reach a remote NTP server did not work. So date was still wrong and both rebooted at different times. Date was probably wrong since host resolution didn't work so couldn't get an IP from the NTP server name. However, fixed this with a few F/W rules:
# ----------------------
# Google DNS
# ----------------------
iptables -A INPUT -s 8.8.8.8 -p tcp --sport 53 -j ACCEPT
iptables -A INPUT -s 8.8.4.4 -p tcp --sport 53 -j ACCEPT
iptables -A INPUT -s 8.8.8.8 -p udp --sport 53 -j ACCEPT
iptables -A INPUT -s 8.8.4.4 -p udp --sport 53 -j ACCEPT
OSPF seems to be running fine on all of the routers with the latest F/W.
By drivers, did you mean this line?
[ 11.050770] wl driver 7.14.164.18 (r692288) failed with code 1
Now date is ok. Resolves to ca.pool.ntp.org just fine. (facepalm). When switching to an external NTP pool from an internal one, forgot external ones may be blocked via iptables. Will see tomorrow or later today to see how things fared and if there's any unintended reboots.
Joined: 16 Nov 2015 Posts: 6414 Location: UK, London, just across the river..
Posted: Sun Jul 03, 2022 9:10 Post subject:
-update drivers as last builds contain security fixes..IMPORTANT 49392 is the current
-use numeric format (IP) for NTP time, best NTP time:
216.239.35.4 or 216.239.35.8 (google)
162.159.200.1 or 162.159.200.123 (cloudflare)
p.s. i rather to stay away from google DNS...@#$%^&*(
i use 9.9.9.9 instead...gives me a better sleep
also you can fairly use the SmartDNS encrypted DNS to improve security ... on the last build 49392 after you turn it on you need to add only few lines in this format to SmartDNS box....
Joined: 08 May 2018 Posts: 14129 Location: Texas, USA
Posted: Sun Jul 03, 2022 17:49 Post subject:
Any wl driver 7.14.164.18 (r692288) and dhd entries in syslog/klog mean proprietary binary driver blobs, usually. Pretty sure that it would likely have brcmfmac entries if it were experimental driver.
@tkmds and still, I hate to be a nag but no report from you whether you are using default driver firmware or experimental.
Ehhh ... though you're referring to the
wl driver 7.14.164.18
driver? Which driver and how do I tell the version?
@Alozaros ---
Tyvm! Had no idea 9.9.9.9 and 1.1.1.1 existed. Appreciate this!
"-update drivers as last builds contain security fixes..IMPORTANT 49392 is the current"
So we can update the wireless driver outside of the FW upgrade? How?
@kernel-panic69 ---
Ya, I moved my FW rules to a file on /jffs or /opt (In the case of WRT3200ACM router) to save on nvram. So many issues went away when I moved the rules to permanent storage. ( Btw, you've stumbled on my blog post. )
Need more time to go through the rest of the posts however. So far with the NTP update, all routers restarted when scheduled, and WiFi has been stable, aside from some above quirks I mentioned and notes on this post:
On the topic of nvam being exhausted, just ran a check for 'backtrace:' in the syslog files collected earlier. Back in April it was indeed caused due to the F/W rules clogging up the nvram it seems:
Joined: 31 Jul 2021 Posts: 2146 Location: All over YOUR webs
Posted: Mon Jul 04, 2022 8:21 Post subject:
tkmds wrote:
the-joker wrote:
@tkmds and still, I hate to be a nag but no report from you whether you are using default driver firmware or experimental.
Ehhh ... though you're referring to the
wl driver 7.14.164.18
driver? Which driver and how do I tell the version?
No, Im talking about the DD-WRT version you flashed.
See screenshot.
Anything in the Experimental folder will likely have issues. And you shouldnt switch between those builds without a nvram reset and re-configuring from scratch just in case.
The regular file will be better as those drivers are pretty much default broadcom and dhd driver has recently been updated.