Posted: Thu Oct 06, 2022 20:54 Post subject: [TESTING] 3xR7800 upgrade from r41813 to r50012 - issues
Hey,
Three R7800 are connected together using WG at different ISPs. I have upgraded firmware and encountered issues below:
1) Random router reboot in approx. 10-20 days.
2) If I access router via SSH and check commands like `ip`, `wg`, it reboots after 5-30 minutes. So it can reboot multiple times in a day. I'm mostly using OpenSSH. I've switched to default Dropbear to see how it goes.
3) WiFi disconnects depending on client computer, every 30-60 minutes 2.4 and 5GHz. Eg two WiFi clients work fine all the time, one has this issue.
4) When I connected additional WG clients (they are laptops), I've noticed WG disconnections "latest handshake" for around 5-10 minutes every few hours. Disconnection happens between R7800 units, not clients such as Macs. So I removed those computers from WG tunnels and it fixed an issue.
P.S. I did nvram erase && reboot
None of those issues I had on r41813, R7800 used to reach 1 year+ uptime under UPSes. So I'm sure these are software issues.
I would like to use latest firmware and happy to spend time on debugging, if any hints offered.
Many thanks
Last edited by LaimisV on Mon Oct 17, 2022 18:34; edited 1 time in total
nvram set wlan0_wl_advanced=1
nvram set wlan0_regdomain=LITHUANIA
nvram set wlan0_mode=ap
nvram set wlan0_net_mode=mixed
nvram set wlan0_channelbw=20
nvram set wlan0_ssid=XXXX-5G
nvram set wlan0_channel=0
nvram set wlan0_closed=0
nvram set wlan0_bridged=1
nvram set wlan1_wl_advanced=1
nvram set wlan1_regdomain=LITHUANIA
nvram set wlan1_mode=ap
nvram set wlan1_net_mode=mixed
nvram set wlan1_channelbw=20
nvram set wlan1_ssid=XXXX-2G
nvram set wlan1_channel=0
nvram set wlan1_closed=0
nvram set wlan1_bridged=1
nvram set wlan0_security_mode=psk2
nvram set wlan0_psk2=1
nvram set wlan0_ccmp=1
nvram set wlan0_wpa_psk="XXXX"
nvram set wlan1_security_mode=psk2
nvram set wlan1_psk2=1
nvram set wlan1_ccmp=1
nvram set wlan1_wpa_psk="XXXX"
The rest is r50012 default.
I used browser's inspect on GUI to find right parameter names. They should be fine, they are double checked both ways.
BTW, I disabled WiFi on DD-3 unit, using `ifconfig wlan0 down; ifconfig wlan1 down`, it has 22 days uptime now. It doesn't need WiFi.
DD-2 has 20 days of uptime, it is testing additional settings:
Code:
nvram set wlan0_channel=5825
nvram set wlan0_wl_advanced=1
nvram set wlan0_fwtype=vanilla
nvram set wlan1_channel=2462
nvram set wlan1_wl_advanced=1
nvram set wlan1_fwtype=vanilla
Last edited by LaimisV on Thu Oct 06, 2022 22:58; edited 1 time in total
There is more stuff on OpenSSH, but it's irrelevant for crashes, I believe. In short, sshd and sudo binaries are used. Additional non root user added just to SSH to it and switch to root.
Last edited by LaimisV on Thu Oct 06, 2022 22:39; edited 3 times in total
WG on DD-1 (equivalent settings on DD-2 and DD-3):
Code:
# wireguard-tools v1.0.20210424
nvram set oet1_en=1
nvram set oet1_peers=2
nvram set oet1_private=XXXX
nvram set oet1_public=XXXX
nvram set oet1_proto=2
nvram set oet1_mit=1
nvram set oet1_natout=0
nvram set oet1_port=51820
nvram set oet1_mtu=1440
nvram set oet1_pbr=""
nvram set oet1_firewallin=0
nvram set oet1_killswitch=0
nvram set oet1_ipaddr="10.1.250.1"
nvram set oet1_netmask=255.255.255.0
nvram set oet_tunnels=1
nvram set oet1_psk0=""
nvram set oet1_bridged=1
nvram set oet1_id=1
nvram set oet1_nat=0
nvram set oet1_namep0="to-240"
nvram set oet1_endpoint0=1
nvram set oet1_peerkey0=XXXX
nvram set oet1_peerport0=51820
nvram set oet1_aip_rten0=1
nvram set oet1_usepsk0=0
nvram set oet1_ip0=10.1.240.1
nvram set oet1_rem0=X.X.X.X
nvram set oet1_aip0="10.1.240.1/32,192.168.240.0/24"
nvram set oet1_ka0=20
nvram set oet1_namep1="to-230"
nvram set oet1_endpoint1=1
nvram set oet1_peerkey1=XXXX
nvram set oet1_peerport1=51820
nvram set oet1_aip_rten1=1
nvram set oet1_usepsk1=0
nvram set oet1_ip1=10.1.230.1
nvram set oet1_rem1=X.X.X.X
nvram set oet1_aip1="10.1.230.1/32,192.168.230.0/24"
nvram set oet1_ka1=20
I can create a script with some stats or enable klogd to remote server. Would it be useful?
Let me know what to include. I'm comfortable with Linux debugging and scripting.
I bet, I have less than 4 issues (maybe 1 or 2 in total). I guess, those 4 issues that I initial listed are somehow connected together.
wlan0 is on the R7800 5Ghz and has e.g. no Turbo-QAM option.
What is the nonsense with the 100 variables actually?
Should someone here make the effort and work through all variables and uncover errors like "wlan0_security_mode=psk2" <- no valid securiry mode
Argh I posted too soon. Fixed channel examples & turboqam to match wlan0 & wlan1 for R7800.
Yes wlan0_security_mode=psk2 and wlan1_security_mode=psk2 are wrong, they should = wpa.
To disable wlan0 wlan1 use Wireless Network Mode (renamed Network Mode) set each Disabled.
And yes it was some effort go through all these. I really should have said just use web interface.
Also after a reset, use egc's documentation & advice a lot of work was done to avoid commands.
I managed to use DD-2 WiFi without disconnects for 7 hours with this addition:
Code:
nvram set wlan0_channel=5825
nvram set wlan0_wl_advanced=1
nvram set wlan0_fwtype=vanilla
nvram set wlan1_channel=2462
nvram set wlan1_wl_advanced=1
nvram set wlan1_fwtype=vanilla
Previously, it was disconnecting every hour. Need more testing to confirm, but so far working longer. Will use additional settings that you suggested, if any disconnect.
----
I have enabled remote logging on DD-1, DD-2 and DD-3 (syslogd -Z -L -R 192.168.250.110). I receive test messages with:
Code:
echo MESSAGE1 > /dev/kmsg
----
BTW, I'm using new firmware with nvram erase && reboot on all three routers. So nvram is in good condition.
----
If it's better, please post a screenshot or parameters in GUI. This is just me (a DevOps guy) who prefers everything in command line: perfectly tracked and easily automated when needed (:
root@dd-2:~# cat /var/log/messages | grep -vE '(\.info|\.debug)' | grep -v auth.err
Oct 7 11:26:35 dd-2 kern.warn kernel: [ 30.363366] ath10k_pci 0000:01:00.0: wmi debug print truncated: 128
...line repeated...
Oct 7 11:26:35 dd-2 kern.notice kernel: [ 68.783981] SCSI subsystem initialized
Oct 7 11:26:35 dd-2 kern.warn kernel: [1796990.049949] ath10k_pci 0000:01:00.0: could not get mac80211 beacon
Oct 7 11:26:35 dd-2 kern.warn kernel: [1796990.149702] ath10k_pci 0000:01:00.0: peer-unmap-event: unknown peer id 0
# line repeated ...
Oct 7 11:26:35 dd-2 kern.warn kernel: [1796990.316020] ath10k_pci 0001:01:00.0: peer-unmap-event: unknown peer id 0
Oct 7 11:26:35 dd-2 kern.warn kernel: [1796996.295131] ath10k_pci 0000:01:00.0: wmi debug print truncated: 128
# line repeated ...
Oct 7 11:26:35 dd-2 kern.warn kernel: [1797003.424785] ath10k_pci 0000:01:00.0: Direct firmware load for ath10k/QCA9984/hw1.0-vanilla/firmware-6.bin failed with error -2
Oct 7 11:26:35 dd-2 kern.warn kernel: [1797012.978948] ath10k_pci 0000:01:00.0: wmi command 36872 timeout, restarting hardware
Oct 7 11:26:35 dd-2 kern.warn kernel: [1797012.979052] ath10k_pci 0000:01:00.0: ani_enable failed from debugfs: -11
Oct 7 11:26:35 dd-2 kern.warn kernel: [1797013.008675] ath10k_pci 0000:01:00.0: cannot restart a device that hasn't been started
# lines repeated ...
all irrelevant, are only debug messages
LaimisV wrote:
Code:
Oct 7 11:26:39 dd-2 daemon.warn dnsmasq[945]: ignoring nameserver 192.168.240.240 - local interface
Oct 7 11:26:39 dd-2 authpriv.warn dropbear[2897]: Failed listening on '22': Error listening: Address in use
Oct 7 11:27:13 dd-2 user.warn kernel: [1831652.657097] MESSAGE
# line repeated ...
also irrelevant, since you configured it yourself.
Port 22 will be already in use because you still have openssh running in parallel.
and that with dnsmasq will be a configuration error, there is as upstream server dnsmasq itself configured so an infinite loop.
And again please post your settings as screenshot or don't do it.
I do not read coffee grounds here
Edit: oh and the build r50012 has bugs, e.g. when changing the configuration dnsmasq is not restarted correctly
As DNSmasq is responsible for DNS and DHCP, I don't use them over WG. Except an attempt to connect few Macbooks using "Peer Tunnel DNS" at "1.1.1.1", if that is related.
So I believe, there are no direct relations to issue reported by Tatsuya.
