Asus RT-AC5300 / Firmware: v3-r47040 std, max 64k NVRAM bug

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware
Goto page 1, 2  Next
Author Message
Hapi12021
DD-WRT Novice


Joined: 22 Jul 2021
Posts: 22

PostPosted: Thu Jul 22, 2021 4:34    Post subject: Asus RT-AC5300 / Firmware: v3-r47040 std, max 64k NVRAM bug Reply with quote
This bug has been around in all the builds I have tested for the RT-AC5300:

While the system can see and utilize all 128KB of NVRAM on this router, the initial configuration is limited to 64KB upon boot.

This has the odd effect of causing the 1st radio, which is 2.4GHz, to disappear and the initial hardware probe fail for all three radios.

I have a configuration that is sane, and then one where I added a few extra iptables rules to the firewall, and upon next boot eth0 was replaced by eth1, eth1 by eth2, etc.

Dropping the NVRAM back below 64KB allows the probe to complete and all radios are present.

This is not a showstopper bug for me, as I can monitor NVRAM usage and be more thrifty in my pragmas, but it should be fixed allowing the platform to use all 128KB.

Reading through past posts seems that there is a 64KB hard-coded limit for the in-memory NVRAM as default.


Last edited by Hapi12021 on Fri Jul 23, 2021 17:18; edited 2 times in total
Sponsor
egc
DD-WRT Guru


Joined: 18 Mar 2014
Posts: 8130
Location: Netherlands

PostPosted: Thu Jul 22, 2021 10:20    Post subject: Reply with quote
You must have some interesting setup that it goes over 64 KB NVRAM Smile

A trick to clear nvram from unused items is adding this to the startup:
Code:
for line in `nvram show | grep =$ `; do var=${line%*=}; nvram unset $var; done
nvram commit


Some dual boot routers divide the nvram, I do not think that this is the case here, if not
consider filing a bug report

_________________
Routers:Netgear R7800, R6400v1, R6400v2, Linksys EA8500, EA6900 (XvortexCFE), E2000 (converted WRT320N), WRT54GS v1.
WireGuard Documents & Guides:https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327397
OpenVPN Documents & Guides: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327398
IPSET: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327261
Install guide R6400v2:http://forum.dd-wrt.com/phpBB2/viewtopic.php?t=316399
Install guide R7800: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=320614
Forum Guide Lines (important read):https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=324087
Hapi12021
DD-WRT Novice


Joined: 22 Jul 2021
Posts: 22

PostPosted: Thu Jul 22, 2021 14:14    Post subject: Reply with quote
Not so interesting, however, there is the fact that I use a real wildcard certificate on the device, which itself is 12KB for the key and chain.

I looked through the code for httpd.c in dd-wrt, as I couldn’t modify /etc/cert.pem and /etc/key.pem, and /jffs/etc overlays didn’t work either. I created /tmp/https_cert and /tmp/https_key, but the server still didn’t read them. Looking at the code, the only avenue to putting the cert into httpd, for the GUI, was to insert it into NVRAM.

Now, I can probably get stunnel via entware and have it proxy the https tunnel with the certificate on /jffs, or as you say, cull the unused and empty NVRAM variables, as I did on an RT-N16 (by killing the second OpenVPN server config, as it was never going to be used). But, this workaround needs to be done at each boot, and with a scenario like on the 5300, there exists the possibility of NVRAM corruption due to truncation.

So, several bugs that led to this situation, but the most pressing is that the silent misbehavior of the system with regards to configuration is remedied.
Hapi12021
DD-WRT Novice


Joined: 22 Jul 2021
Posts: 22

PostPosted: Thu Jul 22, 2021 14:48    Post subject: Reply with quote
SVN ticket #7447 created:

https://svn.dd-wrt.com/ticket/7447
Hapi12021
DD-WRT Novice


Joined: 22 Jul 2021
Posts: 22

PostPosted: Thu Jul 22, 2021 21:07    Post subject: Reply with quote
Much traipsing through SVN has led me to:

https://svn.dd-wrt.com/browser/src/include.v24/include/bcmnvram.h

Quote:
extern int BCMINIT(nvram_getall)(char *buf, int count);
130
131 #endif /* _LANGUAGE_ASSEMBLY */
132
133 #define NVRAM_MAGIC 0x48534C46 /* 'FLSH' */
134 #define NVRAM_VERSION 1
135 #define NVRAM_HEADER_SIZE 20
136 #if defined(HAVE_XSCALE) || defined(HAVE_X86) || defined(HAVE_FONERA) || defined(HAVE_WHRAG108)
137 #define NVRAM_SPACE 0x10000
138 #else
139 #define NVRAM_SPACE 0x8000
140 #endif
141
142 #define NVRAM_MAX_VALUE_LEN 255
143 #define NVRAM_MAX_PARAM_LEN 64


It would appear that even the current system init service still references the K2.4 include for <bcmnvram.h>, and therefore still is based before 2012, with no router models supporting 128KiB of NVRAM.

However, this value is not likely current, although files including this file probably need to be verified for flash sizing.

Next, we see:

https://svn.dd-wrt.com/browser/src/include.bcm/flash.h

I’m not going to detail the whole file, but the largest flash allocation in that header is 8Kx8, so another limit that should be checked.

As to why the full 128KiB is available, once booted: the later routines get their size from the chip ID directly. The system initialization NVRAM routines need to get their low-level chip information from newer sources.

Maintainers, please address.

I’m not going to put this in the ticket, just yet, because it’s still speculative in nature.


Last edited by Hapi12021 on Thu Jul 22, 2021 21:10; edited 1 time in total
kernel-panic69
DD-WRT Guru


Joined: 08 May 2018
Posts: 10001
Location: Texas, USA

PostPosted: Thu Jul 22, 2021 21:10    Post subject: Reply with quote
I see the gotchas gotcha. There are pitfalls if you do not understand the layout on the public svn, and I can guarantee you that is not the files you should be looking at, most likely.
_________________
Official Forum Rules, Guidelines & Helpful InformationFirmware FAQInstallation WikiWhere Do I Download Firmware?
DON'T use Chromium-based browsersRTFM/STFW TL;DR is NOT an excuse. • Why Should I Care What Color the Bikeshed Is?
Please DO NOT PM me with questions; Ask in the forum. ---------------------- Linux User #377467 counter.li.org / linuxcounter.net
Hapi12021
DD-WRT Novice


Joined: 22 Jul 2021
Posts: 22

PostPosted: Thu Jul 22, 2021 21:18    Post subject: Reply with quote
kernel-panic69 wrote:
I see the gotchas gotcha. There are pitfalls if you do not understand the layout on the public svn, and I can guarantee you that is not the files you should be looking at, most likely.


And that’s cool, I haven’t tried to build the whole system, so not familiar with the layout. I have built older versions of Tomato, however, so not entirely clueless on how these values get overridden at different points.

I’m still working my way through the code.

Do you know which module sets the NVRAM value probe_blacklist? That’s seems to be around where the trouble starts.

But, in general, since 128KiB flashes are still not that common, there probably exist a fair amount of #defines for older flash chips that need to have a second look, and the boot code may be misidentifying the AC5300’s size, or still think the largest flash is 64KiB.

64K ‘ought to be enough for anyone, right?
egc
DD-WRT Guru


Joined: 18 Mar 2014
Posts: 8130
Location: Netherlands

PostPosted: Thu Jul 22, 2021 21:24    Post subject: Reply with quote
A lot of routers have 128k nvram I have a Netgear R6400 which is broadcom and a Netgear R7800 which is atheros both having 128k
_________________
Routers:Netgear R7800, R6400v1, R6400v2, Linksys EA8500, EA6900 (XvortexCFE), E2000 (converted WRT320N), WRT54GS v1.
WireGuard Documents & Guides:https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327397
OpenVPN Documents & Guides: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327398
IPSET: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327261
Install guide R6400v2:http://forum.dd-wrt.com/phpBB2/viewtopic.php?t=316399
Install guide R7800: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=320614
Forum Guide Lines (important read):https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=324087
Hapi12021
DD-WRT Novice


Joined: 22 Jul 2021
Posts: 22

PostPosted: Thu Jul 22, 2021 21:30    Post subject: Reply with quote
egc wrote:
A lot of routers have 128k nvram I have a Netgear R6400 which is broadcom and a Netgear R7800 which is atheros both having 128k


Great!

Would you be up for an experiment?

Assuming they run DD-WRT, inject an NVRAM variable that’s ~ 16-32K worth of random data, and ensuring that more than 64K total is used commit and reboot to verify if booting over 64K works on other platforms?

Ideally, dump the contents of NVRAM to a file before and after, and the only variable that should diff out would be the random data.
egc
DD-WRT Guru


Joined: 18 Mar 2014
Posts: 8130
Location: Netherlands

PostPosted: Thu Jul 22, 2021 21:43    Post subject: Reply with quote
I do not have time to do that at the moment but specifying the amount of nvram is fairly simple, when I build for e.g. an R7000 which has 64k I have to specify that in the config file.
For the R6400 I specify 128k then building is done with DHAVE_128K.

_________________
Routers:Netgear R7800, R6400v1, R6400v2, Linksys EA8500, EA6900 (XvortexCFE), E2000 (converted WRT320N), WRT54GS v1.
WireGuard Documents & Guides:https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327397
OpenVPN Documents & Guides: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327398
IPSET: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327261
Install guide R6400v2:http://forum.dd-wrt.com/phpBB2/viewtopic.php?t=316399
Install guide R7800: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=320614
Forum Guide Lines (important read):https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=324087
Hapi12021
DD-WRT Novice


Joined: 22 Jul 2021
Posts: 22

PostPosted: Thu Jul 22, 2021 22:23    Post subject: Reply with quote
egc wrote:
I do not have time to do that at the moment but specifying the amount of nvram is fairly simple, when I build for e.g. an R7000 which has 64k I have to specify that in the config file.
For the R6400 I specify 128k then building is done with DHAVE_128K.


I’m happy to provide any extra information I can. My browse through dmesg and messages yielded no significant debugging data. It was just if wl0 was unplugged from the system. No error messages at all. Likely it was never probed and never had the driver attached.

My own diff (good : bad):

Code:

1677,1678c1689,1690
< probe_blacklist=vlan1 vlan2 eth0
< probe_working=eth3 eth1 eth2 wl1.1
---
> probe_blacklist=vlan1 vlan2
> probe_working=eth1 eth2 wl1.1
1822c1834
< start_time=1626924457
---
> start_time=1626924777
1871c1883
< traff-07-2021=0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 175:3 14052:652 17468:191 3079:548 5006:323 19404:281 1113:44 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 [60297:2042]
---
> traff-07-2021=0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 175:3 14052:652 17468:191 3079:548 5006:323 19404:281 1244:49 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 0:0 [60428:2047]
2118c2130
< wl0_bandlist=b
---
> wl0_bandlist=a
2148c2160
< wl0_hwaddr=F2:2F:74:FA:4B:B3
---
> wl0_hwaddr=F0:2F:74:FA:4B:B4
2333c2345
< wl1.1_hwaddr=F2:2F:74:FA:4B:B5
---
> wl1.1_hwaddr=F2:2F:74:FA:4B:C9
2396c2408
< wl1_hwaddr=F0:2F:74:FA:4B:B4
---
> wl1_hwaddr=F0:2F:74:FA:4B:C8


Currently, on the sane config, with only about 5 iptables lines of difference between the good and bad configs (~ 480 bytes), I see:

2668 entries and size: 65099 bytes (65973 left)
egc
DD-WRT Guru


Joined: 18 Mar 2014
Posts: 8130
Location: Netherlands

PostPosted: Fri Jul 23, 2021 12:58    Post subject: Reply with quote
You really need to hear from the head honcho how he is building.

We have a router matrix but that is rather old and suggests builds are made with 64K NVRAM

I have made a build for your Asus RT5300 with 128K NVRAM specified but the chances that it will permanently brick your router I would estimate at about 80%

On the bright side I did not think it would build at all as the CPU is really different (Quadcore arm A53 at 1.8 GHz ?) so I actually thought I needed an Armv8 toolchain, but then again maybe I do and the build will brick your router Sad
(I know/think that Armv8 should be backwards compatible with Armv7)

_________________
Routers:Netgear R7800, R6400v1, R6400v2, Linksys EA8500, EA6900 (XvortexCFE), E2000 (converted WRT320N), WRT54GS v1.
WireGuard Documents & Guides:https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327397
OpenVPN Documents & Guides: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327398
IPSET: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327261
Install guide R6400v2:http://forum.dd-wrt.com/phpBB2/viewtopic.php?t=316399
Install guide R7800: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=320614
Forum Guide Lines (important read):https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=324087
kernel-panic69
DD-WRT Guru


Joined: 08 May 2018
Posts: 10001
Location: Texas, USA

PostPosted: Fri Jul 23, 2021 14:32    Post subject: Reply with quote
I don't think a CPU version-specific toolchain is required:

https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm/downloads

https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads

*BUT* (and there is always a 'BUT') looking at OpenWRT's toolchains for ARM, they seem to have some specific toolchains used it seems?

I couldn't find anything conclusive in my buildroot notes, though. Personally, I think over-complicating and obfuscating the heck out of things is pointless.

_________________
Official Forum Rules, Guidelines & Helpful InformationFirmware FAQInstallation WikiWhere Do I Download Firmware?
DON'T use Chromium-based browsersRTFM/STFW TL;DR is NOT an excuse. • Why Should I Care What Color the Bikeshed Is?
Please DO NOT PM me with questions; Ask in the forum. ---------------------- Linux User #377467 counter.li.org / linuxcounter.net
Hapi12021
DD-WRT Novice


Joined: 22 Jul 2021
Posts: 22

PostPosted: Fri Jul 23, 2021 16:29    Post subject: Reply with quote
egc wrote:
You really need to hear from the head honcho how he is building.

We have a router matrix but that is rather old and suggests builds are made with 64K NVRAM

I have made a build for your Asus RT5300 with 128K NVRAM specified but the chances that it will permanently brick your router I would estimate at about 80%

On the bright side I did not think it would build at all as the CPU is really different (Quadcore arm A53 at 1.8 GHz ?) so I actually thought I needed an Armv8 toolchain, but then again maybe I do and the build will brick your router Sad
(I know/think that Armv8 should be backwards compatible with Armv7)


The RT-AC5300 is dual-core BCM4709C0 ARMv7 at 1.4GHz. It’s Northstar and not all that different than the RT-AC3200, but with a slightly uprev’d BCM4366 radio chipset for doing MU-MIMO @ 4:4. Physically, it’s a giant box with 8 antennas, but internally it’s pretty familiar territory.


Last edited by Hapi12021 on Fri Jul 23, 2021 17:23; edited 2 times in total
egc
DD-WRT Guru


Joined: 18 Mar 2014
Posts: 8130
Location: Netherlands

PostPosted: Fri Jul 23, 2021 16:54    Post subject: Reply with quote
Right could of course could have looked Smile
https://deviwiki.com/wiki/ASUS_RT-AC5300

Well I have a build with 128K NVRAM for you to test but as said chances of bricking are high at least you should be able to do serial recovery

_________________
Routers:Netgear R7800, R6400v1, R6400v2, Linksys EA8500, EA6900 (XvortexCFE), E2000 (converted WRT320N), WRT54GS v1.
WireGuard Documents & Guides:https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327397
OpenVPN Documents & Guides: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327398
IPSET: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327261
Install guide R6400v2:http://forum.dd-wrt.com/phpBB2/viewtopic.php?t=316399
Install guide R7800: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=320614
Forum Guide Lines (important read):https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=324087
Goto page 1, 2  Next Display posts from previous:    Page 1 of 2
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum