Asus RT-AC5300 / Firmware: v3-r47040 std, max 64k NVRAM bug

Post new topic   Reply to topic    DD-WRT Forum Index -> Broadcom SoC based Hardware
Goto page Previous  1, 2
Author Message
Hapi12021
DD-WRT User


Joined: 22 Jul 2021
Posts: 84

PostPosted: Fri Jul 23, 2021 17:08    Post subject: Reply with quote
egc wrote:
Right could of course could have looked Smile
https://deviwiki.com/wiki/ASUS_RT-AC5300

Well I have a build with 128K NVRAM for you to test but as said chances of bricking are high at least you should be able to do serial recovery


I appreciate the trouble, but if bricking is the likely possibility, then I have to pass for the time being. I don’t have a serial connection for the device, at this time.

I think, even before we start testing experimental builds, we really should validate if the issue is common regardless of the build flags. If D_HAVE=128K really is the magic that needs to be included, then other platforms should demonstrate that such a define makes it all the way down into the sysinit code.

Either this is a one-off case, or the problem is systemic to how DD-WRT sysinit gets its sizing for NVRAM for routines like nvram_match().

ETA:

I also edited the OP / thread title to make clear this problem is for the RT-AC5300, and NOT the GT-AC5300, of which has the faster ARMv8 quad-core CPU, which might have been unclear at outset.
Sponsor
Hapi12021
DD-WRT User


Joined: 22 Jul 2021
Posts: 84

PostPosted: Fri Jul 23, 2021 22:27    Post subject: Reply with quote
I generated some test data, in specific sizes, for anyone that wishes to test their 128KiB platform for correct behavior.

Local storage, JFFS or the ability to SCP files back and forth to the router are requirements.

This (hopefully) won’t cause bricking, but don’t test if you fear this could happen or need your device to be operational during any test.


The general principle is that you investigate your current NVRAM usage:

# nvram show | grep bytes

This shows the amount of space used and how much is free.

Make a determination on which size data block you need, and download from below. The files are random text data created with:

# cat /dev/urandom | base64 | dd of=test_data_8k.txt bs=1024 count=8

Then to get the multiple sizes, the 8k block is concatenated to repeat until the desired size.


Save your configuration, if you have /jffs enabled:

# cd /jffs

# nvram backup ./backup_64k_test.bin

# nvram show | sort > ./before.nvram

If /jffs is not enabled:

# nvram backup /tmp/backup_64k_test.bin

# nvram show | sort > /tmp/before.nvram

Then SCP the files to your PC.


Next, create a new NVRAM variable, e.g. “test_data” and assign it the block of text corresponding to the size you require to go past 64KiB of NVRAM storage. If you need more than 32k, create another variable, e.g. “test_data_2” and add additional blocks to exceed 64KiB storage.

# nvram set test_data=“(use your terminal to paste the block, keep the quotes, no parens)”

# nvram commit

# nvram show | grep bytes (to check usage above 64KiB)

# reboot


When the system comes back up, create the comparison copy:

# cd /jffs; nvram show | sort > after.nvram

Or, if no /jffs

# cd /tmp; nvram show | sort > after.nvram


Now, transfer the two .nvram files to a PC and diff them:

# diff before.nvram after.nvram

Post your results. Redact any passwords or public data (IPs, accounts, etc.). Ideally (correctly), there should be no difference in the configurations, outside of time-series data or other ephemeral values.


Then, restore your prior configuration:

With /jffs:

# nvram unset test_data && nvram commit (just in case)

# cd /jffs; nvram erase && nvram restore ./backup_64k_test.bin

Without /jffs:

SCP the backup_64k_test.bin file to /tmp on the router:

# nvram unset test_data && nvram commit (just in case)

# cd /tmp; nvram erase && nvram restore ./backup_64k_test.bin


In both cases:

# nvram commit

# reboot


Thank you



test_data_32k.txt
 Description:

Download
 Filename:  test_data_32k.txt
 Filesize:  32 KB
 Downloaded:  172 Time(s)


test_data_16k.txt
 Description:

Download
 Filename:  test_data_16k.txt
 Filesize:  16 KB
 Downloaded:  128 Time(s)


test_data_8k.txt
 Description:

Download
 Filename:  test_data_8k.txt
 Filesize:  8 KB
 Downloaded:  124 Time(s)

BrainSlayer
Site Admin


Joined: 06 Jun 2006
Posts: 7463
Location: Dresden, Germany

PostPosted: Tue Jul 27, 2021 3:44    Post subject: Reply with quote
your whole test is bullshit. consider that on this platform there is a "nvram" and a "nvram_cfe" partition. the original cfe nvram keeps untouched. no matter what you're doing
_________________
"So you tried to use the computer and it started smoking? Sounds like a Mac to me.." - Louis Rossmann https://www.youtube.com/watch?v=eL_5YDRWqGE&t=60s
Hapi12021
DD-WRT User


Joined: 22 Jul 2021
Posts: 84

PostPosted: Tue Jul 27, 2021 7:29    Post subject: Reply with quote
BrainSlayer wrote:
your whole test is bullshit. consider that on this platform there is a "nvram" and a "nvram_cfe" partition. the original cfe nvram keeps untouched. no matter what you're doing


Bullshit or not, when I run that test against my system it exhibits the incorrect behavior specified.

Now, could there be something else missing? I’m not dedicated to my own thesis beyond any doubts. I’m interested in the details of finding a solution. If it helps others, even better! So, sure, the test case may not be perfect, or something else may be occurring. That’s why this discussion exists and isn’t all stuffed into the ticket.

As I said, if I can glean any debugging information that could be helpful, I will endeavor to try. Closing the ticket when there is a repeatable and demonstrable problematic behavior seems unhelpful towards fixing whatever is occurring, IMHO.

Should all 128KiB be usable? If so do you have any suggestions to figure out what’s happening. Partition map issues, maybe?
kernel-panic69
DD-WRT Guru


Joined: 08 May 2018
Posts: 14102
Location: Texas, USA

PostPosted: Tue Jul 27, 2021 12:34    Post subject: Reply with quote
The 128k nvram is split between two partitions on the flash. DD-WRT only takes into consideration one of those partitions because the other is not touched. That is what I read here. What is there to fix?
_________________
"Life is but a fleeting moment, a vapor that vanishes quickly; All is vanity"
Contribute To DD-WRT
Pogo - A minimal level of ability is expected and needed...
DD-WRT Releases 2023 (PolitePol)
DD-WRT Releases 2023 (RSS Everything)

----------------------
Linux User #377467 counter.li.org / linuxcounter.net
Hapi12021
DD-WRT User


Joined: 22 Jul 2021
Posts: 84

PostPosted: Tue Jul 27, 2021 17:21    Post subject: Reply with quote
kernel-panic69 wrote:
The 128k nvram is split between two partitions on the flash. DD-WRT only takes into consideration one of those partitions because the other is not touched. That is what I read here. What is there to fix?


Is it? I see no nvram_cfe partition. Did activating JFFS clobber it?

Code:

# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00080000 00020000 "boot"
mtd1: 00180000 00020000 "nvram"
mtd2: 01e00000 00020000 "linux"
mtd3: 01c80000 00020000 "rootfs"
mtd4: 06000000 00020000 "ddwrt"


That shows an “NVRAM” partition size of 1.5 MiB, which I dumped and is plaintext without line breaks, size 74KiB (~ 64KiB + header), followed by 1,462 KiB of value 0xFF.

Further, dumping /dev/mtd0 and checking for embedded NVRAM backups shows no data is contained there. OpenWRT docs suggest BCM47xx/CFE maintain a vendor NVRAM backup in the CFE. Does DD-WRT store board defaults inside the .trx, and inside the “Linux” partition, only?

Additionally, if the design goal of DD-WRT is to restrict the user to 64KiB of NVRAM in normal operation, why does the nvram system command have the ability to access 128KiB and allow storage into that area that can not be used upon subsequent boot? Is the “NVRAM” partition on the flash artificially limited to just the 64KiB for data storage? Because it doesn’t look split, only backfilled with null data to pad it out to the partition size. Or, is the padding the partitioning scheme and the solution just requires a larger “NVRAM” partition and adjustment to the offsets?

Something doesn’t seem right, and at least one of these behaviors appears to need correction, even if it’s just refusing to set NVRAM past 64KiB and emitting a warning to the user that the NVRAM is out of memory. That’s not my preferred solution, however it would prevent possible hair loss by admins attempting otherwise.
mj666
DD-WRT User


Joined: 26 Nov 2010
Posts: 58

PostPosted: Wed Jul 28, 2021 19:26    Post subject: Reply with quote
Something looks to be true on this. Since a while i missing the 2.4GHz WLAN interface on one of my RT-AC88U routers. I posted about this issues a couple of times in various build threads already. I have two of this devices and one is the gateway router and the second older one is now an access point. The old one was used before as the gateway and had the same issue as the new device now. The new device is showing the same behavior and the old one is now working without problems in the new configuration as the access point. The access point has an NVRAM config below 64K and the gateway was slightly above this. After cutting down the gateway NVRAM config also below 64k by removing older traffic information the 2.4GHz interface is showing up again immediately and functioning properly. I had the same problem with the standard and the experimental build. I'm actually running the experimental build since i also had random restarts with the standard build at the gateway router. Looks to be during boot not all of the NVRAM config is available. Dumping NVRAM into a backup later on always shows proper results and i can see the variables as expected.

Hapi12021 good find.

_________________
Asus AC-RT88U (Gateway, asus_rt-ac88u-firmware_52459.trx)
Asus AC-RT88U (AP, asus_rt-ac88u-firmware_52459.trx)
Asus AC-RT68U (AP, asus_rt-ac68u-firmware_52459.trx)
Asus AC-RT66U (AP, dd-wrt-52459-Asus_RT-AC66U.trx)
Linksys E3000 (AP, dd-wrt.v24-52459_NEWD-2_K3.x_mega-e3000.bin)
Linksys E4200 V2 (openwrt-18.06.5-kirkwood-linksys_viper-squashfs-sysupgrade.bin) not used at the moment
Hapi12021
DD-WRT User


Joined: 22 Jul 2021
Posts: 84

PostPosted: Wed Jul 28, 2021 22:04    Post subject: Reply with quote
mj666 wrote:
Something looks to be true on this. Since a while i missing the 2.4GHz WLAN interface on one of my RT-AC88U routers. I posted about this issues a couple of times in various build threads already. I have two of this devices and one is the gateway router and the second older one is now an access point. The old one was used before as the gateway and had the same issue as the new device now. The new device is showing the same behavior and the old one is now working without problems in the new configuration as the access point. The access point has an NVRAM config below 64K and the gateway was slightly above this. After cutting down the gateway NVRAM config also below 64k by removing older traffic information the 2.4GHz interface is showing up again immediately and functioning properly. I had the same problem with the standard and the experimental build. I'm actually running the experimental build since i also had random restarts with the standard build at the gateway router. Looks to be during boot not all of the NVRAM config is available. Dumping NVRAM into a backup later on always shows proper results and i can see the variables as expected.

Hapi12021 good find.


Thank you for the corroboration.

One other thing to point out, from my research into this issue:

Namely, that the size of the contents of the NVRAM partition is a throwback to when NOR flash was used specifically for storing NVRAM data for the system.

The CFE boot loader defines a certain amount of memory to read in the contents of NVRAM for its own use, and that was originally tied to the size of the NOR flash on the system.

As newer systems only contain a linear NAND flash, the original behavior is emulated, only for the boot loader’s benefit, otherwise the size could be completely arbitrary.

If the CFE is defining 128KiB, then that’s the size of the static structure crated to hold the data upon CFE initialization. After the CFE boots, and hands off the system to the Linux Kernel, it’s up to the running system to manage the data on its own. Any NVRAM backup in the CFE is for default restoration via the CFE when performing a 30-30-30, or similar, reset, which apparently DD-WRT handles on its own, restoring defaults only after the kernel is running? Or, is the 10KiB header in the NVRAM partition the default values for configuration reset? Still a few details are missing.

Maybe old news to most of you, but enlightens me as to why the need to keep NVRAM storage so small.
mj666
DD-WRT User


Joined: 26 Nov 2010
Posts: 58

PostPosted: Thu Jul 29, 2021 5:23    Post subject: Reply with quote
I have now upgraded my both RT-AC88U devices to build 47090 and using the standard build again. I cleaned the NVRAM config and kept this below 64k (old traffic information still removed). For now the WLAN interfaces on both devices are functioning properly. Will see if the random reboots will also stop for the gateway router. During runtime of DDWRT itself I cannot see any problem with the contents of the NVRAM. All variables are present as expected. I know since I do automatic weekly configuration backups since quite some time and will be able to compare them and see the differences. Anyhow during boot it looks to be DD-WRT is not be able to see the complete set of variables if the NVRAM configuration is larger than 64k and is not able to initialize all WLAN interfaces
_________________
Asus AC-RT88U (Gateway, asus_rt-ac88u-firmware_52459.trx)
Asus AC-RT88U (AP, asus_rt-ac88u-firmware_52459.trx)
Asus AC-RT68U (AP, asus_rt-ac68u-firmware_52459.trx)
Asus AC-RT66U (AP, dd-wrt-52459-Asus_RT-AC66U.trx)
Linksys E3000 (AP, dd-wrt.v24-52459_NEWD-2_K3.x_mega-e3000.bin)
Linksys E4200 V2 (openwrt-18.06.5-kirkwood-linksys_viper-squashfs-sysupgrade.bin) not used at the moment
Hapi12021
DD-WRT User


Joined: 22 Jul 2021
Posts: 84

PostPosted: Sun Aug 01, 2021 0:39    Post subject: Reply with quote
I’ve now moved to Entware / Stunnel on the RT-AC5300, and reconfigured my firewall rules to run as a script initiated from the firewall command tab in administration.

I ended up dual-mounting the /jffs partition onto /opt as well, and installing Entware there. A little investigation with lsof and find had me symlink all writable directories used for temporary data back onto /tmp, so there are no files open for write on the flash. Total space used was under 10 MiB, and the partition size is over 96 MiB, so ~ 11% utilization.

Final savings: 18KiB removed from NVRAM, and will be further investigating purging empty defines from the config, at boot.

After reviewing the contents and my changes to NVRAM from defaults, it’s still a little disconcerting that there is so much used on a new installation when set to factory settings: nearly 20KiB, most of which is radio data for regions other than mine.

Seems like another opportunity to optimize.

I probably won’t bump this thread any longer, as I know my system limitations and the message has been conveyed on what needs to be fixed.
Goto page Previous  1, 2 Display posts from previous:    Page 2 of 2
Post new topic   Reply to topic    DD-WRT Forum Index -> Broadcom SoC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum