Posted: Thu Jan 09, 2025 17:12 Post subject: R6220 degrading behavior across different versions
BLUF: Problems with firmware started sometime after r50308. By r50963, the boot process was halting. By r51154, it was bootlooping. AFAIK all subsequent versions have been bootlooping. Also, unable to successfully flash any version via DD-WRT web UI. Logs attached here and in first two replies.
Longer version:
I am continuing to try to gather data that might be helpful to troubleshoot problems with DD-WRT on the Netgear R6220. If there are any specific things a developer would like me to try/log, I am willing to do so.
Note: I have never been able to successfully flash a "webflash" firmware file via the DD-WRT WebUI, even for versions where I was able to successfully load the "factory-to-ddwrt" version via nmrpflash. The flash appears to go well, but the R6220 bootloops afterward. All of the data collected below was gathered by loading "factory-to-ddwrt" firmware files via nmrpflash.
Observations (serial logs from those sessions are attached):
r50308: This is the most current working version I'm aware of. (I am still in the process of trying to find the "last working version" before things started to break, so there may be newer versions that still work.)
r50963: Boot process halts around 2 minutes after powerup with serial log entry: "[httpd] : Error on startup, returncode 127 "
r51154: The is the first version I'm aware of that demonstrates the "bootlooping" behavior that has been present in (as far as I can tell) all subsequent versions. I am still trying to find the first bootlooping version.
I will continue to try to find the last working version and first bootlooping version. In the meantime, I have uploaded serial logs capturing the firmware install process and the boot process for the three versions described above. (Due to attachment limits see replies 1 and 2 for additional files.)
R6220_r50308_load_webui_bootloop.txt
Description:
Serial log capturing r50308 "webflash" file being loaded via DD-WRT web UI, then rebooting into a bootloop.
Posted: Thu Jan 09, 2025 17:42 Post subject: additional attachments
See OP for details re: these attachments
R6220_r51154_bootloop.txt
Description:
Serial log of r51154 bootloop behavior. (This same log has been attached on other posts in this forum using the filename "R6220_bootloop_r51154.txt". It is reuploaded here for completeness.
Hah. Well, I don't know how likely it is that this problem will get fixed, but if it doesn't, I don't want it to be because I didn't provide the data needed to track down the bug.
But I also don't want to bury useful data with a lot of noise, so this will probably be the last set of logs I post unless someone specifically asks me to do something else.
I think I've identified where the firmware first starts to bootloop:
r50786 seems to be the last version that boots into a stable config.
r50814 was the first version that bootloops.
I had previously reported that r50963 has a different problem--it halts ~2 minutes after startup. That is still true. But all the other versions I've tried from 50814 onward bootloop: r50814, r51154, r57447, r57595, r59876, r59045.
Serial logs capturing the firmware load and subsequent boot for r50786 (good) and r50814 (bootloop) are attached.
Please let me know if there's anything else I can do to help.
R6220_r50814_load_and_bootloop.txt
Description:
Serial log of firmware install and subsequent boot sequence for first R6220 firmware version exhibiting bootloop behavior (r50814).
Also, I don't know if it's just a coincidence or a clue, but I want to reiterate that *all* the firmware versions I've tried to install via the Firmware Update page under the DD-WRT Admin tab have been unsuccessful--they flash successfully and boot, but they exhibit similar bootloop behavior (kernel panic a few seconds into the startup sequence).
Both issues seem to fail at the same point, with the same error in the log file. Example:
r50308 (works via nmrpflash, but fails this way when flashed via DD-WRT web UI):
Code:
[ 5.707864] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[ 5.722774] Please append a correct "root=" boot option; here are the available partitions:
[ 5.739410] 1f00 1024 mtdblock0
[ 5.739415] (driver?)
[ 5.752421] 1f01 1024 mtdblock1
[ 5.752427] (driver?)
[ 5.765428] 1f02 32768 mtdblock2
[ 5.765432] (driver?)
[ 5.778436] 1f03 28672 mtdblock3
[ 5.778441] (driver?)
[ 5.791466] 1f04 2048 mtdblock4
[ 5.791472] (driver?)
[ 5.804472] 1f05 1024 mtdblock5
[ 5.804477] (driver?)
[ 5.817480] 1f06 80896 mtdblock6
[ 5.817485] (driver?)
[ 5.830502] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 5.846962] Rebooting in 1 seconds..
r50814 (fails this way when flashed via nmrpflash)
Code:
[ 5.731114] hctosys: unable to open rtc device (rtc0)
[ 5.751510] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[ 5.766410] Please append a correct "root=" boot option; here are the available partitions:
[ 5.783045] 1f00 1024 mtdblock0
[ 5.783051] (driver?)
[ 5.796059] 1f01 1024 mtdblock1
[ 5.796064] (driver?)
[ 5.809063] 1f02 32768 mtdblock2
[ 5.809068] (driver?)
[ 5.822088] 1f03 28672 mtdblock3
[ 5.822093] (driver?)
[ 5.835102] 1f04 2048 mtdblock4
[ 5.835107] (driver?)
[ 5.848109] 1f05 1024 mtdblock5
[ 5.848114] (driver?)
[ 5.861118] 1f06 80896 mtdblock6
[ 5.861124] (driver?)
[ 5.874141] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 5.890603] Rebooting in 1 seconds..
Those two log snippets look identical to me. Not sure if that's significant or not, though.
This is how the logs compare between the last-good version (r50786) and the first bootloop version (r50814). Note the differences in the sequence between when NVRAM size is checked and when the VFS attempts to mount root:
Good version (r50786):
Code:
[ 5.119529] searching for nvram
[ 5.125867] nvram size = 2097152
[ 5.703057] found nvram at 1E0000
[ 5.731103] auto-attach mtd3
[ 5.731134] ubi0: attaching mtd3
[ 5.968843] UBI: EOF marker found, PEBs from 197 will be erased
[ 5.968889] ubi0: scanning is finished
[ 5.995024] ubi0 warning: ubi_read_volume_table: volume table copy #1 is corrupted
[ 6.021406] ubi0: volume table was restored
[ 6.029970] ubi0 warning: ubi_eba_init: cannot reserve enough PEBs for bad PEB handling, reserved 14, need 19
[ 6.073204] ubi0: volume 1 ("rootfs_data") re-sized from 9 to 9 LEBs
[ 6.086718] ubi0: attached mtd3 (name "ubi", size 28 MiB)
[ 6.097509] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
[ 6.111197] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[ 6.124713] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
[ 6.138570] ubi0: good PEBs: 223, bad PEBs: 1, corrupted PEBs: 0
[ 6.150532] ubi0: user volume: 2, internal volumes: 1, max. volumes count: 128
[ 6.164914] ubi0: max/mean erase counter: 1/0, WL threshold: 4096, image sequence number: 241259631
[ 6.182924] ubi0: available PEBs: 0, total reserved PEBs: 223, PEBs reserved for bad PEB handling: 14
[ 6.201311] ubi0: background thread "ubi_bgt0d" started, PID 913
[ 6.204134] block ubiblock0_0: created from ubi0:0(rootfs)
[ 6.224302] ubiblock: device ubiblock0_0 (rootfs) set to be root filesystem
[ 6.238184] hctosys: unable to open rtc device (rtc0)
[ 6.254622] VFS: Mounted root (squashfs filesystem) readonly on device 254:0.
[ 6.269304] Freeing unused kernel memory: 208K
[ 6.278249] This architecture does not have kernel memory protection.
Bootloop version (r50814):
Code:
[ 5.119495] searching for nvram
[ 5.125842] nvram size = 2097152
[ 5.703034] found nvram at 1E0000
[ 5.731065] no valid UBI magic found inside mtd3
[ 5.731114] hctosys: unable to open rtc device (rtc0)
[ 5.751510] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[ 5.766410] Please append a correct "root=" boot option; here are the available partitions:
[ 5.783045] 1f00 1024 mtdblock0
[ 5.783051] (driver?)
[ 5.796059] 1f01 1024 mtdblock1
[ 5.796064] (driver?)
[ 5.809063] 1f02 32768 mtdblock2
[ 5.809068] (driver?)
[ 5.822088] 1f03 28672 mtdblock3
[ 5.822093] (driver?)
[ 5.835102] 1f04 2048 mtdblock4
[ 5.835107] (driver?)
[ 5.848109] 1f05 1024 mtdblock5
[ 5.848114] (driver?)
[ 5.861118] 1f06 80896 mtdblock6
[ 5.861124] (driver?)
[ 5.874141] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 5.890603] Rebooting in 1 seconds..
Joined: 06 Jun 2006 Posts: 7711 Location: Dresden, Germany
Posted: Fri Jan 10, 2025 8:11 Post subject:
MysticCobra wrote:
Hah. Well, I don't know how likely it is that this problem will get fixed, but if it doesn't, I don't want it to be because I didn't provide the data needed to track down the bug.
But I also don't want to bury useful data with a lot of noise, so this will probably be the last set of logs I post unless someone specifically asks me to do something else.
I think I've identified where the firmware first starts to bootloop:
r50786 seems to be the last version that boots into a stable config.
r50814 was the first version that bootloops.
I had previously reported that r50963 has a different problem--it halts ~2 minutes after startup. That is still true. But all the other versions I've tried from 50814 onward bootloop: r50814, r51154, r57447, r57595, r59876, r59045.
Serial logs capturing the firmware load and subsequent boot for r50786 (good) and r50814 (bootloop) are attached.
Please let me know if there's anything else I can do to help.
i think i found the problem. but please post only latest logs. not from 50814. thats very old.
i'm build new images right now an will upload them later today. i hope its fixed then _________________ "So you tried to use the computer and it started smoking? Sounds like a Mac to me.." - Louis Rossmann https://www.youtube.com/watch?v=eL_5YDRWqGE&t=60s
i think i found the problem. but please post only latest logs. not from 50814. thats very old.
i'm build new images right now an will upload them later today. i hope its fixed then
Many thanks--I'll keep my eyes open and check it out when I see it.
Sorry for the old data. Thought it might be relevant/helpful to know when the problem first manifested.
Joined: 06 Jun 2006 Posts: 7711 Location: Dresden, Germany
Posted: Sat Jan 11, 2025 3:57 Post subject:
MysticCobra wrote:
BrainSlayer wrote:
i think i found the problem. but please post only latest logs. not from 50814. thats very old.
i'm build new images right now an will upload them later today. i hope its fixed then
Many thanks--I'll keep my eyes open and check it out when I see it.
Sorry for the old data. Thought it might be relevant/helpful to know when the problem first manifested.
old logs are good as reference. thats true but latest logs are important for seeing the error.
your error is no software bug btw. your nand flash has bad sectors before the filesystem starts, so everything is shiftet. i wrote a workaround for this but of course i need your feedback if its working. attached here is just the webflash image. try it if it works and if not, log of course
