Joined: 08 Jun 2006 Posts: 247 Location: Prince Edward Island - Canada
Posted: Thu Dec 08, 2011 16:33 Post subject: Possible SSH problems with builds 17967 and 17990
*** Warning - Long and Involved ***
With build 17967, the dropbear (ssh server/client/key generator) has been updated and appears to function slightly different than before.
Previously the host key files located in the routers /tmp/root/.ssh directory were in 'openssh' format, they were 'ascii' characters that started with a header line '------BEGIN.....' and ended with a footer of '-----END.....'. These key values are also stored in the NVRAM variables sshd_dss_host_key and sshd_rsa_host_key in the same openssh format.
Now, the files in /tmp/root/.ssh are dropbear formatted binary files. All good so far.
If the NVRAM variables were not cleared after the upgrade to 17967 you are unable to connect to your router using SSH. (Yes murrkf -- we know .. we should have cleared nvram after upgrading as outlined in the Peacock Thread )
What the workaround does is to create the two required files in the appropriate format.
Now.. if you are still with me here's were we get to narrow down the problem a bit. There is key converter as part of the dropbear set of programs. You can use this to convert from dropbear to openssh format and I expect that this is what is used behind the scenes to store the values in NVRAM as openssh keys and re-create the binary dropbear files after each boot.
It appears that the process to generate and convert the DSS key works fine. I see an NVRAM variable with a key in it. Unfortunately the RSA conversion doesn't seem to work (the dropbearconvert program segfaults) so it creates the file but does not create the appropriate NVRAM variable sshd_rsa_host_key but leaving it blank.
The downside of this is that when the router reboots it generates a new RSA key file and then (I assume) segfaults while it is trying to create the openssh key for the NVRAM variable.
Is anyone else who reset to defaults seeing this behaviour?
A quick way to tell is that after a reboot your router and start an ssh session (I use putty) it will tell you that the routers key has changed.
[edited to slightly change wording]
Last edited by dc on Mon Dec 12, 2011 14:16; edited 2 times in total
I can confirm 100% all your findings because I experienced the same doing tests by myself after having that first -bad boy not clearing nvram- problem.
I've now done many tests in different ways:
*in my old config, clearing nvram-related variables. Problem was there, rsa was regenerated at every boot.
*after a 30/30/30, setting all again, all working except the same: rsa was new at every reboot and programs was complaining. Tested the file binary content and looks different too.
*after a 30/30/30, firmware flash, 30/30/30, problem was still the same, nvram dss is ok but rsa is not there in any test I've done.
Yes.. that's my original post with a quick 'make it work' fix. After further testing it does appear to be the dropbearconvert segfaulting. If you run it from the command line it things the RSA key is a DSS key and then segfaults. I'll create a new trac ticket.
Please, everyone that is experiencing similar problems, please give an help to confirm/understand and possibly fix this problem, indicating:
*hardware used/tried on
*if the upgrade was done defaulting the settings, ie clear nvram, 30/30/30, or selecting default settings in dd-wrt web ui
*what's the answer when giving these commands in telnet/ssh/web ui command page into administration:
Isn't any problem with latest dropbear (054) if that is in use ?
I am not understanding your question.
dropbear -V gives:
Dropbear sshd v2011.54
and it works good when connected, for exchanging commands and files, all is working there. The problem is with dropbearconvert. And with the fact a reboot / power outage or anything that lets the router restart makes it having a different identification rsa key, which is bad because it's used to verify the server you're connecting to is the right one.
4. I think SSH is regenerating the keys each reboot, but need to test this, just need to do some work now that I'm up and running again.
I can confirm the same result regarding SSH on reboot.
Could you do the tests I was asking for here http://www.dd-wrt.com/phpBB2/viewtopic.php?p=654142#654142 ?
I am trying to troubleshoot the problem since days and reporting any experience here http://svn.dd-wrt.com:8000/ticket/2283
Could you do it too, indicating your hardware, way you used to upgrade (cleared nvram/settings or not), results of the commands I indicate there (mainly dropbearconvert dropbear openssh /tmp/root/.ssh/sshd_host_rsa_key /tmp/root/.ssh/rsa that here gives a segmentation fault, and nvram get sshd_host_rsa_key that here gives a blank line)?
Using "DD-WRT v24-sp2 (12/08/11) big - build 17990M NEWD-2 K2.6 Eko" on my E4200 I get:
root@monty:~# nvram get sshd_rsa_host_key
root@monty:~# dropbearconvert dropbear openssh /tmp/root/.ssh/ssh_host_rsa_key /
Key is a DSS key
root@monty:~# nvram get sshd_rsa_host_key
As with all these builds I do a 30/30/30 prior to upgrade of firmware, upgrade the firmware, perform another 30/30/30, then manually re-enter all my settings. I do not use any backups, except taking one beforehand in case I want to quickly revert to the previous firmware.
don't know if it helps, but i'm running vpnkong - build 17940M with Dropbear sshd v2011.54 on E4200v1. no ssh problems here!
This was VERY useful in fact!!!!
To the devs and the others, this is what I've just tried and my conclusions:
*from 17990M I reverted to Kong's 17940M without clearing nvram. In fact connection to sshd was refused. It's expected, different key kinds. So I've launched
nvram unset sshd_rsa_host_key
At reboot I had a newly generated rsa key that WAS into nvram variable and WAS making dropbear work. Expected, again, ok.
Here it is my rough idea that came out reading that build works: there could be a problem with the currently compiled binary of dropbearmulti (that contains server, convert, key).
*So I've saved all the binaries with an scp connection.
*Now, let's go to 17990M (EKO or BS, no difference between them here), again, without clearing nvram. Connection refused, unset variable, reboot, exactly as before. But, after reboot, nvram variable is empty as expected and dropbear is running with the new expected temp key.
*Now, let's check if there's a problem of some other kind and not only a binary problem. While running 17990M I've sent the Kong's 17940M binary I had saved to /tmp/KONGsbin with scp. I then have created a symlink called "dropbearconvert" to dropbearmulti (the only binary) to be able to launch the converter. Made them executable. Then I've tried those commands to check the difference between their acting on the SAME dropbear type key actually saved into /tmp/root/.ssh/
root@DD-WRT:~/.ssh# cd /tmp/KONGsbin
root@DD-WRT:/tmp/KONGsbin# ./dropbearconvert dropbear openssh /tmp/root/.ssh/ssh_host_rsa_key rsaKONG
Key is a RSA key
Wrote key to 'rsaKONG'
root@DD-WRT:/tmp/KONGsbin# dropbearconvert dropbear openssh /tmp/root/.ssh/ssh_host_rsa_key rsaBS
Key is a DSS key
As you can see, Kong's "old?" binary works fine!!! The actual one into the last compilation does not. Dimensions are exactly the same but the new binary doesn't seem to be correctly compiled and working as expected. Also, this could be the reason why we were not understanding why those build weren't accepting the "old" keys while they should have to, because we are speaking of the exactly same dropbear build as far as I understand:
So, 2011.54, same version, different behavior. I think this is not good and something has gone wrong while compiling for reasons I can't understand or due to some new compilation settings (I barely remember a compression mod of the makefile, but I don't know if it's important here).
I think it could be great for all to put out a fixed build containing the old dropbear's binaries compiled from Kong's or from an older build. I don't get how could we make the root / writable and wouldn't want to go further and make damages, I think that a new build would be simpler for everyone
Sorry for the long thread. Hope it helped and doesn't contain nonsense, it could as I'm not an expert/developer or anything similar
Last edited by ErMeglio on Sun Dec 11, 2011 0:29; edited 1 time in total