Removing unnecessary/wasterful ^M's from nvram strings

Post new topic   Reply to topic    DD-WRT Forum Index -> General Questions
Goto page 1, 2  Next
Author Message
puterboy2
DD-WRT User


Joined: 24 Feb 2019
Posts: 115

PostPosted: Fri Aug 26, 2022 1:12    Post subject: Removing unnecessary/wasterful ^M's from nvram strings Reply with quote
It seems that at least on Windows, multi-line nvram string variables such as keys, certificates, and some multi-line commands have a carriage return (^M) appended to the end of each line harking back to the old DOS '/n/r' convention (though it doesn't seem to be consistent).

Adding somewhat random ^M's to the end of each line is "messy", inconsistent and wastes valuable NVRAM space.

Any suggestions on how to remove them? (beyond writing a kludgy script to rewrite the variables).
Any reason that the default dd-wrt GUI doesn't apply a 'tr' or 'sed' to the inputs to remove these old DOS convention line endings.
Sponsor
dale_gribble39
DD-WRT Guru


Joined: 11 Jun 2022
Posts: 749

PostPosted: Fri Aug 26, 2022 1:43    Post subject: Reply with quote
What exactly are you using to view / create a text-only version of your nvram variables? This looks like a Windows / Notepad (not Notepad++) problem. I've never noticed this in PuTTY or Terminal.
_________________
"The woods are lovely, dark and deep,
But I have promises to keep,
And miles to go before I sleep,
And miles to go before I sleep." - Robert Frost

"I am one of the noticeable ones - notice me" - Dale Frances McKenzie Bozzio

<fact>code knows no gender</fact>

This is me, knowing I've ruffled your feathers, and not giving a ****
Some people are still hard-headed.

--------------------------------------
Mac Pro (Mid 2012) - Two 2.4GHz 6-Core Intel Xeon E5645 processors 64GB 1333MHz DDR3 ECC SDRAM OpenSUSE Leap 15.4
blkt
DD-WRT Guru


Joined: 20 Jan 2019
Posts: 4580

PostPosted: Fri Aug 26, 2022 1:50    Post subject: Reply with quote
Windows & Linux handle carriage return, line feed differently. Play with Notepad++, do some research CR LF BOM.
You can guess which OS has more unnecessary and wasteful text files. :) PuTTY Telnet / SSH as root, nvram show.
Enable logs printable output, or click upper left copy all to clipboard, or right click top bar not terminal it will paste.
puterboy2
DD-WRT User


Joined: 24 Feb 2019
Posts: 115

PostPosted: Fri Aug 26, 2022 2:48    Post subject: Reply with quote
I actually do all my editing in text-based emacs in Linux (via PuTTy) but I run my browser on Windows. I never use Notepad.
This is coming from cutting & pasting from a PuTTy window into chrome on my browser.

Not sure where/how the ^M is introduced...
dale_gribble39
DD-WRT Guru


Joined: 11 Jun 2022
Posts: 749

PostPosted: Fri Aug 26, 2022 2:57    Post subject: Reply with quote
Into "what" in your Chrome Browser? What are you pasting to? I've never seen this copying from PuTTY in Windows and pasting into Notepad++, saving the file and then opening it up in nano or vi.
_________________
"The woods are lovely, dark and deep,
But I have promises to keep,
And miles to go before I sleep,
And miles to go before I sleep." - Robert Frost

"I am one of the noticeable ones - notice me" - Dale Frances McKenzie Bozzio

<fact>code knows no gender</fact>

This is me, knowing I've ruffled your feathers, and not giving a ****
Some people are still hard-headed.

--------------------------------------
Mac Pro (Mid 2012) - Two 2.4GHz 6-Core Intel Xeon E5645 processors 64GB 1333MHz DDR3 ECC SDRAM OpenSUSE Leap 15.4
blkt
DD-WRT Guru


Joined: 20 Jan 2019
Posts: 4580

PostPosted: Fri Aug 26, 2022 3:05    Post subject: Reply with quote
^M is generated when Windows native apps see LF only (Linux). Problem? Windows uses CRLF. Please do research.
puterboy2
DD-WRT User


Joined: 24 Feb 2019
Posts: 115

PostPosted: Fri Aug 26, 2022 3:11    Post subject: Reply with quote
Actually, I can reproduce this without any cutting or pasting...
I tried typing a random multiline string *directly* into a textbox in the GUI from a chrome browser window on a Win11 PC. For example, I typed:
Code:
hello
goodbye

*directly* into "Additional Configuration" under Services/VPN without any editor.

Then running 'ssh <router> nvram get openvpn_config | cat -v' gives:
Code:
DD-WRT v3.0-r49532 std (c) 2022 NewMedia-NET GmbH
Release: 07/20/22
Board: Netgear R6700 v3
hello^M
goodbye
puterboy2
DD-WRT User


Joined: 24 Feb 2019
Posts: 115

PostPosted: Fri Aug 26, 2022 3:15    Post subject: Reply with quote
blkt wrote:
^M is generated when Windows native apps see LF only (Linux). Problem? Windows uses CRLF. Please do research.


I know that. I have been using Linux for almost 30 years, since version 0.99. I don't need to "do research" to know about dos vs. linux conventions for end of line..

As per my earlier post, dd-wrt is being inconsistent.
It seems like the \n\r is being introduced by chrome under Windows but sometimes dd-wrt filters out the \r (e.g., in the 'commands' textbox) and sometimes it doesn't.

Typing the same strings into 2 different text boxes shouldn't have such different behaviors.
puterboy2
DD-WRT User


Joined: 24 Feb 2019
Posts: 115

PostPosted: Fri Aug 26, 2022 3:16    Post subject: Reply with quote
dale_gribble39 wrote:
Into "what" in your Chrome Browser? What are you pasting to? I've never seen this copying from PuTTY in Windows and pasting into Notepad++, saving the file and then opening it up in nano or vi.


Replicate what I did -- no cutting or pasting.
Simply type a multi-line string into the text boxes I mentioned in Chrome under Windows.
blkt
DD-WRT Guru


Joined: 20 Jan 2019
Posts: 4580

PostPosted: Fri Aug 26, 2022 3:17    Post subject: Reply with quote
<facepalm> Then do not use Windows Terminal SSH. Do not use Windows Chrome running commands in WebUI.
puterboy2
DD-WRT User


Joined: 24 Feb 2019
Posts: 115

PostPosted: Fri Aug 26, 2022 3:47    Post subject: Reply with quote
Meanwhile, I "fixed" the problem by first identifying the "offending" variables by running
Code:
 ssh <router> nvram show | grep -v

Then for each such variable, running the 1-liner
Code:
(var=<variable>; ssh -q <router> "nvram set $var=\"\$(nvram get $var | sed 's/\r$//')\"")
puterboy2
DD-WRT User


Joined: 24 Feb 2019
Posts: 115

PostPosted: Fri Aug 26, 2022 4:01    Post subject: Reply with quote
blkt wrote:
<facepalm> Then do not use Windows Terminal SSH. Do not use Windows Chrome running commands in WebUI.


<facepalm> Nothing to do with SSH.
It happens so long as you use *ENTER* text (whether directly typing or pasting) into the GUI when running on a Windows browser.
I confirmed this behavior in both Chrome and Edge.

So basically, unless the answer is not to use Windows, then I maintain that this is a bug or at least undesirable behavior and an inconsistency in how the dd-wrt GUI reads in text strings.

Any half-decent program will filter out the useless \r characters.
It seems that dd-wrt does this for some GUI text fields but not others.

I posted a way to manually fix this after the fact so it's no longer an issue for me.
The right fix would be to chomp the final character in any line that ends in \r.
Best practice is to do that so that strings are formatted and stored consistently independent of which browser/OS is used to enter them.

If the moderators/DD-WRT Gurus think the current behavior is the desired behavior then so be it... you can bring a horse to water...
yoyoma2
DD-WRT User


Joined: 24 Sep 2016
Posts: 330

PostPosted: Fri Aug 26, 2022 4:14    Post subject: Reply with quote
I've also wondered why dd-wrt nvram variables use CR LF line endings for multi-line variables instead of the native linux standard. Even though my "Additional Jobs" are entered from firefox linux, the following shows a ^M after each line. A hex editor shows the familiar DOS 0D 0A line endings.
Code:
root@DD-WRT:~# nvram get cron_jobs > var.bin
root@DD-WRT:~# vi var.bin

Perhaps the goal was to make nvram dumps more readable for Windows users? Even WordPad from decades ago could display linux text files properly so not much use.

Someone could ask BS or log a bug (it's not really a bug).

PS. There's a function in the code (forgot where) that translates any type of line endings in nvram variables to \n before putting them in the GUI so your hack should work until dd-wrt puts the CRLFs back in.
puterboy2
DD-WRT User


Joined: 24 Feb 2019
Posts: 115

PostPosted: Fri Aug 26, 2022 4:27    Post subject: Reply with quote
yoyoma2 wrote:
I've also wondered why dd-wrt nvram variables use CR LF line endings for multi-line variables instead of the native linux standard. Even though my "Additional Jobs" are entered from firefox linux, the following shows a ^M after each line. A hex editor shows the familiar DOS 0D 0A line endings.
Code:
root@DD-WRT:~# nvram get cron_jobs > var.bin
root@DD-WRT:~# vi var.bin

Perhaps the goal was to make nvram dumps more readable for Windows users? Even WordPad from decades ago could display linux text files properly so not much use.

Someone could ask BS or log a bug (it's not really a bug).

PS. There's a function in the code (forgot where) that translates any type of line endings in nvram variables to \n before putting them in the GUI so your hack should work until dd-wrt puts the CRLFs back in.


And my hack works until/unless you re-save/re-apply the settings on a page that has such variables.

I agree it's not really a bug in that nothing is broken - it's just messy, inconsistent, and a waste of a char per line when nvram can be scarce.

I seems ridiculous that cron_jobs adds a \r while rc_startup doesn't -- what rationale is there for that????
yoyoma2
DD-WRT User


Joined: 24 Sep 2016
Posts: 330

PostPosted: Fri Aug 26, 2022 13:14    Post subject: Reply with quote
The fix_cr function called when getting multi-line nvram variables is here. I totally agree it should be called when setting nvram variables to save space and be consistent with the native platform.

It's probably a browser thing that dd-wrt chose to fix on the way out of nvram rather than on the way into nvram. Log a bug pointing out the nvram savings aspect and that your hack doesn't cause problems. Worst-case it gets closed as not a bug.
Goto page 1, 2  Next Display posts from previous:    Page 1 of 2
Post new topic   Reply to topic    DD-WRT Forum Index -> General Questions All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum