easyddup a simple upgrade/downgrade utility

Post new topic   Reply to topic    DD-WRT Forum Index -> Contributions Upload
Goto page Previous  1, 2, 3 ... 5, 6, 7 ... 19, 20, 21  Next
Author Message
IONK
DD-WRT Guru


Joined: 19 Aug 2011
Posts: 951

PostPosted: Sat Mar 27, 2021 3:22    Post subject: Reply with quote
yoyoma2 wrote:
The shell on dd-wrt is ash so "the backslash ''\'' acts as an escape character, causing the following character to be treated literally" e.g.

\$ becomes $

Try adding special characters one by one and preceding the problematic characters with a backslash. You might have to html encode some characters like spaces? Please let us know what works.

Make a copy of your checknewversion.sh file so it doesn't get lost in the next upgrade. I was thinking of renaming it to checknewversion.sh.sample in the tar file to avoid overwriting user changes.
sorry I was very busy last week and didn't have time for trying it out.
Some suggestions:
  • (from stackoverflow link in checknewversion.sh) remove --insecure switch (tested working for gmail without that switch - but haven't tried other hosts yet)
  • (from stackoverflow link in checknewversion.sh) use the .netrc file to store login details - probably don't need to worry about escaping special chars, and it will be easier to update easyddup
  • seperate ini file for settings
    • host name/port
    • dest email (or emails)
    • path to .netrc file,
    • option to use user's email template instead of the default email template in checknewversion.sh
    • extra param when calling easyddup.sh (e.g. -r reva)
    • extra param for curl (e.g. --insecure)
  • in email template: include forum link for new build thread, so that user can check before flashing. The link should directly point to the thread matching chipset (broadcom/atheros/others) and build number (in the rare case that there's no match: use the subforum link for matching hardware only)


EDIT: any update for easyddup from the board of moderators?
EDIT2: @kernel-panic69 mentioned about adding easyddup to ddwrt firmware, which I don't think it will happen at all, but how about adding easyddup to entware repo?

_________________
▫ RSS feed for DD-WRT releases (2024): https://rsseverything.com/feed/7d36ab68-7733-46c3-bd8a-9c54c5cef08c.xml
easyddup - A simple upgrade/downgrade utility by yoyoma2 --- as featured in 📌 Reference Links, stickies retired back to forum (Moderator's Pick 🌟)
Sponsor
yoyoma2
DD-WRT User


Joined: 24 Sep 2016
Posts: 371

PostPosted: Sat Mar 27, 2021 21:05    Post subject: Reply with quote
New version posted in OP (March 27th).

Changes:

First attempt at a checknewversion.ini file. The tar file now includes a checknewversion.ini.sample so you can install overtop your old easyddup without losing your email settings.

The easyddup arguments are now in the ini file so revision arguments like -r or -R for routers that need it can be added there.

Note that if you don't use less secure apps for a long time, gmail turns the setting off automatically. Found out the hard way...

Removed the --insecure and it still works. I wonder if the --cacert stuff can also be removed now. BS builds have upgraded some ssh related stuff since I initially encountered this.

Those not using email notifications are not affected by the changes in this version.

Comments:

Entware is a general purpose embedded thing unrelated to dd-wrt.

As a noob I fell into the router database trap and ddup was a factor in my choice of kong builds over BS builds. An easyddup light (without USB storage) would be a great asset for dd-wrt IMHO. I really don't care as I use the community build so easyddup is useless to me other than the email notification feature.
IONK
DD-WRT Guru


Joined: 19 Aug 2011
Posts: 951

PostPosted: Sun Mar 28, 2021 9:56    Post subject: Reply with quote
Thanks for the update.
_________________
▫ RSS feed for DD-WRT releases (2024): https://rsseverything.com/feed/7d36ab68-7733-46c3-bd8a-9c54c5cef08c.xml
easyddup - A simple upgrade/downgrade utility by yoyoma2 --- as featured in 📌 Reference Links, stickies retired back to forum (Moderator's Pick 🌟)
IONK
DD-WRT Guru


Joined: 19 Aug 2011
Posts: 951

PostPosted: Sun Mar 28, 2021 10:07    Post subject: Reply with quote
IONK wrote:
For cleaning up the nvram backup, I think there should be another script to check & cleanup the duplicated files, then there's a param in the ini file to enable/disable calling that new cleanup script. The way to identify the duplicate is to compare the hashes of nvram-all-* files. If the nvram-all- are the same then the nvram-ini-, nvram-restore-, nvram-usr- should be identical also. Only the older duplicate files should be deleted so that nvram-restore-latest.sh is not broken. Probably it's like this:
  1. sort nvram-all-* files by name in reverse order (ls nvram-all-* | sort -r)
  2. for each nvram-all- file in sorted list
    1. get unique part of file name (date time mac behind the nvram-all- prefix)
    2. get hash (md5sum from busybox)
    3. compare with previous file's hash (null if first file)
      1. if same hash: delete nvram-all-, nvram-restore-, nvram-usr- with the same date time mac above
      2. else previous file's hash = current file's hash
I've opened the backup setting files and realized that the nvram-all-*.txt files do not have the exact same structure: the settings are not in the fixed sequence but they are listed in a random(?) order, so my idea above is not 100% applicable.
However, I've created a (probably working) script that will sort before comparing the settings. Please take a look, improve and include in easyddup if it's good enough.

_________________
▫ RSS feed for DD-WRT releases (2024): https://rsseverything.com/feed/7d36ab68-7733-46c3-bd8a-9c54c5cef08c.xml
easyddup - A simple upgrade/downgrade utility by yoyoma2 --- as featured in 📌 Reference Links, stickies retired back to forum (Moderator's Pick 🌟)
yoyoma2
DD-WRT User


Joined: 24 Sep 2016
Posts: 371

PostPosted: Mon Mar 29, 2021 1:10    Post subject: Reply with quote
New version posted in OP (March 28th).

Changes:

Added a callback mechanism for user addons. The first one is cleanupCB.sh by @IONK (thanks!). The addons are sourced/included so they have access to the easyddup variables so handle with care. Added an mkdir in cleanup.txt to avoid an error message and put the whole thing in a function called cleanup_duplicate_nvram_backups. I have not tested or even looked at the cleanupCB.sh addon's code so enable at your own risk.

Addons come in two types, startup and pre-burn depending on when you want it called you put your function name in the appropriate list (STARTUP_CB or PREBURN_CB). See easyddup.ini.sample for a commented out example.

This callback mechanism should also allow implementing a report generation addon discussed earlier in this thread.

Comments:

Noticed that the nvram tools used by easyddup don't work on the community build. They work on an old BS r7800 build. If the following command fails when run in the Commands screen then nvram tools will also fail.

readlink -f ~root

This script could easily be used as a workaround if this becomes an issue.
IONK
DD-WRT Guru


Joined: 19 Aug 2011
Posts: 951

PostPosted: Mon Mar 29, 2021 4:41    Post subject: Reply with quote
Thanks for the update.

- In easyddup.ini I need to add the path to the callback file: "source $scriptdir/cleanupCB.sh". Is it because my pwd is "/tmp/root" ?

- I've attached a fixed cleanupCB.sh, please help me update to easyddup bundle

- In nvram-all-*.txt I see the lines "start_time=1616942853" and "end_time=1616942778", these values sometimes change. Do you know what's the meaning of these values?

- I'm trying to add a cron job and I see the "check_version" file in easyddup bundle, but I'm not sure how to use it. Am I suppose to copy the content of that file to /jffs/etc/crontab as in DD-WRT CRON Wiki?

_________________
▫ RSS feed for DD-WRT releases (2024): https://rsseverything.com/feed/7d36ab68-7733-46c3-bd8a-9c54c5cef08c.xml
easyddup - A simple upgrade/downgrade utility by yoyoma2 --- as featured in 📌 Reference Links, stickies retired back to forum (Moderator's Pick 🌟)
yoyoma2
DD-WRT User


Joined: 24 Sep 2016
Posts: 371

PostPosted: Mon Mar 29, 2021 15:33    Post subject: Reply with quote
I added the script path in the sample ini (thanks for the bug report) and inserted your updated cleanupCB addon. Had to run dos2unix on it as your line endings went back to CRLF. Notepad++ can edit files with linux line endings.

The start_time and end_time nvram variables indicate when dd-wrt booted up and shutdown and could probably be ignored in a duplicate backup check.

I add a line like the one in check_version in "Administration->Management->Additional Cron Jobs". Tip if you add the comments below at the top of "Additional Cron Jobs", it will appear directly in /tmp/cron.d/cron_jobs without the extra startup commands suggested in the wiki.
Code:
# minute (0-59),
# |      hour (0-23),
# |      |       day of the month (1-31),
# |      |       |       month of the year (1-12),
# |      |       |       |       day of the week (0-6 with 0=Sunday).
# |      |       |       |       |       commands
IONK
DD-WRT Guru


Joined: 19 Aug 2011
Posts: 951

PostPosted: Mon Mar 29, 2021 18:30    Post subject: Reply with quote
yoyoma2 wrote:
I added the script path in the sample ini (thanks for the bug report) and inserted your updated cleanupCB addon. Had to run dos2unix on it as your line endings went back to CRLF. Notepad++ can edit files with linux line endings.

The start_time and end_time nvram variables indicate when dd-wrt booted up and shutdown and could probably be ignored in a duplicate backup check.

I add a line like the one in check_version in "Administration->Management->Additional Cron Jobs". Tip if you add the comments below at the top of "Additional Cron Jobs", it will appear directly in /tmp/cron.d/cron_jobs without the extra startup commands suggested in the wiki.
Code:
# minute (0-59),
# |      hour (0-23),
# |      |       day of the month (1-31),
# |      |       |       month of the year (1-12),
# |      |       |       |       day of the week (0-6 with 0=Sunday).
# |      |       |       |       |       commands
Noted on the line ending format, probably WinSCP messed it up.
I've updated cleanupCB to ignore the start & end time, please help me update on the next release.
Thanks for the cron job tip. I'm thinking of adding the cron job for FW upgrade, then add the .wanup script to send the partially filled report via email. Do you have any tip/trick for the .wanup file? Script Execution - DD-WRT Wiki

_________________
▫ RSS feed for DD-WRT releases (2024): https://rsseverything.com/feed/7d36ab68-7733-46c3-bd8a-9c54c5cef08c.xml
easyddup - A simple upgrade/downgrade utility by yoyoma2 --- as featured in 📌 Reference Links, stickies retired back to forum (Moderator's Pick 🌟)
yoyoma2
DD-WRT User


Joined: 24 Sep 2016
Posts: 371

PostPosted: Tue Mar 30, 2021 23:29    Post subject: Reply with quote
IONK wrote:
Do you have any tip/trick for the .wanup file?
I don't know anything about .wanup scripts. My first thought would be to run easyddup as a @reboot cron job but a quick test found a bug with @reboot jobs. A new reportCB.sh addon could create a partial report in a pre-burn callback function and augment/complete the report in a separate startup callback function also in
reportCB.sh.

Added a couple easyddup.ini keywords to wait for an internet connection and clock initialization (EDDUP_WAIT_FOR_INTERNET, EDDUP_WAIT_FOR_CLOCK). Also included your latest cleanupCB.sh.

Edit: Almost forgot, added a check to only allow a single instance to run at a time so the @reboot bug may not be a huge issue.
IONK
DD-WRT Guru


Joined: 19 Aug 2011
Posts: 951

PostPosted: Wed Mar 31, 2021 16:37    Post subject: Reply with quote
Thanks for the update.
I created the file send_report_easyddup.wanup in /jffs/etc/config/ as instructed in the wiki, and the script really sent out the email after auto FW upgrade.
Unluckily I didn't have time to update easyddup to version 30Mar before the auto FW upgrade, so I couldn't test the new features of easyddup.

Just now I come across this post in the new build thread:
rnix wrote:
Asus RT-N18U firmware is only about 12MB when it used to be around 24MB. It can't be right?
I think it's a good observation. Can we include that check in easyddup also? There's a chance that something goes wrong during the FW compilation process and the FW file is corrupted/incomplete, so even if the downloaded file's size matches the provided size on the server then it's still not safe to flash the FW. I think I've seen this kind of report before.
If we specify the expected size of the firmware in the ini file then easyddup can spot the potential issue then raise a warning and ask for confirmation before proceed with flashing, or add the warning line in the email sent when checing new version.

Probably we need a few more params in the ini file: (the name of the param is anyhow picked from the air)
- EDDUP_EXPECTED_FW_SIZE_MIN (let user fill in, blank means no checking min size)
- EDDUP_EXPECTED_FW_SIZE_MAX (let user fill in, blank means no checking max size)
- EDDUP_EXPECTED_FW_SIZE_CHECK_PROMPT : Y: show prompt waiting for user input if size check fails, N: directly return error (abort operation) if size check fails

because checknewversion.sh depends on the return value of easyddup.sh, maybe it's needed to use return code -1 to indicate file size checking fails.

_________________
▫ RSS feed for DD-WRT releases (2024): https://rsseverything.com/feed/7d36ab68-7733-46c3-bd8a-9c54c5cef08c.xml
easyddup - A simple upgrade/downgrade utility by yoyoma2 --- as featured in 📌 Reference Links, stickies retired back to forum (Moderator's Pick 🌟)
yoyoma2
DD-WRT User


Joined: 24 Sep 2016
Posts: 371

PostPosted: Wed Mar 31, 2021 20:05    Post subject: Reply with quote
Added EDDUP_EXPECTED_FW_SIZE_MIN and EDDUP_EXPECTED_FW_SIZE_MAX. When the expected size check fails the error also ends up in easyddup.log. Also added a few more messages to the log when the burn succeeds so a script can tell the difference.

BS suggests using "Administration->Commands" Startup/Shutdown for your use case
IONK
DD-WRT Guru


Joined: 19 Aug 2011
Posts: 951

PostPosted: Sat Apr 03, 2021 3:17    Post subject: Reply with quote
Thanks for the update.
I didn't read carefully the wiki and missed out the part about the script being executed multiple times whenever WAN is up. At the moment I've moved my send email script to startup command and stolen the part of easyddup to wait for internet and to disable running multiple instances.

Is it ok to add easyddup to my signature again?

_________________
▫ RSS feed for DD-WRT releases (2024): https://rsseverything.com/feed/7d36ab68-7733-46c3-bd8a-9c54c5cef08c.xml
easyddup - A simple upgrade/downgrade utility by yoyoma2 --- as featured in 📌 Reference Links, stickies retired back to forum (Moderator's Pick 🌟)
yoyoma2
DD-WRT User


Joined: 24 Sep 2016
Posts: 371

PostPosted: Sat Apr 03, 2021 15:28    Post subject: Reply with quote
IONK wrote:
stolen the part of easyddup
You did what?!?! Evil or Very Mad Just kidding. Laughing

About the signature. For a moment it seems BS thought easyddup code had some relation to kong's code. This has been cleared up. BS even took the time to give a tip about Startup commands regarding these scripts so he seems fine with it.
msj100
DD-WRT Novice


Joined: 30 Jul 2007
Posts: 33
Location: Melbourne, Australia

PostPosted: Sat Apr 10, 2021 3:45    Post subject: Reply with quote
IONK wrote:
I'm playing with the send email function, but I encountered curl error 67 login denied. Is there any pointer/tip/trick?

my settings:
MAILSERVER=smtps://smtp.gmail.com:465
MYMAILSERVER=mydomain.com
EMAILUSERLOCAL="mylocalmailuser@$MYMAILSERVER"
EMAILUSER="******@gmail.com"
EMAILCRED="******"
EMAILTO="******@gmail.com"

.
.
.

EDIT:
my password has special characters/symbols *$#@!



Try changing the surrounding quotation marks for EMAILCRED from double quotes to single quotes - ie:

EMAILCRED='password_with_metacharacters_here'

_________________
Asus RT-AC87U - dd-wrt
Asus RT-AC68U - dd-wrt
Asus RT-N16 - dd-wrt
TP-Link TL-MR3020 - OpenWRT
VAPs / VLANs / PBR / Entware
msj100
DD-WRT Novice


Joined: 30 Jul 2007
Posts: 33
Location: Melbourne, Australia

PostPosted: Sat Apr 10, 2021 4:28    Post subject: Reply with quote
yoyoma2 wrote:
blkt wrote:
So, any advice on easyddup? Scrap it or continue?
If people keep giving good feedback as recently I'll maintain it. The invitation given earlier for a dd-wrt enthusiast to take over still stands. Once it's working for all routers, I don't foresee many changes required to keep it going.


Just discovered this thread - I've been looking for an easier way to upgrade my routers and this looks like a great candidate - so thanks for writing this!

I tried to upgrade my Asus RT-16N (currently running v3.0-r41379 big) - however, because there's no RT-16N specific device directory, it fails:

Current build: 41379
Select a build to install (or Q): 34
Selected 04-09-2021-r46316
./easyddup.sh: line 209: files: No builds found in folder asus-rt-n16


..the directory I need to use is the (generic) broadcom_K3X - full path:

https://download1.dd-wrt.com/dd-wrtv2/downloads/betas/2021/04-09-2021-r46316/broadcom_K3X/dd-wrt.v24-46316_NEWD-2_K3.x-big-RT-N16.trx

..however, easyddup.ini does not accommodate this 'broadcom exception' at the moment.

A thought...if the device-specific directory isn't found, for broadcom devices ( grep -i "system type.*broadcom" /proc/cpuinfo ), perhaps check the generic broadcom directories for the upgrade image?

There are 4 generic 'broadcom' directories -

broadcom_K26
broadcom_K3X
broadcom_K3X_mipsel32r1
broadcom

..although I've only ever used broadcom_K26 and broadcom_K3X - which I'm guessing could be distinguished using a `uname -r`.

I have a few RT-N16s and can test if you're open to making changes to accommodate?

_________________
Asus RT-AC87U - dd-wrt
Asus RT-AC68U - dd-wrt
Asus RT-N16 - dd-wrt
TP-Link TL-MR3020 - OpenWRT
VAPs / VLANs / PBR / Entware
Goto page Previous  1, 2, 3 ... 5, 6, 7 ... 19, 20, 21  Next Display posts from previous:    Page 6 of 21
Post new topic   Reply to topic    DD-WRT Forum Index -> Contributions Upload All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum