Using git to download source codes of DD-WRT

Post new topic   Reply to topic    DD-WRT Forum Index -> General Questions
Author Message
mwchang
DD-WRT Guru


Joined: 26 Mar 2013
Posts: 1858
Location: Hung Hom, Hong Kong

PostPosted: Tue Aug 15, 2023 10:41    Post subject: Using git to download source codes of DD-WRT Reply with quote
It's possibly better to start a new thread about downloading source codes of DD-WRT.

ho1Aetoo wrote:
I have a R7800 but I wouldn't use it for something like this because it's a small router and not a workstation. Rolling Eyes
and why don't you use git or svn?
Code:
opkg install git-http
git clone https://github.com/mirror/dd-wrt.git
cd dd-wrt
git pull

then you don't have to download and unpack a 20GB archive every time there are updates

Following your advice, seemed that 'git clone' will first compress the codes:
Code:
# git clone https://github.com/mirror/dd-wrt.git
Cloning into 'dd-wrt'...
remote: Enumerating objects: 2228322, done.
remote: Counting objects: 100% (115263/115263), done.
remote: Compressing objects: 100% (92174/92174), done.
Receiving objects:  21% (478625/2228322), 687.91 MiB | 1.70 MiB/s

Um... and it's using the SWAP partition!

_________________
Router: Asus RT-N18U (rev. A1)

Drink, Blink, Stretch! Live long and prosper! May the Force and farces be with you!

Facebook: https://www.facebook.com/changmanwai
Website: https://sites.google.com/site/changmw
SETI@Home profile: http://setiathome.berkeley.edu/view_profile.php?userid=211832
GitHub: https://github.com/changmw/changmw


Last edited by mwchang on Mon Aug 21, 2023 7:54; edited 3 times in total
Sponsor
mwchang
DD-WRT Guru


Joined: 26 Mar 2013
Posts: 1858
Location: Hung Hom, Hong Kong

PostPosted: Tue Aug 15, 2023 11:04    Post subject: Reply with quote
Related:

DD-WRT :: View topic - Unable to search source codes of DD-WRT in GitHub
https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=334677

DD-WRT :: View topic - Compiling custom kernel
https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=333511

DD-WRT :: View topic - [SOLVED]About source codes of DD-WRT
https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=333023

DD-WRT :: View topic - [SOLVED] Searching source codes
https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=332814

DD-WRT :: View topic - Compiling my own DD-WRT firmware?
https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=332258

_________________
Router: Asus RT-N18U (rev. A1)

Drink, Blink, Stretch! Live long and prosper! May the Force and farces be with you!

Facebook: https://www.facebook.com/changmanwai
Website: https://sites.google.com/site/changmw
SETI@Home profile: http://setiathome.berkeley.edu/view_profile.php?userid=211832
GitHub: https://github.com/changmw/changmw
mwchang
DD-WRT Guru


Joined: 26 Mar 2013
Posts: 1858
Location: Hung Hom, Hong Kong

PostPosted: Tue Aug 15, 2023 11:15    Post subject: Reply with quote
This git clone process really uses up memory....

Code:
# git clone https://github.com/mirror/dd-wrt.git
Cloning into 'dd-wrt'...
remote: Enumerating objects: 2228322, done.
remote: Counting objects: 100% (115263/115263), done.
remote: Compressing objects: 100% (92174/92174), done.
Receiving objects:  81% (1821351/2228322), 4.32 GiB | 1.70 MiB/s

Code:
# free
              total        used        free      shared  buff/cache   available
Mem:         254492      147300       35720           0       71472       13548
Swap:       1048572      111712      936860

_________________
Router: Asus RT-N18U (rev. A1)

Drink, Blink, Stretch! Live long and prosper! May the Force and farces be with you!

Facebook: https://www.facebook.com/changmanwai
Website: https://sites.google.com/site/changmw
SETI@Home profile: http://setiathome.berkeley.edu/view_profile.php?userid=211832
GitHub: https://github.com/changmw/changmw
mwchang
DD-WRT Guru


Joined: 26 Mar 2013
Posts: 1858
Location: Hung Hom, Hong Kong

PostPosted: Tue Aug 15, 2023 11:28    Post subject: Reply with quote
Oh well, my first attempt failed!!!

Code:
Cloning into 'dd-wrt'...                                                                                               
remote: Enumerating objects: 2228322, done.
remote: Counting objects: 100% (115263/115263), done.
remote: Compressing objects: 100% (92174/92174), done.
remote: Total 2228322 (delta 23669), reused 112785 (delta 21737), pack-reused 2113059
Receiving objects: 100% (2228322/2228322), 5.20 GiB | 1.57 MiB/s, done.
fatal: fetch-pack: invalid index-pack output

_________________
Router: Asus RT-N18U (rev. A1)

Drink, Blink, Stretch! Live long and prosper! May the Force and farces be with you!

Facebook: https://www.facebook.com/changmanwai
Website: https://sites.google.com/site/changmw
SETI@Home profile: http://setiathome.berkeley.edu/view_profile.php?userid=211832
GitHub: https://github.com/changmw/changmw
ho1Aetoo
DD-WRT Guru


Joined: 19 Feb 2019
Posts: 3006
Location: Germany

PostPosted: Tue Aug 15, 2023 16:55    Post subject: Reply with quote
try with

Code:
git clone --depth=1 https://github.com/mirror/dd-wrt.git


if that's not enough then

Code:
git config --global pack.packSizeLimit "10m"
git config --global pack.windowMemory "10m"
git config --global pack.threads "1"
git clone --depth=1 https://github.com/mirror/dd-wrt.git


if that's not enough then

don't do it on a fucking router with 256mb ram and a 800Mhz processor - that's insane!

_________________
Quickstart guides:
use Pi-Hole as simple DNS-Server with DD-WRT
VLAN configuration via GUI - 1 CPU port
VLAN configuration via GUI - 2 CPU ports (R7800, EA8500 etc)

Routers
Marvell OCTEON TX2 - QHora-322 - OpenWrt 23.05.3 - Gateway
Qualcomm IPQ8065 - R7800 - DD-WRT - WAP
mwchang
DD-WRT Guru


Joined: 26 Mar 2013
Posts: 1858
Location: Hung Hom, Hong Kong

PostPosted: Wed Aug 16, 2023 14:24    Post subject: Reply with quote
ho1Aetoo wrote:
if that's not enough then

don't do it on a fucking router with 256mb ram and a 800Mhz processor - that's insane!

No, it's very sane! I just don't understand GitHub.

I am merely downloading files. I question the "git clone" command is abusing memory, very likely due to the dictionary size of data compression. If no compression was used and without packaging and without protocol encryption, "256mb ram and a 800Mhz processor" should do it. If it was just simple FTP or SCP files transfer...

I had no problem copying the 4G dd-wrt-master.zip from my Windows PC to Asus RT-N18U via ksmbd. But downloading the 4G file from GitHub via HTTPS was also not working well. I would blame un-necessary protocol encryption for wasting memory and bandwidth.

And remember my complaint about searching DD-WRT's source codes via GitHub website?

Will try your suggestions anyway.

_________________
Router: Asus RT-N18U (rev. A1)

Drink, Blink, Stretch! Live long and prosper! May the Force and farces be with you!

Facebook: https://www.facebook.com/changmanwai
Website: https://sites.google.com/site/changmw
SETI@Home profile: http://setiathome.berkeley.edu/view_profile.php?userid=211832
GitHub: https://github.com/changmw/changmw
kernel-panic69
DD-WRT Guru


Joined: 08 May 2018
Posts: 14249
Location: Texas, USA

PostPosted: Wed Aug 16, 2023 14:46    Post subject: Reply with quote
Yes, I still have no clue why searching the code functionality on GitHub went screwy - whether it was a GitHub change or repository change put in place by BS. There have been several times that the mirror interface was -NOT- in sync with SVN, and SVN is still the preferred way to retrieve full repository code and Linux is still the preferred platform for doing anything with it. I must agree with @ho1Aetoo, trying to use your router as a desktop PC or server to retrieve the GitHub mirror is an exercise in futility.
_________________
"Life is but a fleeting moment, a vapor that vanishes quickly; All is vanity"
Contribute To DD-WRT
Pogo - A minimal level of ability is expected and needed...
DD-WRT Releases 2023 (PolitePol)
DD-WRT Releases 2023 (RSS Everything)

----------------------
Linux User #377467 counter.li.org / linuxcounter.net
ho1Aetoo
DD-WRT Guru


Joined: 19 Feb 2019
Posts: 3006
Location: Germany

PostPosted: Wed Aug 16, 2023 17:26    Post subject: Reply with quote
I wonder why you don't just use your notebook or PC for this, it surely has more RAM and CPU power available.

You don't want to use a VM?
Have you ever heard of dual boot?

You can also just install Linux as a second OS, you can also install Linux on a USB stick and boot the USB stick if needed.

As you may have noticed, your router has 256MB of RAM, which is way too little to unpack 12GB archives or to clone a GIT with 50.000 commits.

On my PC it works fine and an update of the GIT takes a few seconds. (32GB RAM 4x 4,1GHz and a fast SSD)

Is the rent for your penthouse so expensive that you can't afford a decent PC? Wink

_________________
Quickstart guides:
use Pi-Hole as simple DNS-Server with DD-WRT
VLAN configuration via GUI - 1 CPU port
VLAN configuration via GUI - 2 CPU ports (R7800, EA8500 etc)

Routers
Marvell OCTEON TX2 - QHora-322 - OpenWrt 23.05.3 - Gateway
Qualcomm IPQ8065 - R7800 - DD-WRT - WAP
mwchang
DD-WRT Guru


Joined: 26 Mar 2013
Posts: 1858
Location: Hung Hom, Hong Kong

PostPosted: Thu Aug 17, 2023 12:33    Post subject: Reply with quote
"depth=1" took me further, but connection then died. I had similar connection problem when downloading dd-wrt-master.zip using Window$.

OTOH, I am having stuttering issues when watching YouTube video with DD-WRT build 53323! Never had this problem in previous builds. Not sure whether it's YouTube's new behavior.

Maybe I should rollback DD-WRT to build 53045 and try git clone again?

Code:
Cloning into 'dd-wrt'...
remote: Enumerating objects: 614645, done.
remote: Counting objects: 100% (614645/614645), done.
remote: Compressing objects: 100% (453137/453137), done.
Receiving objects:  87% (539154/614645), 1.16 GiB | 1.70 MiB/s
remote: Total 614645 (delta 271185), reused 410859 (delta 147403), pack-reused 0
Receiving objects: 100% (614645/614645), 1.66 GiB | 1.61 MiB/s, done.
Resolving deltas: 100% (271185/271185), done.185)
error: rev-list died of signal 11
fatal: remote did not send all necessary objects

_________________
Router: Asus RT-N18U (rev. A1)

Drink, Blink, Stretch! Live long and prosper! May the Force and farces be with you!

Facebook: https://www.facebook.com/changmanwai
Website: https://sites.google.com/site/changmw
SETI@Home profile: http://setiathome.berkeley.edu/view_profile.php?userid=211832
GitHub: https://github.com/changmw/changmw


Last edited by mwchang on Thu Aug 17, 2023 13:58; edited 5 times in total
mwchang
DD-WRT Guru


Joined: 26 Mar 2013
Posts: 1858
Location: Hung Hom, Hong Kong

PostPosted: Thu Aug 17, 2023 12:34    Post subject: Reply with quote
kernel-panic69 wrote:
I must agree with @ho1Aetoo, trying to use your router as a desktop PC or server to retrieve the GitHub mirror is an exercise in futility.

ho1Aetoo wrote:
On my PC it works fine and an update of the GIT takes a few seconds. (32GB RAM 4x 4,1GHz and a fast SSD)

Is the rent for your penthouse so expensive that you can't afford a decent PC? Wink

I do have a more powerful PC than the Asus RT-N18U. I understand that I am asking too much from a slow router. Thank you both. Smile

_________________
Router: Asus RT-N18U (rev. A1)

Drink, Blink, Stretch! Live long and prosper! May the Force and farces be with you!

Facebook: https://www.facebook.com/changmanwai
Website: https://sites.google.com/site/changmw
SETI@Home profile: http://setiathome.berkeley.edu/view_profile.php?userid=211832
GitHub: https://github.com/changmw/changmw
mwchang
DD-WRT Guru


Joined: 26 Mar 2013
Posts: 1858
Location: Hung Hom, Hong Kong

PostPosted: Fri Aug 18, 2023 10:59    Post subject: Reply with quote
mwchang wrote:
"depth=1" took me further, but connection then died. I had similar connection problem when downloading dd-wrt-master.zip using Window$.

OTOH, I am having stuttering issues when watching YouTube video with DD-WRT build 53323! Never had this problem in previous builds. Not sure whether it's YouTube's new behavior.

Maybe I should rollback DD-WRT to build 53045 and try git clone again?

I fell back to build 52569 and finished depth=1 pull. Now trying 'git fetch --unshallow'.

Update:
Possibly unrelated to DD-WRT builds. Flashed back to 53323 again!

Update 2:
Well, it went further this time with more git config options:
Code:
# cat ~/.gitconfig
[core]
        compression = 0
        packedGitWindowSize = 50m
        packedGitLimit = 50m
[pack]
        threads = 1
        packSizeLimit = 50m
        windowMemory = 50m
        deltaCacheSize = 25m
[http]
        maxRequests = 1
#
# git clone --depth 1 https://github.com/mirror/dd-wrt.git
Cloning into 'dd-wrt'...
remote: Enumerating objects: 614651, done.
remote: Counting objects: 100% (614651/614651), done.
remote: Compressing objects: 100% (453161/453161), done.
remote: Total 614651 (delta 271198), reused 410670 (delta 147383), pack-reused 0
Receiving objects: 100% (614651/614651), 1.66 GiB | 1.61 MiB/s, done.
Resolving deltas: 100% (271198/271198), done.
Checking connectivity: 614651, done.
Segmentation fault7% (740263/844124)


How can I resume a partial git clone??
Code:
fatal: destination path 'dd-wrt' already exists and is not an empty directory.

_________________
Router: Asus RT-N18U (rev. A1)

Drink, Blink, Stretch! Live long and prosper! May the Force and farces be with you!

Facebook: https://www.facebook.com/changmanwai
Website: https://sites.google.com/site/changmw
SETI@Home profile: http://setiathome.berkeley.edu/view_profile.php?userid=211832
GitHub: https://github.com/changmw/changmw
Display posts from previous:    Page 1 of 1
Post new topic   Reply to topic    DD-WRT Forum Index -> General Questions All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum