RSYNC over ssh, and QoS services packet counts?

Post new topic   Reply to topic    DD-WRT Forum Index -> Atheros WiSOC based Hardware
Author Message
ArjenR49
DD-WRT Guru


Joined: 05 Oct 2008
Posts: 666
Location: Helsinki, Finland / nr. Alkmaar, Netherlands

PostPosted: Wed Sep 28, 2022 0:59    Post subject: RSYNC over ssh, and QoS services packet counts? Reply with quote
I have been running a backup job between two sites for many years.
It uses rsync over ssh on a non-standard port.

The router has QoS enabled. Under services priority I have defined a priority for the port used (currently both tcp and udp).

I expected to get a high packet count for this port.
The connection on this port is active as can be seen from the connections table, and data is going out per the bandwidth graphs, too.

However, the packet count is just 1 even though data is obviously going out as the job finishes at some point in time (upload is slow and the files are big). I can see the files at the other end, so yes, the backup job is working and has been for years.

What port is the data going through or how can it be explained that QoS does not see it except for the one packet?


Last edited by ArjenR49 on Wed Sep 28, 2022 6:38; edited 1 time in total
Sponsor
ArjenR49
DD-WRT Guru


Joined: 05 Oct 2008
Posts: 666
Location: Helsinki, Finland / nr. Alkmaar, Netherlands

PostPosted: Wed Sep 28, 2022 6:37    Post subject: Reply with quote
After many hours of backup file content going out, the packet count for the rsync over ssh port has gone up to 14. However, the count for udp packages on any of ports 54 and above has gone up by the 10s of thousands. With nothing else in particular going on, this would indicate that rsync over ssh uses udp for the bulk of transport on some unknown port 54 or above.
However, there is no such connection in the active connections table.

When I check the situation at the other end using SFTP/FileZilla the same port as for rsync over ssh is used, so that may also explain the small increase from the initial 1 packet count.

SFTP and Rsync are running on different machines at this end, but the target is the same NAS at the other end.
DWCruiser
DD-WRT User


Joined: 15 Aug 2016
Posts: 223
Location: Melbourne, Australia

PostPosted: Wed Sep 28, 2022 7:38    Post subject: Reply with quote
Glad that you got your issue partly sorted out.

I can't answer your query but if i may add a new angle to widen the discussion by noting the following which you may be well aware of.

Rsync is well known for its delta-transfer algorithm. It copies only the differences between the source files and the existing files in the destination. In other words, only the changed paragraphs within, but not the whole book again, so to speak. (It was developed by an Australian working at Australian National University).

But Rsync needs to:
i) first do the comparison of each file, between source and destination,
ii) copy the parts over if changed, and finally,
iii) take a new snapshot of the copied parts (for comparison in later run).

On top of the above, SSH encryption also adds time to the whole process. Together, it makes Rsync slow.

I used Rsync for several years to back up my NAS contents (files are encrypted, prior to backup at each end, for privacy). It's run over an OpenVPN link to my friend's NAS. This way, we save money as well as having total control of the process by leveraging each other's NAS as cloud storage. It was great. But lately, i received multiple 'Permission denied (13)' flags from Rsync runs due to various reasons. I would probably still use Rsync otherwise. Out of familiarity.

But facing with the 'Permission denied (13)' issue forced me to re-evaluate my needs for Rsync.

Rsync was certainly most suitable when a limited down/upload allowance per month was set by ISP. With unlimited down/upload plan where i live nowadays, Rsync offers less advantage than other backup such as Syncovery.

Wishing you a good day.

_________________
Life is a journey; travel alone makes it less enjoyable and lonely.
ArjenR49
DD-WRT Guru


Joined: 05 Oct 2008
Posts: 666
Location: Helsinki, Finland / nr. Alkmaar, Netherlands

PostPosted: Wed Sep 28, 2022 9:19    Post subject: Reply with quote
Interesting to learn of an alternative. It is new to me and I need to look into it. I did put a lot of time into my backup server here. All cobbled together myself many years ago. A pool of two BTRFS drives on a Raspberry Pi3B+ with a UPS to keep it safe against blackouts when I am away.
The other end is a Netgear NAS.

I sorted out the permission errors long ago. Your friend may have changed something on his NAS along the way. In a commercial NAS it is sometimes just a guess what exactly will happen if you change something. What is the best way to set owners and permissions for the shares? Slowly you find out more.

The files I am backing up currently are Raspberry Pi disk backup images. Over 2 GB. Some are way over. And totally new every time as far as rsync is concerned. It is just my upload speed at this end which is slow, about 1 Mb/s. I do use the --partial option so if something goes wrong the transfer does not start from 0.

Anyway, the speed is not the problem. It is just the way it is. My connection is on a compound with 200 cottages. People do not want to pay for a faster connection as it is secondary, this being only a holiday residence for most. Download is less than 20 Mb/s, but it is enough to stream TV; during the pandemic a handfull of people started working from here and asking for higher speeds.


My original question still stands, even though on the one hand it does look like the bulk of rsync traffic is not handled on the designated port. However, on the other hand, to account for this there is no udp connection in the router connections table in the range 54-65535, which is where the bulk of packets are counted by QoS.

That elusive UDP port is likely a random one. The connection opened by the other, remote, site. If that is so, does it mean it will not show up in the active connections table?

Rsync documentation only speaks about how to make it use ssh over a non-standard port, not that the bulk of data will go another path ...

(Temporarily I have to work on a wrong )US= keyboard which is minute and often makes the cursor suddenly jump somewhere else. Bits may end up in the wrong places in the text, plus I cannot find the apostroph Sad
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 6410
Location: UK, London, just across the river..

PostPosted: Wed Sep 28, 2022 11:10    Post subject: Reply with quote
while in cli mode via ssh try this command
tcpdump -nnS -i eth0
or any interface the flow comes out
could do with netstat -np

_________________
Atheros
TP-Link WR740Nv1 ---DD-WRT 55179 WAP
TP-Link WR1043NDv2 -DD-WRT 55303 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55460 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55460 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55363 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
ArjenR49
DD-WRT Guru


Joined: 05 Oct 2008
Posts: 666
Location: Helsinki, Finland / nr. Alkmaar, Netherlands

PostPosted: Wed Sep 28, 2022 11:54    Post subject: Reply with quote
While the data transfer is chugging along, these are two 'random' subsequent lines from the command you proposed:
13:31:55.179795 IP 10.10.aa.bbb.qqqqq > 82.181.xx.yy.ppppp: Flags [P.], seq 2461993636:2461995084, ack 3511820633, win 735, options [nop,nop,TS val 779027328 ecr 1599184326], length 1448
13:31:55.192892 IP 82.181.xx.yy.ppppp > 10.10.aa.bbb.qqqqq: Flags [.], ack 2461869108, win 23320, options [nop,nop,TS val 1599184328 ecr 779026472], length 0

IP 10.10.aa.bbb is my 'WAN' address. I am behind a corporate firewall ...
IP 82.181.xx.yy is the WAN address at the remote end.

port qqqqq is a random number which I have not seen before.

port ppppp is the port rsync is supposed to use with ssh; it is the port assigned in the command.

Upload data going out in packets of 1448 bytes and being acknowledged by packets of 0 length.
Is that correct?

So ... because of the extra firewall and NAT, a different port qqqqq is used in between router and my (bridged) cable modem.
Correct?

(still on the awful keyboard (now set to US int), so some stray characters may have been entered)

Edit:
Your suggestion came right in time! The backup finished a few minutes after I tried the tcpdump command adding | grep ppppp with the newly found pipe symbol.

The US int. keyboard setting lacks Scandinavian characters but has the pipe symbol; this small keyboard doesn't have all the physical keys needed. A proper Scandinavian keyboard is on order; my laptop's keyboard recently became more and more unreliable ...
Alozaros
DD-WRT Guru


Joined: 16 Nov 2015
Posts: 6410
Location: UK, London, just across the river..

PostPosted: Wed Sep 28, 2022 13:36    Post subject: Reply with quote
yep after the IP is the port used the thing you need to.. i guess
Laughing source and destination ports could be different unless specified
for example port 22 destination can connect to random port client side

_________________
Atheros
TP-Link WR740Nv1 ---DD-WRT 55179 WAP
TP-Link WR1043NDv2 -DD-WRT 55303 Gateway/DoT,Forced DNS,Ad-Block,Firewall,x4VLAN,VPN
TP-Link WR1043NDv2 -Gargoyle OS 1.15.x AP,DNS,QoS,Quotas
Qualcomm-Atheros
Netgear XR500 --DD-WRT 55460 Gateway/DoH,Forced DNS,AP Isolation,4VLAN,Ad-Block,Firewall,Vanilla
Netgear R7800 --DD-WRT 55460 Gateway/DoT,AD-Block,Forced DNS,AP&Net Isolation,x3VLAN,Firewall,Vanilla
Netgear R9000 --DD-WRT 55363 Gateway/DoT,AD-Block,AP Isolation,Firewall,Forced DNS,x2VLAN,Vanilla
Broadcom
Netgear R7000 --DD-WRT 55460 Gateway/SmartDNS/DoH,AD-Block,Firewall,Forced DNS,x3VLAN,VPN
NOT USING 5Ghz ANYWHERE
------------------------------------------------------
Stubby DNS over TLS I DNSCrypt v2 by mac913
ArjenR49
DD-WRT Guru


Joined: 05 Oct 2008
Posts: 666
Location: Helsinki, Finland / nr. Alkmaar, Netherlands

PostPosted: Wed Sep 28, 2022 14:02    Post subject: Reply with quote
It is probably no use setting a QoS priority for the assigned port ...

Since the actual data connection seems to be created by the NAS at the other end, it is unlikely that it could somehow inherit the priority level from the setting in my router at this end for the port assigned in the rsync command ...

but then what do I know?

Anyway, it looks like my idea failed Sad better forget it.
Thanks for helping along.
DWCruiser
DD-WRT User


Joined: 15 Aug 2016
Posts: 223
Location: Melbourne, Australia

PostPosted: Thu Sep 29, 2022 4:23    Post subject: Reply with quote
ArjenR49 wrote:
Interesting to learn of an alternative. It is new to me and I need to look into it. I did put a lot of time into my backup server here. All cobbled together myself many years ago. A pool of two BTRFS drives on a Raspberry Pi3B+ with a UPS to keep it safe against blackouts when I am away.
The other end is a Netgear NAS.

I built a NAS (8-disk box) running TrueNAS, whereas the friend's one is a Synology DS 1815+. I set up and configured them both.

Quote:
Your friend may have changed something on his NAS along the way. In a commercial NAS it is sometimes just a guess what exactly will happen if you change something. What is the best way to set owners and permissions for the shares? Slowly you find out more.

You're right. I changed the permission settings on my NAS box at one stage. And the Rsync CLI that i run on Windows started throwing up permission errors since, no matter what i tried. Not least are the inherent conflicts in permissions b/w TrueNAS and Windows as i found out afterwards. Using built-in Rsync on Synology box would, on the other hand, send our traffic via a Synology server. Not a good point in our desire to be totally independent. Hence our Rsync was run via CygWin on Windows.

In the end, too many variables on different platforms in my settings create more potential failure points. A bad design on my part. I lost a few hairs in the process. Mind you.

Quote:
The files I am backing up currently are Raspberry Pi disk backup images. Over 2 GB. Some are way over. And totally new every time as far as rsync is concerned. It is just my upload speed at this end which is slow, about 1 Mb/s. I do use the --partial option so if something goes wrong the transfer does not start from 0.

Anyway, the speed is not the problem. It is just the way it is. My connection is on a compound with 200 cottages. People do not want to pay for a faster connection as it is secondary, this being only a holiday residence for most. Download is less than 20 Mb/s, but it is enough to stream TV; during the pandemic a handfull of people started working from here and asking for higher speeds.

Thanks for sharing that.

We've actually been using Syncovery to backup encrypted PC files to our own NAS, prior to cloud backup jobs, using Rsync. But now, we also use Syncovery for cloud backup as well. Syncovery gives a great visual aid to what's going to happen as a result of the backup everytime. Selection of what to backup is much easier using its GUI.

Quote:
My original question still stands, even though on the one hand it does look like the bulk of rsync traffic is not handled on the designated port. However, on the other hand, to account for this there is no udp connection in the router connections table in the range 54-65535, which is where the bulk of packets are counted by QoS.

That elusive UDP port is likely a random one. The connection opened by the other, remote, site. If that is so, does it mean it will not show up in the active connections table?

Rsync documentation only speaks about how to make it use ssh over a non-standard port, not that the bulk of data will go another path ...

(Temporarily I have to work on a wrong )US= keyboard which is minute and often makes the cursor suddenly jump somewhere else. Bits may end up in the wrong places in the text, plus I cannot find the apostroph Sad

Sorry for not having offered any real help for your issues. I recall some encrypted traffic neutralises MikroTik's QoS settings. But you're talking about DD-WRT here. But, hopefully, you got the jumpy Smile keyboard sorted out.

_________________
Life is a journey; travel alone makes it less enjoyable and lonely.
ArjenR49
DD-WRT Guru


Joined: 05 Oct 2008
Posts: 666
Location: Helsinki, Finland / nr. Alkmaar, Netherlands

PostPosted: Thu Sep 29, 2022 10:46    Post subject: Reply with quote
DWCruiser wrote:

You're right. I changed the permission settings on my NAS box at one stage. And the Rsync CLI that i run on Windows started throwing up permission errors since, no matter what i tried. Not least are the inherent conflicts in permissions b/w TrueNAS and Windows as i found out afterwards. Using built-in Rsync on Synology box would, on the other hand, send our traffic via a Synology server. Not a good point in our desire to be totally independent. Hence our Rsync was run via CygWin on Windows.


I found it helpful to arrange SFTP access to the Netgear NAS through FileZilla. It shows me the file permissions and owner info for each file. It allowed me to find out what the purpose is of the Netgear ReadyNAS OS 'resetting' file permissions and study relevant Linux commands. And doing away with the permission errors.

The keyboard is still throwing the cursor around and I don't know why. The only thing I was able to improve on it, was installing US keyboards to my laptop in addition to the Finnish one I need.

I grew up in the Netherlands, but have lived in Helsinki for many decades. In my home country they generally sell and use US keyboards, even though that is odd and missing diacritics. It is because of the hobby that owning a PC once was.
I once needed an emergency keyboard for my Raspberry Pis and got a small chinese Rii keyboard so that is what I have to use now until I get the Finnish one I ordered here in the Netherlands. It turned out that nowadays you can actually order a keyboard for another language in some Dutch shops. It was just about time, as Europe is such a blanket of patches(?) and these days students often study outside their own country. If you are from Scandinavia you just need a corresponding keyboard.

New problem:
it now looks like the internet provider has taken to seriously limiting my speeds .... after I ran my very long backup uploads. As usual there are no proper specifications whatsoever about the services. Clients are treated as nitwits and client service is on the same level. You do not get to the core, speak with somebody who knows what they are talking about.
The janitor here knows nothing although the network gear is on his attic.
Maybe speeds go back up after some time ...

Unless it is a DD-WRT problem after setting up the router from scratch ...

I also use BackInTime and Timeshift. Backintime stores user files in such a way that you can even find a file without the BackInTime application and it keeps previous situations/copies using a Linux feature.
Timeshift allows simple OS recovery from an update or install that went wrong. I have occasionally needed it.

Last year I set up an off-site backup system for a friend using two different commercial NASes. Synology and Netgear. In addition I had him return the modem/router provided by the network provider and buy a more decent piece of gear. It was a lot of work ... I would not do it again ...
DWCruiser
DD-WRT User


Joined: 15 Aug 2016
Posts: 223
Location: Melbourne, Australia

PostPosted: Thu Sep 29, 2022 11:30    Post subject: Reply with quote
Shooting from the dark at your keyboard issue.

My native language is not English. So when i need to write in my mother's language, i turn on a small software app that allows me to use certain keys for diacritical signs.

When i get back to write in normal English, the keyboard behaves unexpectedly in some random fashion. It took me a while to realize the conflict of leaving that app on when not needed. So i simply turn off that app when not in use.

Not sure if this may be relevant in your case.
_________________

Thanks for the suggestions with different apps re backup. I think i may try next time when i come to an issue with backup again. For now, i want to leave it as is.

Cheers.

_________________
Life is a journey; travel alone makes it less enjoyable and lonely.
Display posts from previous:    Page 1 of 1
Post new topic   Reply to topic    DD-WRT Forum Index -> Atheros WiSOC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum