[TESTING] Public IP used instead local IP as source on oet1

Post new topic   Reply to topic    DD-WRT Forum Index -> Advanced Networking
Goto page 1, 2  Next
Author Message
LaimisV
DD-WRT User


Joined: 01 Mar 2016
Posts: 63

PostPosted: Thu Sep 01, 2022 9:47    Post subject: [TESTING] Public IP used instead local IP as source on oet1 Reply with quote
Let me know in DM if I can somehow help DD-WRT contributors/community in response of solved issue.

----

This is happening on R7800 router.

Why I've got public IP calling local IP on WG oet1? Does it look right to you? Exactly those packets don't reach remote WG endpoint. Other packets work fine.

Please see tcpdump below for "7.X.X.110 > 192.168.250.111"
Packet arrives from from a local network.

Is there any way to replace source IP from "7.X.X.110 > 192.168.250.111" to "192.168.240.111 > 192.168.250.111" ?

Any work around?

Code:

root@dd-2:~# ifconfig oet1
oet1      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:10.1.240.1  P-t-P:10.1.240.1  Mask:255.255.255.0
...

root@dd-2:~# ip r
default via 7.X.X.254 dev eth0
10.1.220.1 via 10.1.240.1 dev oet1
10.1.230.1 via 10.1.240.1 dev oet1
10.1.240.0/24 dev oet1 scope link  src 10.1.240.1
10.1.250.1 via 10.1.240.1 dev oet1
7.X.X.0/22 dev eth0 scope link  src 7.X.X.110
127.0.0.0/8 dev lo scope link
192.168.220.0/24 dev oet1 scope link
192.168.230.0/24 dev oet1 scope link
192.168.240.0/24 dev br0 scope link  src 192.168.240.240
192.168.250.0/24 dev oet1 scope link
root@dd-2:~#

root@dd-2:~# sudo tcpdump -i oet1 -n | grep -e "IP 7.X.X.110"
...
03:31:42.166816 IP 7.X.X.110 > 192.168.250.111: IP 10.233.66.0.33409 > 10.233.100.219.8443: Flags [S], seq 1921032154, win 65495, options [mss 65495,sackOK,TS val 388916867 ecr 0,nop,wscale 7], length 0


A note that issue happens occasionally. When it happens, I have to restart this DD-WRT router (192.168.240.240) or restart sender (192.168.240.111). Sometimes multiple restarts are needed or keeping any of those two units offline for ~10 minutes. It helps, but I don't know why.

Firmware: DD-WRT v3.0-r41813 std (12/29/19)

Wireguard version v1.0.20191226

Many thanks!


Last edited by LaimisV on Mon Oct 17, 2022 18:35; edited 2 times in total
Sponsor
egc
DD-WRT Guru


Joined: 18 Mar 2014
Posts: 12812
Location: Netherlands

PostPosted: Thu Sep 01, 2022 9:53    Post subject: Reply with quote
You have a very old build and a lot of things have changed

Upgrade to the latest build 50012, *after* upgrade reset to defaults and put settings in manually, never restore from a backup (to a different build that is)

WireGuard documentation:
https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=327397

Not sure what your setup is, are you using WireGuard to access your home net work from the internet?

_________________
Routers:Netgear R7000, R6400v1, R6400v2, EA6900 (XvortexCFE), E2000, E1200v1, WRT54GS v1.
Install guide R6400v2, R6700v3,XR300:https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=316399
Install guide R7800/XR500: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=320614
Forum Guide Lines (important read):https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=324087
LaimisV
DD-WRT User


Joined: 01 Mar 2016
Posts: 63

PostPosted: Thu Sep 01, 2022 10:31    Post subject: Reply with quote
I have three R7800 that connect different locations via Wireguard.

Those locations are remote. So doing an upgrade means traveling. Probably multiple times until it got stable.

It would be great firstly to try something remotely in existing firmware (I always test each action with "command; sleep 5m; reboot"). So unless we know that there is non trivial chance to solve an issue I can do an upgrade.

----

I believe I had same issue with OpenVPN, before migration to WG.

Successful flow is this:

192.168.230.111 (sender) -> 192.168.230.230 (DD-3) -> source address on oet1 is 192.168.230.111 -> public IP -> internet -> another public IP -> 192.168.250.250 (DD-1) -> 192.168.250.111. So it reached last IP via WG successfully.

Unsuccessful flow is this:

192.168.240.111 (sender) -> 192.168.240.240 (DD-2) -> source address on oet1 is PUBLIC IP NOT 192.168.240.111 AS EXPECTED -> public IP -> internet -> another public IP -> 192.168.250.250 (DD-1) -> 192.168.250.111. So packet fails without even reaching an internet out. Locally on DD-2.

----

It is more complicated than this. 192.168.230.111, 192.168.240.111 , 192.168.250.111 can always communicate to each other successfully via WG (over internet), but the issue happens only with specific packets initiated on between three guys (Calico Kubernetes networking based on IPIP). So I'm using IPIP (Calico) in IPIP (Wireguard). It's working fine, but sometimes DD-WRT decides to use public IP on oet1.
You may can skip this part to avoid complexity. I don't think issue is in servers. It is DD issue based on tcpdump, etc.
LaimisV
DD-WRT User


Joined: 01 Mar 2016
Posts: 63

PostPosted: Thu Sep 01, 2022 11:04    Post subject: Reply with quote
How it looks on tcpdump

Successful flow

Code:
era@master-3:~$ host master-3
master-3 has address 192.168.230.111

era@master-3:~$ ping 10.233.79.26 -s 1300 -c 1
PING 10.233.79.26 (10.233.79.26) 1300(1328) bytes of data.
1308 bytes from 10.233.79.26: icmp_seq=1 ttl=63 time=7.68 ms

era@master-3:~$ sudo tcpdump -i any -n | grep ' 1308$'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
11:53:02.965152 IP 10.233.77.0 > 10.233.79.26: ICMP echo request, id 1245, seq 1, length 1308
11:53:02.972655 IP 10.233.79.26 > 10.233.77.0: ICMP echo reply, id 1245, seq 1, length 1308


# dd-3 is 192.168.230.230
root@dd-3:~# sudo tcpdump -i any proto 4 -n | grep ' 1308$'
13:53:02.962650 IP 192.168.230.111 > 192.168.250.111: IP 10.233.77.0 > 10.233.79.26: ICMP echo request, id 1245, seq 1, length 1308
13:53:02.962665 IP 192.168.230.111 > 192.168.250.111: IP 10.233.77.0 > 10.233.79.26: ICMP echo request, id 1245, seq 1, length 1308
13:53:02.962685 IP 192.168.230.111 > 192.168.250.111: IP 10.233.77.0 > 10.233.79.26: ICMP echo request, id 1245, seq 1, length 1308
13:53:02.969522 IP 192.168.250.111 > 192.168.230.111: IP 10.233.79.26 > 10.233.77.0: ICMP echo reply, id 1245, seq 1, length 1308
13:53:02.969542 IP 192.168.250.111 > 192.168.230.111: IP 10.233.79.26 > 10.233.77.0: ICMP echo reply, id 1245, seq 1, length 1308
13:53:02.969549 IP 192.168.250.111 > 192.168.230.111: IP 10.233.79.26 > 10.233.77.0: ICMP echo reply, id 1245, seq 1, length 1308
LaimisV
DD-WRT User


Joined: 01 Mar 2016
Posts: 63

PostPosted: Thu Sep 01, 2022 11:04    Post subject: Reply with quote
Unsuccessful flow

Code:
era@master-2:~$ host master-2
master-2 has address 192.168.240.111

era@master-2:~$ ping 10.233.79.26 -s 1300 -c 1
PING 10.233.79.26 (10.233.79.26) 1300(1328) bytes of data.

--- 10.233.79.26 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

era@master-2:~$ sudo tcpdump -i any -n | grep ' 1308$'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
11:57:04.381414 IP 10.233.66.0 > 10.233.79.26: ICMP echo request, id 10356, seq 1, length 1308
# request looks good, but there is no response

# dd-2 is 192.168.240.240
root@dd-2:~# sudo tcpdump -i any proto 4 -n | grep ' 1308$'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), snapshot length 262144 bytes
13:57:04.372490 IP 192.168.240.111 > 192.168.250.111: IP 10.233.66.0 > 10.233.79.26: ICMP echo request, id 10356, seq 1, length 1308
13:57:04.372490 IP 192.168.240.111 > 192.168.250.111: IP 10.233.66.0 > 10.233.79.26: ICMP echo request, id 10356, seq 1, length 1308
13:57:04.372574 IP 7.X.X.110 > 192.168.250.111: IP 10.233.66.0 > 10.233.79.26: ICMP echo request, id 10356, seq 1, length 1308
# request received in dd-2, but it weirdly tried to use 7.X.X.110 so this can be root cause why there is no response.


R7800 are configured in exactly same way, servers are configured in exactly same way. I ping same endpoint. One location can reach it, another cannot reach it. I'm sure all locations are working perfectly fine when this issue doesn't appear (it appears occasionally).

It's really hard issue, isn't? Neutral TBH, I have this issue for 2 years now. This is the only one that is unsolvable by me. If you can't go into details due to time consumption, any high view ideas where I should dive deep?

Thanks
egc
DD-WRT Guru


Joined: 18 Mar 2014
Posts: 12812
Location: Netherlands

PostPosted: Thu Sep 01, 2022 11:28    Post subject: Reply with quote
That build is really old and has security issues, like I said a lot has changed especially regarding WireGuard, It is now much easier to setup and you do not need any scripts etc, all can be handled via the GUI.

You have a multi site-to-site setup some examples (Mesh, hub and spoke) are covered in the advanced guide but it assumes you are running a recent build.

Unfortunately I cannot help you very well if you stay on this old build.

One tip (which is in the troubleshooting guide) disable SFE (shortcut forwarding engine on Setup page)

_________________
Routers:Netgear R7000, R6400v1, R6400v2, EA6900 (XvortexCFE), E2000, E1200v1, WRT54GS v1.
Install guide R6400v2, R6700v3,XR300:https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=316399
Install guide R7800/XR500: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=320614
Forum Guide Lines (important read):https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=324087
LaimisV
DD-WRT User


Joined: 01 Mar 2016
Posts: 63

PostPosted: Thu Sep 01, 2022 12:40    Post subject: Reply with quote
Thanks. I will consider an upgrade in a while. Security points are also interesting (even if I have custom dynamic-secret IP whitelist, non root SSH login + key using custom user, etc).

Your docs are really good, followed them and set up WG successfully in the beginning of this year.
Seems I missed a part about SFE. I had it set as sfe=1 , now I switched to zero.
Likely, it didn't help, because I had to restart DD-WRT twice to solve this oet1/publicIP issue.

This issue will repeat in around 2-10 weeks. It's hard to fix when this happens occasionally. Indeed, it happens rarely and hard to reproduce on demand.

If you ever need to explore off topics that I mentioned in my posts and they are useful, feel free to get in touch.

Until upgrade, I left this query on serverfault as well. Probably, it will exhaust without solutions - https://serverfault.com/questions/1109644/public-ip-address-used-instead-of-local-ip-as-source-address-on-wireguard-oet1-n?noredirect=1#comment1448995_1109644
egc
DD-WRT Guru


Joined: 18 Mar 2014
Posts: 12812
Location: Netherlands

PostPosted: Thu Sep 01, 2022 13:35    Post subject: Reply with quote
SFE can play routing tricks, too bad it did not help for you.
(see: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=329728)

WireGuard also has had bug fixes along the way so maybe an upgrade will solve it, hard to tell

_________________
Routers:Netgear R7000, R6400v1, R6400v2, EA6900 (XvortexCFE), E2000, E1200v1, WRT54GS v1.
Install guide R6400v2, R6700v3,XR300:https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=316399
Install guide R7800/XR500: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=320614
Forum Guide Lines (important read):https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=324087
LaimisV
DD-WRT User


Joined: 01 Mar 2016
Posts: 63

PostPosted: Thu Sep 01, 2022 14:19    Post subject: Reply with quote
That's good reading. Now I have basic understanding how SFE works. I've checked https://forum.dd-wrt.com/wiki/index.php/Hardware#Flow_Acceleration.2C_SFE_and_Cut-Through_Forwarding as well. Let me gradually reboot all related servers and outstanding two routers without SFE, once I'm available.

Also I'll check tcpdump & netstat. If In understood correctly TCP/IP stack was somehow limited with SFE - that is why it performs better. I personally prefer to firstly ensure quality over performance.
egc
DD-WRT Guru


Joined: 18 Mar 2014
Posts: 12812
Location: Netherlands

PostPosted: Thu Sep 01, 2022 14:28    Post subject: Reply with quote
Indeed, besides the R7800 is powerful enough to do without SFE, I have seen some weird things happen with SFE also related to OpenVPN, it causes some hard to diagnose routing and also latency problems.
_________________
Routers:Netgear R7000, R6400v1, R6400v2, EA6900 (XvortexCFE), E2000, E1200v1, WRT54GS v1.
Install guide R6400v2, R6700v3,XR300:https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=316399
Install guide R7800/XR500: https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=320614
Forum Guide Lines (important read):https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=324087
LaimisV
DD-WRT User


Joined: 01 Mar 2016
Posts: 63

PostPosted: Thu Sep 01, 2022 14:38    Post subject: Reply with quote
Intuitively, I believe, frequency of this issue was reduced, if not completely disappeared, when I had backups turned off in our servers for 3-4 months. Those backups were running very frequently and generated traffic and connections through WG links. Backups are turned on again few months ago. The issue is back with its frequency.

I've checked maximum connections value on DD, it is at low level, but maybe there is something else related to load and because of this DD goes crazy with this issue.

A note that CPU is at normal levels, let's say average is at 10-30% on routers.

Also the fact that I have to turn off the server or dd for 10 minutes or do multiple restarts to temporary fix an issue, shows that maybe entries in layer 2/layer 3 have to be fully cleared to start fresh without this issue.

BTW, any of three R7800 is not free of this issue. Issue appears in any of them, randomly.
Per Yngve Berg
DD-WRT Guru


Joined: 13 Aug 2013
Posts: 6855
Location: Romerike, Norway

PostPosted: Thu Sep 01, 2022 15:38    Post subject: Reply with quote
Have you checked for a NAT rule on DD-2?
LaimisV
DD-WRT User


Joined: 01 Mar 2016
Posts: 63

PostPosted: Thu Sep 01, 2022 18:09    Post subject: Reply with quote
Per Yngve Berg wrote:
Have you checked for a NAT rule on DD-2?


Code:

root@dd-2:~# iptables -t nat -L PREROUTING -n -v
Chain PREROUTING (policy ACCEPT 41067 packets, 2503K bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DNAT       tcp  --  *      *       8.x.x.4/31           7.X.X.110       tcp dpt:22 to:192.168.240.240:22
 1714 84650 DNAT       icmp --  *      *       0.0.0.0/0            7.X.X.110       to:192.168.240.240
  168  8664 DNAT       tcp  --  *      *       0.0.0.0/0            7.X.X.110       tcp dpt:80 to:192.168.240.112:80
  665 36056 DNAT       tcp  --  *      *       0.0.0.0/0            7.X.X.110       tcp dpt:443 to:192.168.240.112:443
    2    88 DNAT       tcp  --  *      *       0.0.0.0/0            7.X.X.110       tcp dpt:6443 to:192.168.240.111:6443
   14   604 DNAT       tcp  --  *      *       0.0.0.0/0            7.X.X.110       tcp dpt:8080 to:192.168.240.110:8080
    3   214 DNAT       udp  --  *      *       0.0.0.0/0            7.X.X.110       udp dpt:8080 to:192.168.240.110:8080
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            7.X.X.110       tcp dpt:9443 to:192.168.240.110:9443
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            7.X.X.110       udp dpt:9443 to:192.168.240.110:9443
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            7.X.X.110       tcp dpt:7999 to:192.168.240.110:7999
    0     0 DNAT       udp  --  *      *       0.0.0.0/0            7.X.X.110       udp dpt:7999 to:192.168.240.110:7999
 2297  116K TRIGGER    0    --  *      *       0.0.0.0/0            7.X.X.110       TRIGGER type:dnat match:0 relate:0
root@dd-2:~# iptables -t nat -L POSTROUTING -n -v
Chain POSTROUTING (policy ACCEPT 36650 packets, 2248K bytes)
 pkts bytes target     prot opt in     out     source               destination
 2948  184K SNAT       0    --  *      eth0    192.168.240.0/24     0.0.0.0/0           to:7.X.X.110
    0     0 SNAT       0    --  *      eth0    10.1.240.0/24        0.0.0.0/0           to:7.X.X.110
    0     0 MASQUERADE  0    --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x80000000/0x80000000
    0     0 MASQUERADE  0    --  *      *       10.0.240.0/24        0.0.0.0/0
root@dd-2:~#


As usual I've hidden public IP.

A note that Kubernetes Calico networking (let's say app) requirement is not NATed Wireguard. It should see local IPs as they are. So NAT was removed two years ago.

Also attached oet1 interface settings. If anything is related with this source address issue, I can play with those values. Eg never tried to disable "Masquerade / NAT" in Networking section for oet1.
LaimisV
DD-WRT User


Joined: 01 Mar 2016
Posts: 63

PostPosted: Fri Sep 02, 2022 16:23    Post subject: Reply with quote
I put more efforts on it last days, as you tried to help me.

So I managed to upgrade reported router dd-2 (192.168.240.240) to DD-WRT 50012 08/31/22.

To early to speak, will let you know how it is working in a while.
LaimisV
DD-WRT User


Joined: 01 Mar 2016
Posts: 63

PostPosted: Wed Oct 05, 2022 22:46    Post subject: Reply with quote
Ok, after some time using build 50012 on all three R7800 routers:

1) The issue that I reported on oet1 (public IP -> local IP incorrect call) - seems didn't happen. That's promising.

2) Random router reboot in approx. 10-20 days.

3) If I access router via SSH and check commands like `ip`, `wg`, it reboots after 5-30 minutes. So it can reboot multiple times in a day. I'm mostly using OpenSSH. I've switched to default Dropbear to see how it goes.

4) WiFi disconnects depending on client computer, every 30-60 minutes. Eg two WiFi clients work fine, one has this issue.

5) When I connected additional WG clients with Macbooks (probably, OS doesn't matter), I've noticed WG disconnections "latest handshake" for around 5-10 minutes every few hours. Disconnection happens between R7800 units, not clients such as Macs. So I removed those computers from WG tunnels and it fixed an issue.

I know that it could be just my configuration, but in r41813, I din't have those additional issues 2-5.

If you have any advice what to do, how to debug precisely or how to select another firmware version for stability, that would be great.
Goto page 1, 2  Next Display posts from previous:    Page 1 of 2
Post new topic   Reply to topic    DD-WRT Forum Index -> Advanced Networking All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum