Put an additional entry in the GUI for "Link MTU" and make it so that you can EITHER put in a figure for "Tunnel MTU" or "Link MTU" but not both - that is, if you put in a figure for Tunnel MTU it forces the Link MTU box to be blank - if you put in a figure for Link MTU it forces the Tunnel MTU to be blank. tun-mtu and link-mtu settings are mutually exclusive - OpenVPN will not even start if you supply both of them - if you supply one then OpenVPN calculates the other. In "the normal case" OpenVPN is supposed to KNOW the link-mtu and calculates tun-mtu. But in our K3xx and above kernels, it doesn't know it.
Set the default so that Tunnel MTU is blank, and Link MTU has a default of 1472. This is 1500 subtracting the 28 bytes for the packet header. link-mtu is the maximum size of the packet after encapsulation minus the headers. This lets OpenVPN calculate the correct tun-mtu for whatever cipher and so on is in use.
The process illustrated above which tries to calculate the correct tun-mtu is doing it backwards only because we don't have a way to set the link-mtu in dd-wrt and it's forcing a tun-mtu into the openvpn config.
I set my tunnel MTU to 1350 and this is what I get in the logs:
20210512 00:11:31 W NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
20210512 00:11:31 W WARNING: normally if you use --mssfix and/or --fragment you should also set --tun-mtu 1500 (currently it is 1350)
20210512 00:11:31 Control Channel MTU parms [ L:1472 D:1212 EF:38 EB:0 ET:0 EL:3 ]
20210512 00:11:31 Data Channel MTU parms [ L:1472 D:1450 EF:122 EB:381 ET:0 EL:3 ]
20210512 00:11:31 Local Options String (VER=V4): 'V4 dev-type tun link-mtu 1400 tun-mtu 1350 proto UDPv4 comp-lzo cipher AES-128-GCM auth [null-digest] keysize 128 key-method 2 tls-client'
20210512 00:11:31 Expected Remote Options String (VER=V4): 'V4 dev-type tun link-mtu 1400 tun-mtu 1350 proto UDPv4 comp-lzo cipher AES-128-GCM auth [null-digest] keysize 128 key-method 2 tls-server'
Notice the different mssfix entries for the Data and Control channels, there's a lot of overhead OpenVPN is stuffing in there.
Now, for the testing through the VPN on this setup:
From a main net FreeBSD system to a remote Linux server:
I can ping a 4000 byte packet though the VPN and I get a 4000 byte response which is indicating that packet fragmentation and reassembly is properly taking place.
I can sweep sizes every 10th of a second with this command:
ping -G 2000 -g 1000 -i .1 172.16.100.2
And I get exactly 2000 bytes back on every single entry once more indicating packet fragmentation and reassembly is correctly taking place
If I sweep the range with packets sized in a range of 1400 to 1480 with the DF set I get packets sized up to 1330 through fine, packets above that I get "frag needed and DF set" and packets above 1472 I get "message too long. And, there are no black holes
From a remote Linux server on the client net to a main net FreeBSD system:
ping -S 4000 172.16.1.9
gives me 4008 bytes back indicating frag and reassembly is working OK even if the VPN is tacking something extra on the end.
And the while loop sweep:
while ping -s $size -c1 -M do 172.16.1.1 || true ;
do ((size+=1)); done
The packets go though until they get to a size of 1323 then I get "message too long" up through 1500 MTU once more with no black holes.
And, my phones are staying connected. They don't care if their registration packets get fragmented they just care if they get trashed or if there's a black hole.
Unless and until my proposal gets implemented, this is my fix.
Just for my understanding a recap, correct me if I am wrong:
PMTUD on OpenVPN seems broken therefore the default tun-MTU value of 1500 is not adequately lowered because of a blackhole situation.
No. There is no evidence that PMTUD is broken other than because it will break under the garbage-in, garbage-out principle. What the evidence is showing is that OpenVPN is unable to obtain the correct MTU for interfaces on the dd-wrt router, so it is starting out with a ridiculously high MTU of 1550 or something. Since link-mtu and tun-mtu are all derived from the physical MTU on the interfaces, when it can't get the correct physical MTU it calculates a way too high link-MTU value.
If OpenVPN were to get the correct physical MTU of 1500 it would calculate a link-mtu of 1472 and if the user then set an impossibly high tun-mtu of 1500 this just tells OpenVPN it's OK to frag the incoming packet. OpenVPN knows it's impossible to use a tun-mtu of 1500 since there's not enough room for the packet so it just throws out the 1500 and ignores it. That's why tun-mtu has to be set at 1500 if you want to mess around with mss settings, it's a way to keep people from shooting themselves in the foot. Only if tun-mtu is lower than the calculated tun-mtu does OpenVPN actually lower it - which then adds unused space in the packet.
egc wrote:
So it is better to lower the default tun-mtu value to a more realistic size.
In my opinion tun-mtu should have never been brought out as a knob to twaddle with without also bringing out link-mtu. tun-mtu has to be brought out to be set to 1500 to allow the mss frag settings to be twiddled with but by default, tun-mtu and those mss settings need to be blank and empty and link-mtu should have been defaulted at 1472.
egc wrote:
The problem manifests itself when using UDP, packets should not be fragmented so we need the maximum packetsize which does not fragment the packets (maximum because that gives the highest throughput).
No. There is nothing inherent about a UDP packet that makes it wrong for a router to fragment it. An application MAY decide for whatever reason that it does not want it's UDP packet fragmented but logically if an app developer is going to put that kind of restriction on a UDP packet then they are going to have to implement some sort of path-discovery themselves - such as sending initial UDP packets of different sizes with DF set and seeing what works and what gets an ICMP response.
egc wrote:
In daily usage this does not seem like a problem as most applications factor this in and I have not seen a packetsize over 1350, but applications using UDP (like VoIP, streaming media) which just try to use the mtu do not work as these are fragmented.
No. Most applications use TCP and the stacks have black-hole-path-discovery and so will work over whatever broken and borked up tunnel OpenVPN provides. Also, they will check for trashed packets (invalid checksums and such) that OpenVPN is trashing and just reject them and resend with a smaller size.
The UDP apps are having problems because OpenVPN believes the interface it's forwarding it's encrypted packet out is larger than it is. So the large packets these apps send through get destroyed and the receiver drops them because the checksums are wrong.
egc wrote:
So how about a tun-mtu value of 1420 for a more realistic tun-mtu default value?
The correct way is to let OpenVPN decide the tun-mtu or if you have users insistent on setting this to provide the box to allow themselves to shoot themselves in the foot.
In a perfect world we would also default link-mtu to blank as well. But that is only possible when OpenVPN knows the actual MTU. Defaulting it to 1472 (or 1464 if PPPoE) is a quick hack to workaround the real problem.
Yes that will work. Let me know when you get it built and I'll test it. Of course they will have to add tun-mtu 1500 as an additional option if they want to play with the tunnel UDP MSS-Fix but that's OK.
You know, we don't actually know if OpenVPN is unable to determine the physical MTU on a PPPoE interface. I don't have such a connection to test with. I wonder if it DOES work over PPPoE without setting the link MTU?
Joined: 04 Aug 2018 Posts: 1447 Location: Appalachian mountains, USA
Posted: Wed May 12, 2021 14:26 Post subject:
tedm wrote:
There is only ONE "data channel mtu parms" line in the entire log and it is what I posted. There is no "data channel mtu parms" line _after_ the peer connection entry in the log on either router.
Well then, great! Apparently all this depends to some degree on other things not accounted for here.
But all that aside, I love where you've gone with this. I had begun to wonder whether perhaps we should just set link-mtu from the GUI as well, but I don't understand the procedure through which OpenVPN starts at tun-mtu=1500 and works from there to the correct number, so the alternatives were unclear to me. What's available through googling is such a chaotic mess of sloppy advice giving that I feel lucky to have even realized that a procedure exists. In any case, it certainly seems that you have far more experience in all of this than I. I'm just a networking beginner with a sometimes unjustified amount of persistence.
@egc: Maybe a "MTU" section with three buttons and a user-value window. Buttons "default", "set tun-mtu", and "set link-mtu" with only one chosen at a time. The two "set" choices start with the window empty until the user enters something. The "default" option might lock the window into a greyed-out (or not) entry like "(negotiated)" or "(automatic)" or something in that spirit. Or the "default" button should perhaps be labeled "automatic" and be selected by default.
I'm probably overthinking that. _________________ 2x Netgear XR500 and 3x Linksys WRT1900ACSv2 on 53544: VLANs, VAPs, NAS, station mode, OpenVPN client (AirVPN), wireguard server (AirVPN port forward) and clients (AzireVPN, AirVPN, private), 3 DNSCrypt providers via VPN.
Joined: 18 Mar 2014 Posts: 12884 Location: Netherlands
Posted: Wed May 12, 2021 17:48 Post subject:
Thanks Jeff that seems a bit overdone for now
Lets first see how link-mtu setting is going to work out.
This implementation will have one advantage, if you use 0 it will not set link-mtu, so you are free to set tun-mtu in Additional config and can experiment some more.
Will make a protopype this evening.
@tedm are you having a Netgear R7000 or Netear AC1450 to test?
I have a Linksys EA6200 (unfortunately that one only has 32k nvram so openvpn keeps losing it's config anytime the router reboots but if you want to do a build for it that strips out the extra hotspot stuff that clutters up the nvram I'd be all over that) a Netgear R6300v2 and a Netgear WNDR4500 at the client site and I am going to be there tonight through Thursday night.
On Friday I'll be back at the main site where I have 2 Netgear AC1450's and 1 Netgear R7000. The remote site is setup with 2 devices one in a backup configuration so if I remotely push a firmware load that the primary doesn't like, I can switch over to the backup without driving there.
And I have the usual 5v serial cable for debricking if needed.
Incidentally has there been any thought of producing a special openvpn version of dd-wrt that will fit all the certificates and such into 32k nvram? There's a few oddball routers in addition to the EA6200 that have respectable ram and flash and CPU but crippled 32k nvram and Mega and Big are defining around 29-30k of nvram variables at bootup in the current version. Once the router is up and running you can strip out all the blank variables and get enough nvram to fit the certificates in, but it seems that during reboots that is still an issue.
Joined: 18 Mar 2014 Posts: 12884 Location: Netherlands
Posted: Wed May 12, 2021 20:15 Post subject:
tedm wrote:
I have a Linksys EA6200 (unfortunately that one only has 32k nvram so openvpn keeps losing it's config anytime the router reboots but if you want to do a build for it that strips out the extra hotspot stuff that clutters up the nvram I'd be all over that) a Netgear R6300v2 and a Netgear WNDR4500 at the client site and I am going to be there tonight through Thursday night.
On Friday I'll be back at the main site where I have 2 Netgear AC1450's and 1 Netgear R7000. The remote site is setup with 2 devices one in a backup configuration so if I remotely push a firmware load that the primary doesn't like, I can switch over to the backup without driving there.
And I have the usual 5v serial cable for debricking if needed.
Incidentally has there been any thought of producing a special openvpn version of dd-wrt that will fit all the certificates and such into 32k nvram? There's a few oddball routers in addition to the EA6200 that have respectable ram and flash and CPU but crippled 32k nvram and Mega and Big are defining around 29-30k of nvram variables at bootup in the current version. Once the router is up and running you can strip out all the blank variables and get enough nvram to fit the certificates in, but it seems that during reboots that is still an issue.
Is the EA6200 a dual partition type linksys?
I have an EA6900 which had the same problem but fortunately an alternate CFE was available so that it could be converted to a single partition with 64K nvram.
I do not know if that is available for the EA6200 but perhaps worth researching.
There are builds for routers with 16 MB flash RAM (some Archer C9) although not directly related to the used nvram there is a relation.
Does the EA6200 uses the builds for the EA6300 if so I can see what I can do later.
I am building now so should be ready in an hour.
If it is any good I will build for the AC1450, R7000 and R6300v2 those are more or less the same and will send them to you tomorrow morning by PM.
I know my builds run on R6400, R6700v3, Archer C9 and Asus AC68U (@eibgrad has one and he did a lot of testing for OpenVPN).
So it must work for your routers too but you must be able to recover if anything goes wrong (and you are )
I had never thought someone might have done anything with it but sure enough there is a modded CFE out there for it that fixes the nvram issue. But I'll have to do some reading on that before applying that.
I also have an EA6400 and it has to be CFE modded to load dd-wrt on it. Maybe later this year I'll do that.
Joined: 18 Mar 2014 Posts: 12884 Location: Netherlands
Posted: Thu May 13, 2021 6:46 Post subject:
Will deal with the EA routers later, in the mean time builds are ready, it is a prototype so needs polishing but it does what needs to be done and that is setting the link-mtu and if link-mtu is 0 is does not set anything at all so you can use tun-mtu in additional config to play with.
However I slightly changed it, when using link-mtu = wan-mtu -28 you get a ridiculously low tun-mtu, maybe we must think backwards and add 28 with the wan-mtu at least that is what I did in this build and I think you can safely add the lowest amount needed for encryption so that the default link-mtu is wan-mtu + 28 + 38
If I use 1566 as link-mtu (I use chacha-poly) than it seems reasonable and no black hole
But still a lot of complaining and warnings from OpenVPN, explicitly warning for MTU problems
So I am still unsure what is the best way to deal with this vexing problem
Joined: 18 Mar 2014 Posts: 12884 Location: Netherlands
Posted: Thu May 13, 2021 10:34 Post subject:
I did test with AES-256-CBC/SHA512 which gives much more overhead (96) and OpenVPn seems to calculate a tun MTU size higher than for Chacha-Poly which has much lower overhead and of course it does not work.
So it seems that OpenVPN has no idea what it is doing (or maybe it is me, did not have my coffee yet)
Edit: Final conclusion is that OpenVPN is often miscalculating the mtu (tun-mtu and link-mtu).
The default OpenVPN tun-mtu is obviously too high and OpenVPN does not adequately lower it.
So builds from 46681 will use a tun-mtu of 1400, but you can set it to 0 which means it is not set so that you can let OpenVPN decide or use link-mtu to play with.
Perfect. So in summary there are FIVE approaches to setting the correct MTU:
1) Leave it at the default. This should allow for no lost packets TCP and may allow for no lost packets UDP given the broken calculations in OpenVPN and the cipher in use. Packets may be smaller than optimal depending on the cipher used. OpenVPN uses the tun-mtu of 1400 as a basis for (mis)calculating the other MTUs.
For the rest of the options it's helpful to set verb 4 in additional options as it increases log output. You might also want to set the sysctl mentioned below.
2) Set tun-MTU to 0. This will allow OpenVPN to probe the operating system for the MTU then calculate the different MTUs. While OpenVPN is currently doing this wrong (at least, on our DD-WRT kernels) and calculating a link MTU of around 1550 (way too high) apparently in most implementation there are no black holes, and if you are sending mainly TCP data with "ordinary" small ICMP and UDP data, the TCP/IP stacks will use PMTUD (https://en.wikipedia.org/wiki/Path_MTU_Discovery) to figure out the optimum packet to send across the link.
3) Tuning the tun-mtu through trial-and-error. I found 1350 tun-mtu seems to work for my environment. I recommend this for tricky cases because since OpenVPN mtu calculations are borked there's no telling what it's doing internally and you can only depend on the OpenVPN documentation describing what the OpenVPN developers WANT to happen not what IS happening. Changing ciphers will mean this figure needs to be changed.
For example on dd-wrt release r46640 with a tun of 1350 set and verb 4 and AES-128-CGM I get the following in the log:
20210515 21:47:07 W WARNING: normally if you use --mssfix and/or --fragment you should also set --tun-mtu 1500 (currently it is 1350)
20210515 21:47:07 Control Channel MTU parms [ L:1472 D:1212 EF:38 EB:0 ET:0 EL:3 ]
20210515 21:47:07 Data Channel MTU parms [ L:1472 D:1450 EF:122 EB:381 ET:0 EL:3 ]
20210515 21:47:07 Local Options String (VER=V4): 'V4 dev-type tun link-mtu 1400 tun-mtu 1350 proto UDPv4 comp-lzo cipher AES-128-GCM auth [null-digest] keysize 128 key-method 2 tls-client'
20210515 21:47:07 Expected Remote Options String (VER=V4): 'V4 dev-type tun link-mtu 1400 tun-mtu 1350 proto UDPv4 comp-lzo cipher AES-128-GCM auth [null-digest] keysize 128 key-method 2 tls-server'
this creates a link-mtu of 1472 which after the wrapper header is applied the packet will be 1500 bytes.
4) Setting the tun-mtu to 0 and setting link-mtu in the additional options. For a regular Ethernet router that has an MTU of 1500 on the interfaces, that would be setting link-mtu 1472 on the server side. OpenVPN does not consider the underlying interface's MTU as a starting value for its link-mtu/tun-mtu, nor does it perform PMTU discovery even when its mtu-disc option is set to yes
I have tried this in testing the beta you sent me and noted that even when link-mtu was set the same on both sides, the client and server calculated the other parameters differently and complained in the log.
5) Setting the tun-mtu to 1500 and setting the MSS. MSS is MTU - 40 so on our routers that would be MSS of thus set mssfix to 1460. Many documents out on the web recommending leaving the other options unspecified and just playing with fragment and mssfix settings. While this may optimize TCP traffic it may trash UDP traffic.
Note that in addition to this you can set these options on the server and allow the client to have autodiscover options and calculate them based on what the server is doing. You can also have asymmectric options for mssfix and the other parameters and OpenVPN may dynamically resize them.
"On Ubuntu 14.04 HWE and newer (or equivalent kernels up to v5.7.x) you will also have to set sysctl net.ipv4.ip_forward_use_pmtu=1 in order to have Linux perform said ICMP/fragmentation when forwarding other machines' traffic, whereas that additional sysctl is not required for outbound traffic originated by the server machine itself."
This applies to dd-wrt as well which has the pmtu sysctl set to 0. Unfortunately that sysctl is not brought out into the dd-wrt interface under Administration but can be set in the startup config
Note the warning in https://sysctl-explorer.net/net/ipv4/ip_forward_use_pmtu/
this is why that sysctl is defaulted to 0. However that does state to enable it if you have user-space software which tries to discover path mtus by itself and depends on the kernel honoring this information. And OpenVPN is user-space.
Some dd-wrt builds may lack the sysctl binary in that case: echo 1 > /proc/sys/net/ipv4/ip_forward_use_pmtu will work.
Lastly, you may want to raise the UDP timeout from 120 to 600
I have found that during the day that the amount of UDP packet loss on the general Internet (which my VPN traverses) is high enough to interfere with the Cisco IP phones SIP I have traversing my VPN link. The problem appears to be the worst from 2pm to 8pm - I'm on the West Coast so I assume what's causing it is people watching TV over the Internet. Interestingly, the Polycom phones are not affected.
Setting the OpenVPN to use tcp4 instead of udp4 fixes the problem. However that does require dropping the tun to 1348 from 1350 because using TCP as a transport adds 2 bytes more of overhead.
