Zyxx
DD-WRT Guru
Joined: 28 Dec 2018
Posts: 963
|
 Posted: Fri Oct 10, 2025 15:35 Post subject: iperf3 ipv6 upstream slow with NAT66, "rx-gro-list" |
 |
Hi,
over the last few releases I encountered one interesting behaviour of my main x86 router, powered by DD-WRT.
The IPv6 upstream throughput from my LAN was way too low, like almost my upstream divided by 100.
It is an industrial mini PC with two i226V RJ45 connectors, currently sporting the latest release r62268 / Linux 6.12.51 #273 SMP Thu Oct 9 04:29:04 +07 2025 x86_64.
I do not know what release was the first with this behaviour!
I have it setup to use PPPoE over GPON modem towards my ISP over one RJ45 (eth0), the line does provide 600/300mbit/s down/up.
The router gets an public IPv4 and public IPv6 from my ISP, the setting under Setup --> IPv6 is DHCPv6 with Prefix Delegation, the Prefix Length is 56.
It provides LAN access on its other RJ45 (eth1) towards my LAN.
DHCP for LAN is disabled, since another router of mine is doing this job.
I use NAT66 with IPv6 ULA fdXY::/64 for my whole network.
This is enabled in Administration --> Commands --> Firewall section:
| Code: |
ip6tables -t nat -I POSTROUTING -o ppp0 -j MASQUERADE
ip -6 addr add fdXY::f00/64 dev br0
|
So far so good, everything working.
My other DHCP/radvd router provides this prefix to all my devices in LAN/wifi, working like a charm.
BUT one thing was/is astray:
iperf3 upstream measurements stated ipv6 speed to be limited. Regardless if OS was android, Windows, Linux also Wired or not wireless no difference.
I exchanged cables, LAN switches and such, but to no avail.
Here one iperf on a linux powered 8 core ARM Machine of mine:
| Code: |
iperf3 -c edited -6
Connecting to host edited, port 5201
[ 5] local fd01:ab00:cd00:ef00:192:168:1:88 port 50628 connected to edited port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.23 MBytes 10.3 Mbits/sec 31 22.2 KBytes
[ 5] 1.00-2.00 sec 1.79 MBytes 15.0 Mbits/sec 3 19.4 KBytes
[ 5] 2.00-3.00 sec 183 KBytes 1.50 Mbits/sec 16 2.77 KBytes
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 10 2.77 KBytes
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 11 2.77 KBytes
[ 5] 5.00-6.00 sec 183 KBytes 1.50 Mbits/sec 10 2.77 KBytes
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 13 2.77 KBytes
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 11 1.39 KBytes
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 10 2.77 KBytes
[ 5] 9.00-10.00 sec 183 KBytes 1.50 Mbits/sec 10 2.77 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.56 MBytes 2.98 Mbits/sec 125 sender
[ 5] 0.00-10.01 sec 3.12 MBytes 2.62 Mbits/sec receiver
iperf Done.
|
Like really only ONE DIGIT Mbit/s figures while tested on any device in my network towards external ipv6 iperf servers.
Only when iperf tests are run on the X86 router this limit isnt there:
| Code: |
iperf3 -c edited -6
Connecting to host edited, port 5201
[ 5] local edited:e6ce:c006:1f67 port 48028 connected to edited port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 17.6 MBytes 148 Mbits/sec 0 632 KBytes
[ 5] 1.00-2.00 sec 18.8 MBytes 157 Mbits/sec 0 632 KBytes
[ 5] 2.00-3.00 sec 17.9 MBytes 150 Mbits/sec 0 632 KBytes
[ 5] 3.00-4.00 sec 18.0 MBytes 151 Mbits/sec 0 632 KBytes
[ 5] 4.00-5.00 sec 18.4 MBytes 154 Mbits/sec 0 632 KBytes
[ 5] 5.00-6.00 sec 18.5 MBytes 155 Mbits/sec 0 632 KBytes
[ 5] 6.00-7.00 sec 17.8 MBytes 149 Mbits/sec 0 632 KBytes
[ 5] 7.00-8.00 sec 18.4 MBytes 154 Mbits/sec 0 632 KBytes
[ 5] 8.00-9.00 sec 17.8 MBytes 149 Mbits/sec 0 632 KBytes
[ 5] 9.00-10.00 sec 17.6 MBytes 148 Mbits/sec 0 632 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 181 MBytes 151 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 181 MBytes 151 Mbits/sec receiver
iperf Done.
|
Two or more streams max out the upstream reliably on this device.
I double checked my settings, made sure mss and mtu are right and clamped, mangled, no faulty ip6tables entries etc.
Took me quite some time without any idea why.
Finally according to the internet people on openwrt had similar problems in the past.
The culprit was the feature "rx-gro-list" on the Ethernet nic.
So I went ahead, installed entware on this x86 release to get access to ethtool.
This is the result for the LAN facing i226v eth1:
| Code: |
root@N4000:~# ethtool -k eth1
Features for eth1:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: on
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: on
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
|
>>rx-gro-list: on <<
I issued:
| Code: | | ethtool -K eth1 rx-gro-list off |
and tested iperf3 again on multiple devices.
iperf3 performance was up and nice again, regardless wired, wifi or the used OS:
| Code: |
iperf3 -c edited -6
Connecting to host edited, port 5201
[ 5] local fd01:ab00:cd00:ef00:192:168:1:88 port 52422 connected to edited port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 39.1 MBytes 328 Mbits/sec 21 241 KBytes
[ 5] 1.00-2.00 sec 20.0 MBytes 168 Mbits/sec 0 297 KBytes
[ 5] 2.00-3.00 sec 23.8 MBytes 199 Mbits/sec 0 352 KBytes
[ 5] 3.00-4.00 sec 28.8 MBytes 241 Mbits/sec 0 409 KBytes
[ 5] 4.00-5.00 sec 33.8 MBytes 283 Mbits/sec 0 466 KBytes
[ 5] 5.00-6.00 sec 36.2 MBytes 305 Mbits/sec 0 523 KBytes
[ 5] 6.00-7.00 sec 40.0 MBytes 336 Mbits/sec 0 580 KBytes
[ 5] 7.00-8.00 sec 36.2 MBytes 304 Mbits/sec 1 466 KBytes
[ 5] 8.00-9.00 sec 37.5 MBytes 315 Mbits/sec 0 519 KBytes
[ 5] 9.00-10.00 sec 40.0 MBytes 336 Mbits/sec 0 566 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 335 MBytes 281 Mbits/sec 22 sender
[ 5] 0.00-10.01 sec 332 MBytes 278 Mbits/sec receiver
|
Using two or more parallel streams maxes out the line.
Yeah, problem solved?
The figures did not change at all when I changed this feature to off on the WAN facing ETH0 nic, therefore I let it enabled.
As I understand this "rx-gro-list" is a feature introduced way back then in older kernel.
There i checked my other routers like MR7350, the feature being set to "off" on those devices.
Now, why is it enabled on my X86 router by default?
Are there any reasons why this feature should NOT be turned off on this device, too?
Since I changed this setting there were no negative impacts, at least I encountered none.
I kindly ask for any input regarding this.
Why does this feature prohibit good upstream results? Is this a bug? |
|
