Joined: 09 Jul 2013
|Posted: Sat May 18, 2019 16:20 Post subject: Insights: R7000 w/ 4x VLAN Trunks & Bridges, 5x WAP
|Thanks to a great bugfix from BrainSlayer and other help, I've learned a ton in the last few weeks and am sharing lessons learned here.
Summary (what you'll find below)
- Diagnosing/fixing stability issues with multiple WiFi interfaces
- Actual unique-MAC requirements
- Enabling many WiFI interfaces, vLANs, Bridges, etc
- Diagnosing performance limits of DD-WRT, and a solution
- Changing from an all-DD-WRT solution, to DD-WRT as WAP
- VLAN Trunking: undocumented req'ts
- A clean/simple multi-interface, multi-subnet WAP example
- Five WiFI SSID's on multiple subnets (2 public, 2 private, 1 for IoT)
- OpenVPN server linking to remote small office with net-to-net SMB networking
- Internal gigabit LAN for work, email hosting, 2 NAS boxes, etc
- Upgrade from 40mbit fiber to gigabit fiber
- R7000 with Kong firmware
- Multiple not-so-smart switches
- Couldn't get the WiFi's to stay up with stability over time
- R7000 CPU overload at gigabit speed, let alone OpenVPN
I learned the hard way that the instability was somehow related to MAC address uniqueness. But it wasn't clear until recently. Now I know:
Instability: All WiFi interface MAC addresses MUST be unique.
Here are (Shell/SSH) commands to understand your reality:
(this shows overview incl MAC for each interface)
#wl -i <interface> bssid
(this shows at a low level)
Important: Bridges can all share one MAC. VLANs likewise. They are not the issue.
Performance: How to determine overload sources?
#top (shows per-process loads, and overall interrupt loads)
#iperf3 (throughput to/from router. Usually you want *through* router tests between two endpoints. Download iperf for your various boxen ;) )
Result: Neither R7000 nor other "affordable" routers can offload much of the packet processing or encryption to hardware.
* As of BS changeset 39780 a week ago, the MAC address issue is fixed. It bumps the 5G WiFi MAC to be another 16 addresses bigger than 2.4G MAC. No danger of overlap.
* You can manually poke this into your own nvram:
|nvram show|grep pci/2/1/macaddr
(add one to the upper nibble of the last byte. E.g. if E7 change to F7. Should be ok to change F7 to 07.)
nvram set "pci/2/1/macaddr"=(your fixed MAC)
It might take 2-3 reboots for the change to propagate everywhere.
* In addition, I'm using the startup "workaround" script others have used. In my experience, making changes to the WiFi interfaces is NOT effective until I reboot and allow this script to run. I do NOT know if all of it is necessary.
sleep 10;stopservice nas;stopservice wlconf;startservice wlconf;startservice nas
wlconf eth1 down
wlconf eth1 up
wlconf eth2 down
wlconf eth2 up
Let me say up front... if it were not for my OpenVPN requirement, I'm not at all sure that all of this was necessary. It was cheap and I could do it ...
HOWEVER, a single DD-WRT Wifi Router/Switch/OpenVPN Server is incredibly simpler to configure and maintain. Incredibly.
First, I spent $50 on two things:
* $40 on a Netgear GSS108e 8 port smart switch (what a deal, lifetime warranty)
* $10 on a good gigabit ethernet PCIe card
I have a spare old quad core mini-desktop case computer, and a spare SSD. Combine the computer, the spare ethernet card, and the SSD... voila a great platform for the pfSense firewall/router!
(ISP) <--> (pfSense) <--> (Smart Switch) <--> (DDWRT)
* DDWRT only handles WiFi, VLAN trunking and bridging
* pfSense handles OpenVPN, DNSmasq, DHCP, etc
* 6 trunked VLANs, pfSense to Smart Switch
* 4 trunked VLANs, Smart Switch to DDWRT
* 2 VLANs uniquely Smart Switch to wired subnets
* 2 VLANs both wired and WiFi, handled at the Smart switch
* 2 VLANs are pure WiFi
* *.1 subnet gateways are at pfSense
* *.250 are subnet access to DD-WRT (SSH and browser)
* Pretty much all PC-type ethernet cards easily offload network I/O, checksums, VLAN tagging and more
For each VLAN-WiFi combination:
- Configure one new bridge (I have br9, br11, br12, br15)
- Configure the desired VLANs on one DD-WRT port (I use port 4. VLANs 9, 11, 12, 15 are all there, and tagged)
- Set up each WiFi interface as Bridged
- Attach one VLAN and one or more WiFI's to each bridge
- Define IP info on each bridge (Setup->Networking), this defines the IP of that subnet at your DD-WRT box. Leave things like Masq/NAT, Net Isolation, etc OFF. Those happen at pfSense in my configuration.
- (For my config, my DD-WRT firewall is off, dnsmasq off, etc.)
- Remember to manually configure the nvram as described in the Wiki entry for Switched Ports: https://wiki.dd-wrt.com/wiki/index.php/Switched_Ports
* Both pfSense box and DDWRT are coasting
* Full performance to the limit of my networks
* (My sense: Quad core is overkill except for OpenVPN)
* pfSense supports multiple MAC for one IP. VERY nice for laptops and such!
* Don't use VLAN 1 to connect to other boxes. It usually has special meaning.
* Always preserve a simple NON-VLAN way to connect to a port
* It helps to have a spare port that can be configured for one untagged VLAN
* When trunking VLANs, the trunk port MUST only use tagged VLANs. NO UNTAGGED. It will cause grief. This is true with DD-WRT, with pfSense, and with a Smart switch.
* For diagnostics, local tcpdump is your friend. I still don't have WireShark showing VLAN info. On tcpdump:
tcpdump -env VLAN [nn] -- shows all VLAN tagged traffic (optionally for a single vlan)
tcpdump -env VLAN and port 67 -- all VLAN tagged traffic for a given port
tcpdump -env port 67 and VLAN -- DOES NOT WORK. VLAN must come first. Don't know why.