Posted: Mon May 16, 2011 15:40 Post subject: dd-wrt mangling content?
Yes - as WEIRD as it sounds, and 100% reproducible! Other than the issue listed below, the router works fine.
The issue is related to the scriptaculous.js version used in the Apple trailers site. When I mention "download", below, I mean download with wget (to avoid caches, cookies, blablabla).
When my DD-WRT router is "in line" (i.e. between my PC and the internet), the downloaded file is the correct size, but is constructed of a gzipped version of itself at the head, and the remainder of the size is the normal uncompressed version of itself.
When my DD-WRT router is "off line" (i.e. computer directly connected to the internet), the downloaded file is 100% uncompressed. I have attached a file with the downloads, for your perusal.
By executing: gunzip < scriptaculous.js.ddwrt
You will obtain: scriptaculous.js.clear
The ".clear" file is the correct file.
The router is a WRT300N v1, running DD-WRT v24-sp2 (08/07/10) std - build 14896 (Broadcom BCM4704 chip rev 9 @ 264MHz).
I work in IT and I understand how networking works, which is why it was such a shock to me when I discovered this issue yesterday - dd-wrt is NOT configured (to my knowledge) to inspect (let alone manipulate in any way shape or form) downloaded content.
I've scoured the settings and haven't seen anything that would jump out at me as being responsible (or at least tangentially responsible) for this behavior. There's no visible squid instance running transparently either.
Can you guys offer any insights or advice?
I can offer as much detail and info as you'd like.
It could be your ISP caching or the apple server screwing up and only triggered by having the router in place (like the TTL difference) or your config is more complicated than you're letting on. Try to reproduce it with minimal settings (ie. only configure the WAN) after a hard reset of the router. If it still occurs then try it with the stock firmware too. If it still occurs after that then the router is just triggering the real cause. _________________ Read the forum announcements thoroughly! Be cautious if you're inexperienced.
Available for paid consulting. (Don't PM about complicated setups otherwise)
Looking for bricks and spare routers to expand my collection. (not interested in G spec models)
Even stranger - rebooting the router fixed it (if only for a while), only to have it happen again later.
Thus, I think this establishes the following:
a) It's not my ISP's caching (I've long suspected them of doing this, but if that were the case it's doubtful the caching would care either way whether the TTL is "whatever" - I've set up these kinds of caches before)
b) My config isn't really complex at all. I seriously don't have anything set up on the router except port forwarding
As I type this, I just rebooted the router and the site is working. I'm sure later on today it will fail again.
I seem to recall the existence of an exploit for dd-wrt via a specially-crafted JPEG image. This would suggest that, in some way, dd-wrt does indeed inspect/manipulate traffic flowing through vs. just routing it via packet headers.
The behavior of a fresh router boot fixing the issue (sometimes, this doesn't always work) would also suggest that this is related to the software.
I'll keep it under observation to see how it behaves.
Posted: Thu May 19, 2011 16:28 Post subject: Confirmed on a brand new Buffalo router
Ok...so I kept it under observation and the same erratic behavior persisted: could reboot the router, the site would work fine for a few minutes, but then would fail again after a short while.
So I bought a brand new Buffalo WZR-HP-AG300H router which, as I understand it, comes with DD-WRT pre-installed.
Guess what? SAME BEHAVIOR.
A) it's not the hardware
B) it's not the network (I executed your suggested test by turning everything off and leaving only the WAN routing enabled, no change)
C) it's not my PC (as mentioned above, if I connect the PC directly to the cablemodem, everything works like a charm)
So... what further information can I provide you guys to help you dig out this issue or - at least - find a cause?
UPDATE: Also, it's not the desktop machine's OS (happens with both Windows and Linux indiscriminately), it's not the website (other people on the same ISP's network segment are able to view it perfectly well), it's not browser cache (I'm dumb, but not THAT dumb )...
At this point, the only moving part that I see could be influencing things is DD-WRT. I made a big boo-boo in buying the Buffalo router since I expected (nay, hoped) that it would work fine out of the box.
In retrospect, I should have bought something else to facilitate the isolation of the issue...
Still hope to work with "someone" (anyone!) to try to figure out what could be wrong here.
Ok...I found the original firmware and installed it, and the problem persists. So the problem is definitely not DD-WRT exclusive.
That leaves two possibilities: my ISP, or Akamai (who I believe is the one serving up the file).
Can someone with more networking knowledge than I have a look at the tcpdump files I uploaded and see if they find any leads?
I'll be happy to help the investigation in any way I can, but I'm just not knowledgeable enough on the precise details of networking at that low level to spot broken things - aside the fact that it seems odd to me that on one dump (from dd-wrt) the packets reordering would be so "rampant" (and indeed, would appear that a re-download is requested from the size of the dump), vs. the direct dump.
OK...please read the entire description above. This problem happens WITHOUT TOUCHING THE BROWSER, simply touching CABLES.
Again - please read the ENTIRE post and problem description before you post your suggestions. Also, when you do, try to substantiate why you think the problem/solution is what you suggest (even if cursorily) - this will be tremendously helpful in and of itself.
This doesn't make sense, since if I eliminate the wireless router from the equation, the file is downloaded perfectly and, as far as Akamai can see, it's the same IP address requesting the file.
If what you say is true, then the file would be corrupted as well when downloaded without the router in-line. This is not the case.
There are thousands of caching servers in their network. Which one you get routed to in any point in time is a function of complex load balancing algorithms... Just because you were sent to one server a previous time doesn't mean you will get the same one next time.
Download the file like 10-20 times (or even more) with and without a router in-line and compare your results.
We've used Akami and other CDNs in the past. Sometimes a stale file gets left out in the network but there's only a slight chance you end up getting it. In instances like those, we had to send out a "flush cache" so that all the Akami servers would re-download the content from our main servers.
The symptoms have changed - now it seems that plugging the PC to the cable modem directly no longer works either. Furthermore, testing with another internet link (neighbor's) from a completely different ISP produced identical results, with and without the (same) router in-line.
Thus, we can conclude that:
1) it's not dd-wrt (tested with the same router, and with and without the dd-wrt firmware)
2) it's not the ISP per-se (more accurately, both ISP's could be suffering the same low-probability problem in any putative caching infrastructure, which in and of itself is an even longer shot)
3) it's not the browser (happens with all browsers)
This leaves Akamai as the likely culprit. Now...how does one go about reporting this kind of content problem?