dd-wrt mangling content?

Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware
Goto page 1, 2  Next
Author Message
diego.rivera
DD-WRT Novice


Joined: 16 May 2011
Posts: 11

PostPosted: Mon May 16, 2011 15:40    Post subject: dd-wrt mangling content? Reply with quote
Yes - as WEIRD as it sounds, and 100% reproducible! Other than the issue listed below, the router works fine.

The issue is related to the scriptaculous.js version used in the Apple trailers site. When I mention "download", below, I mean download with wget (to avoid caches, cookies, blablabla).

When my DD-WRT router is "in line" (i.e. between my PC and the internet), the downloaded file is the correct size, but is constructed of a gzipped version of itself at the head, and the remainder of the size is the normal uncompressed version of itself.

When my DD-WRT router is "off line" (i.e. computer directly connected to the internet), the downloaded file is 100% uncompressed. I have attached a file with the downloads, for your perusal.

By executing: gunzip < scriptaculous.js.ddwrt

You will obtain: scriptaculous.js.clear

The ".clear" file is the correct file.

The router is a WRT300N v1, running DD-WRT v24-sp2 (08/07/10) std - build 14896 (Broadcom BCM4704 chip rev 9 @ 264MHz).

I work in IT and I understand how networking works, which is why it was such a shock to me when I discovered this issue yesterday - dd-wrt is NOT configured (to my knowledge) to inspect (let alone manipulate in any way shape or form) downloaded content.

I've scoured the settings and haven't seen anything that would jump out at me as being responsible (or at least tangentially responsible) for this behavior. There's no visible squid instance running transparently either.

Can you guys offer any insights or advice?

I can offer as much detail and info as you'd like.

UPDATE:

Forgot to add the URL for the file: http://trailers.apple.com/trailers/global/scripts/lib/scriptaculous.js



scriptaculous.tar.gz
 Description:
Tar file containing the correct version of scriptaculous and the "dd-wrt mangled" version.

Download
 Filename:  scriptaculous.tar.gz
 Filesize:  80.96 KB
 Downloaded:  576 Time(s)

Sponsor
phuzi0n
DD-WRT Guru


Joined: 10 Oct 2006
Posts: 10143

PostPosted: Mon May 16, 2011 19:51    Post subject: Reply with quote
It could be your ISP caching or the apple server screwing up and only triggered by having the router in place (like the TTL difference) or your config is more complicated than you're letting on. Try to reproduce it with minimal settings (ie. only configure the WAN) after a hard reset of the router. If it still occurs then try it with the stock firmware too. If it still occurs after that then the router is just triggering the real cause.
_________________
Read the forum announcements thoroughly! Be cautious if you're inexperienced.
Available for paid consulting. (Don't PM about complicated setups otherwise)
Looking for bricks and spare routers to expand my collection. (not interested in G spec models)
diego.rivera
DD-WRT Novice


Joined: 16 May 2011
Posts: 11

PostPosted: Mon May 16, 2011 20:29    Post subject: Reply with quote
Even stranger - rebooting the router fixed it (if only for a while), only to have it happen again later.

Thus, I think this establishes the following:

a) It's not my ISP's caching (I've long suspected them of doing this, but if that were the case it's doubtful the caching would care either way whether the TTL is "whatever" - I've set up these kinds of caches before)

b) My config isn't really complex at all. I seriously don't have anything set up on the router except port forwarding

As I type this, I just rebooted the router and the site is working. I'm sure later on today it will fail again.

I seem to recall the existence of an exploit for dd-wrt via a specially-crafted JPEG image. This would suggest that, in some way, dd-wrt does indeed inspect/manipulate traffic flowing through vs. just routing it via packet headers.

The behavior of a fresh router boot fixing the issue (sometimes, this doesn't always work) would also suggest that this is related to the software.

I'll keep it under observation to see how it behaves.
diego.rivera
DD-WRT Novice


Joined: 16 May 2011
Posts: 11

PostPosted: Thu May 19, 2011 16:28    Post subject: Confirmed on a brand new Buffalo router Reply with quote
Ok...so I kept it under observation and the same erratic behavior persisted: could reboot the router, the site would work fine for a few minutes, but then would fail again after a short while.

So I bought a brand new Buffalo WZR-HP-AG300H router which, as I understand it, comes with DD-WRT pre-installed.

Guess what? SAME BEHAVIOR.

This means:

A) it's not the hardware
B) it's not the network (I executed your suggested test by turning everything off and leaving only the WAN routing enabled, no change)
C) it's not my PC (as mentioned above, if I connect the PC directly to the cablemodem, everything works like a charm)

So... what further information can I provide you guys to help you dig out this issue or - at least - find a cause?

UPDATE: Also, it's not the desktop machine's OS (happens with both Windows and Linux indiscriminately), it's not the website (other people on the same ISP's network segment are able to view it perfectly well), it's not browser cache (I'm dumb, but not THAT dumb Smile )...

At this point, the only moving part that I see could be influencing things is DD-WRT. I made a big boo-boo in buying the Buffalo router since I expected (nay, hoped) that it would work fine out of the box.

In retrospect, I should have bought something else to facilitate the isolation of the issue...

Still hope to work with "someone" (anyone!) to try to figure out what could be wrong here.

Thanks!
barryware
DD-WRT Guru


Joined: 26 Jan 2008
Posts: 13049
Location: Behind The Reset Button

PostPosted: Thu May 19, 2011 17:09    Post subject: Reply with quote
flash your 300 back to stock linksys firmware for the router and then test.

that takes dd-wrt out of the picture.

_________________
[Moderator Deleted] Shocked
diego.rivera
DD-WRT Novice


Joined: 16 May 2011
Posts: 11

PostPosted: Thu May 19, 2011 17:12    Post subject: Reply with quote
Ok...so more info (see attached TCPDUMP capture). It MAY have to do with fragmented packet reassembly when the fragments come in out of order.

At least, that's what the dump leads me to believe because I see abnormal amounts of "out of order" packets from a fragmented PDU.

And, of course, the end result is the corrupted file.

Anyway, have a look and tell me how clueless I am Smile

Thanks!

UPDATE: Added scriptaculous-direct.bin which provides a similar tcpdump of a fetch of the same file, over the same ISP link, but without the router "in-line" (i.e. computer direct to the cable modem).



scriptaculous-direct.bin
 Description:
Same capture as before, but done without the dd-wrt router "in-line" (i.e. computer direct to cable modem).

Download
 Filename:  scriptaculous-direct.bin
 Filesize:  128.13 KB
 Downloaded:  589 Time(s)


scriptaculous.bin
 Description:
Scriptaculous wget dump taken from INSIDE the dd-wrt router.

Download
 Filename:  scriptaculous.bin
 Filesize:  256.49 KB
 Downloaded:  655 Time(s)

diego.rivera
DD-WRT Novice


Joined: 16 May 2011
Posts: 11

PostPosted: Thu May 19, 2011 17:26    Post subject: Reply with quote
barryware wrote:
flash your 300 back to stock linksys firmware for the router and then test.

that takes dd-wrt out of the picture.


Where can I go about getting that stock firmware?

UPDATE: Nevermind, it seems I'm too stupid to use Google Smile


Last edited by diego.rivera on Thu May 19, 2011 19:03; edited 1 time in total
diego.rivera
DD-WRT Novice


Joined: 16 May 2011
Posts: 11

PostPosted: Thu May 19, 2011 19:00    Post subject: Reply with quote
Ok...I found the original firmware and installed it, and the problem persists. So the problem is definitely not DD-WRT exclusive.

That leaves two possibilities: my ISP, or Akamai (who I believe is the one serving up the file).

Can someone with more networking knowledge than I have a look at the tcpdump files I uploaded and see if they find any leads?

I'll be happy to help the investigation in any way I can, but I'm just not knowledgeable enough on the precise details of networking at that low level to spot broken things - aside the fact that it seems odd to me that on one dump (from dd-wrt) the packets reordering would be so "rampant" (and indeed, would appear that a re-download is requested from the size of the dump), vs. the direct dump.
EOC_Jason
DD-WRT User


Joined: 04 Jan 2010
Posts: 90

PostPosted: Thu May 19, 2011 19:07    Post subject: Reply with quote
As I was reading your thread, I thought to myself, "It's probably Akami or a similar CDN"...

My guess is as you request the file, there's one machine that is out of whack on their network and causing the issue... Best to report it to apple...
diego.rivera
DD-WRT Novice


Joined: 16 May 2011
Posts: 11

PostPosted: Thu May 19, 2011 19:13    Post subject: Reply with quote
EOC_Jason wrote:
As I was reading your thread, I thought to myself, "It's probably Akami or a similar CDN"...

My guess is as you request the file, there's one machine that is out of whack on their network and causing the issue... Best to report it to apple...


This doesn't make sense, since if I eliminate the wireless router from the equation, the file is downloaded perfectly and, as far as Akamai can see, it's the same IP address requesting the file.

If what you say is true, then the file would be corrupted as well when downloaded without the router in-line. This is not the case.

Therefore, I doubt that the problem is Akamai, unless somehow the TTL value is affecting it (which I also doubt).

At any rate, before I report anything, I need to have a more solid grasp of what the problem is, lest my report be tossed aside because it's not specific or useful enough to track the problem.

Cheers.
Fractal
DD-WRT Guru


Joined: 19 Apr 2010
Posts: 1243

PostPosted: Thu May 19, 2011 20:26    Post subject: Reply with quote
this problem is a javascript problem, ususally found on a mac... you need to make sure your browser doesnt have popup blocking the javascript.

Fractal
diego.rivera
DD-WRT Novice


Joined: 16 May 2011
Posts: 11

PostPosted: Thu May 19, 2011 20:31    Post subject: Reply with quote
Fractal wrote:
this problem is a javascript problem, ususally found on a mac... you need to make sure your browser doesnt have popup blocking the javascript.

Fractal


OK...please read the entire description above. This problem happens WITHOUT TOUCHING THE BROWSER, simply touching CABLES.

This is NOT a JavaScript (or "usually found on a mac") problem. I have a Mac laptop. I've tested with it as well - the results are consistent.

Again - please read the ENTIRE post and problem description before you post your suggestions. Also, when you do, try to substantiate why you think the problem/solution is what you suggest (even if cursorily) - this will be tremendously helpful in and of itself.

Thanks for all the attention, though!
EOC_Jason
DD-WRT User


Joined: 04 Jan 2010
Posts: 90

PostPosted: Sat May 21, 2011 14:39    Post subject: Reply with quote
diego.rivera wrote:
This doesn't make sense, since if I eliminate the wireless router from the equation, the file is downloaded perfectly and, as far as Akamai can see, it's the same IP address requesting the file.

If what you say is true, then the file would be corrupted as well when downloaded without the router in-line. This is not the case.


There are thousands of caching servers in their network. Which one you get routed to in any point in time is a function of complex load balancing algorithms... Just because you were sent to one server a previous time doesn't mean you will get the same one next time.

Download the file like 10-20 times (or even more) with and without a router in-line and compare your results.

We've used Akami and other CDNs in the past. Sometimes a stale file gets left out in the network but there's only a slight chance you end up getting it. In instances like those, we had to send out a "flush cache" so that all the Akami servers would re-download the content from our main servers.
dc
DD-WRT User


Joined: 08 Jun 2006
Posts: 247
Location: Prince Edward Island - Canada

PostPosted: Sat May 21, 2011 15:23    Post subject: Reply with quote
diego.rivera wrote:

<snip>

This doesn't make sense, since if I eliminate the wireless router from the equation, the file is downloaded perfectly and, as far as Akamai can see, it's the same IP address requesting the file.

If what you say is true, then the file would be corrupted as well when downloaded without the router in-line. This is not the case.

Therefore, I doubt that the problem is Akamai, unless somehow the TTL value is affecting it (which I also doubt).

what happens if use dd-wrt to clone your pc's Mac address (ie do you get a different external ip address if you attach your pc directly to your cable modem?)
diego.rivera
DD-WRT Novice


Joined: 16 May 2011
Posts: 11

PostPosted: Mon May 23, 2011 23:13    Post subject: Reply with quote
New developments:

The symptoms have changed - now it seems that plugging the PC to the cable modem directly no longer works either. Furthermore, testing with another internet link (neighbor's) from a completely different ISP produced identical results, with and without the (same) router in-line.

Thus, we can conclude that:

1) it's not dd-wrt (tested with the same router, and with and without the dd-wrt firmware)
2) it's not the ISP per-se (more accurately, both ISP's could be suffering the same low-probability problem in any putative caching infrastructure, which in and of itself is an even longer shot)
3) it's not the browser (happens with all browsers)

This leaves Akamai as the likely culprit. Now...how does one go about reporting this kind of content problem? Smile

Any ideas? Suggestions?
Goto page 1, 2  Next Display posts from previous:    Page 1 of 2
Post new topic   Reply to topic    DD-WRT Forum Forum Index -> Broadcom SoC based Hardware All times are GMT

Navigation

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum