Over on AusGamers, we run a moderately popular download service for files. We push out around, oh, 30 terabytes a month of data (this is a lot).
Our file servers work pretty hard, but we prefer the work they do be related to just reading files off risk and throwing them down the wire at users. Unfortunately sometimes they have to do other things – like deal with bad requests from really terrible download software.
In this case, FlashGet is the bad download software. It is really annoying. Here’s a few reasons why:
- If you give it a URL that 404s or 403s (ie, a URL that doesn’t exist or is forbidden), FlashGet inexplicably wants to keep retrying that URL, over and over every two seconds.
- It incorrectly identifies itself as an IE5-based browser. This is just rude at best, and flat-out lying at worst.
I have written about this earlier, but now that I’ve seen the following data from a single month of usage on our file servers, I think the time has come to do something more:
The top entry here is FlashGet, with over 16 million hits to our server. The vast, majority of these hits are 403 or 404 errors from repeatedly trying to access files that are no longer there or that it no longer has access to.
At this stage the plan is to block FlashGet users. This is harder than it sounds because it is so stupid it ignores things like 403s and 404s and keeps retrying. What I am thinking we’ll do is detect FlashGet via the user-agent string and then redirect them to a different file. The file will be a little video file that explains why their download failed.
Okay great but you did not suggest downlaoders that don’t do this.