Spammy User-Agent “Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)” is Probably FlashGet

If you’ve ever run a remotely popular Apache web server you might have used the mod_limitipconn module, which stops people from making too many simultaneous connections from the same IP address. With this module active, anyone trying to make too many connections will get an HTTP 503 Service Temporarily Unavailable error message.

Now, over the years watching log files for downloads from AusGamers, we’ve seen a lot of weird shit. One of the more common problems we’ve seen is a constant stream of spammy requests coming from many users that have all been identifying as “Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)”. I looked at this ages ago and didn’t figure it out then, but after getting annoyed with it again I spent a bit more time today and have since figured out that this is the default User-Agent of the popular download manager, FlashGet.

Why it chooses to identify, by default, as IE running on Windows 98 is a bit beyond me, but that’s something I find annoying, because it made this harder than it should have to diagnose.

The real problem though is how FlashGet handles 503 errors. The RFC seems to imply that, without a Retry-After header, it should act as it should with a regular 500 error (“server encountered an unexpected condition which prevented it from fulfilling the request”).

What FlashGet does though is re-attempts the download every 3 seconds – whether or not the Retry-After header is present! Every 3 seconds to me seems a bit unnecessarily aggressive, but failing to respect the Retry-After header is the real problem here, as there’s nothing server administrators can do to reduce the number of excess attempts (short of blocking this User-Agent completely – which, realistically, probably isn’t that bad an idea).

This means that, if you’re downloading using FlashGet and you’re using the defaults, the whole time you’re downloading from a server that is using 503s to try to block you from making too many connections, you’re also spamming the server with requests for files every 3 seconds. The default (as of version 1.9.6.1073) is to try to create 5 connections at once.

As a user, you probably don’t give a shit, but it’s a real pain in the ass for people running servers, as it means log files quickly fill up with thousands upon thousands of these attempts over the course of a single download from a single user. Start adding in thousands of users and you quickly end up with a really annoying situation.

What FlashGet should do:

Assuming that I’m right about the above (which I’m relatively confident about after some testing, though not 100%; it’s certainly possible I screwed up or missed something), here’s some changes I’d like to see (in increasing order of importance):

1) Change their default User-Agent to identify as FlashGet.
2) Change the default behaviour on 503s to wait longer than 3 seconds. I think 60 seconds is a reasonable “bare minimum”, though I would say the longer the better.
3) Make it respect the Retry-After header. This is super-important.

I have posted this as a suggestion on the official FlashGet forum.