The Giganews Accelerator is a free Windows application distributed by Usenet provider Giganews to provide compressed, encrypted header downloads. It also provides in-built speed control and some statistics about connections.
I recently used it to download the full list of headers from alt.binaries.games (I accidentally blew away all the AusGamers NZBs from our file server and needed to recreate them), which worked out to be around 9gb of data (once stuffed into a MySQL database).
My header retrieval script estimated it was going to take about 30 hours from start to finish. I recalled something about compressed headers and did some Googling and found this, and then tried it out – the new estimate was less than 10 hours, and I think it actually finished faster than that (I left it running overnight and it was done by the time I got up).
There’s a couple of annoying things about it:
1) There’s only a Windows version. A Linux version would have saved me heaps of dicking around (my header stuff usually runs on Linux on an AusGamers server, so I had to move the whole thing onto my work box and Windowsify it all).
2) The Windows version will only listen on localhost. There’s no way to make it listen on all IPs for a box, so you can’t (for example) run it on a Windows box and connect to it from some other box (which would have obviated the need for number 1, at least for me in this particular scenario, as I have a Windows server I could have used).
I tried a few things I found – notably, netsh for Windows, which looks really handy but deserves another post all on its own, if only for the fact that it’s advertised purpose doesn’t seem to actually work on Windows XP because of some bug.
Overall though, if you’re doing anything on Usenet that involves getting a lot of headers, getting this thing working is pretty handy.