trog's haus

I’ve been uploading stacks of files to Usenet over on AusGamers lately to try and make it easier for gamers to get them, doing it all on Linux using newspost. I’ve been struggling with an issue for a few months – an error which stops a posting session in its tracks.

WARNING: unexpected server response: 240 post
WARNING: unexpected server response: 240 post
Socket error: Broken pipe

Fortunately newspost is awesome and with some careful checking I’ve been able to resume broken uploads by manually posting parts of files, but uploading large files (like the 10gb LOTRO client I uploaded over the weekend) is a total pain in the ass, as it means I’m constantly spending time restarting it (and it’s not uploading).

Well, I finally decided to try and fix it, but Googled the error first, and of course someone already had. Debian have a patch available which fixes this error. The official newspost site hasn’t been updated since 2003, so if you’re like me and just want a source tarball, I’ve patched the original source with it and made it available right here.

Either someone is doing an excellent job phishing for MBF member details, or they screwed up sending out their most recent email newsletter.

The newsletter’s subject is “Your chance to win 10K worth of prizes” and it contains the following text:

Hi William,

At MBF we want to help you get the most from your health insurance.

Register to receive our updates via email today and we can help to keep you healthy and happy.

To register click here, login to myMBF and make sure you complete your contact details and update your communications preferences to email. If you’re already registered, make sure you check your details in case anything has changed since you last visited.

Two things immediately jump out at me here:

1. My name isn’t William. Something has gone wrong with their mail merge and it’s put someone else’s name on my email. Embarrassing, but not fatal.
2. I have all my MBF communications set to ‘post’ – all email options (that I can find) are switched off.

To complicate things further, the links are all of the form: http://p1.eservicesesp.com/cts/click?q=randomcharactershere – no doubt this is a mailing service that they’re using to track newsletter click throughs, but to a casual observer, it looks like it might be a phishing attempt.

The net result of this email is that it looks like a phishing attempt. But it’s almost certainly just a screw-up on behalf of MBF and/or the marketing agency they’ve employed to handle their bulk emails.

To make things even more painful, there’s no easy way to report the problem to MBF listed in the email (a simple “if you believe you have received this message in error, click here” link would have done the trick, or on their website. Even logged in, you don’t get an option to provide feedback over the web or via email – you have to use the phone or write them something called “a letter”.

Sending newsletters to lots of people is hard. But it’s not that hard. We’ve done it at Mammoth every week for almost eight years, sending millions of emails to hundreds of thousands of subscribers, so if you need some advice on how to do it right, maybe you should ask us

This story does not make web advertisers look good, although it’s almost so obviously phony that it barely warrants mentioning.

I randomly clicked on an island ad on a site that I was visiting (I sometimes do this to give them a click-through and show some support).

I got sent to the URL http://www.newswebdaily.com/health/white-teeth/index1.php, which then redirects to a new URL with a few different parameters in the query string.

It’s an ad that tells the story of Becky Bell, a teacher who wanted to try a teeth-whitening product:

For a split second, I thought “wow, that’s weird – she’s in Brisbane, Queensland, just like me!” Then I decided that seemed a bit too much like a coincidence, so I activated my US-based proxy server (handy for web development and testing) and got the following page:

So, to be crystal clear – this ad changes based on the location that you are viewing the page from, presumably to give you some feeling of confidence that it was a “local gal” that benefited from this product. Clearly, Becky Bell is not from Brisbane and Dallas at the same time.

Moral of the story: beware of advertising that just so happens to have your exact city and country in it like this.

Nats showed me this awesome little trick a while back – if you’ve just added a stack of new files to a directory and need to easily add them all into SVN from the command line (for example, if you’ve just unpacked a new WordPress install and there are new files in it that you need to add), you can use the following command:

svn add `svn st $DIR | grep -E '^\?' | cut -b 8-`

If you are like me, you find streaming trailers from Apple is a pain in the ass. I still prefer having the trailer locally and playing it locally, even if it means a bit of screwing around. Here’s a quick way to download trailers:

1. Install the Firefox web browser
2. Install the Greasemonkey add-on for Firefox
3. Install wget for Windows (most Linux distros will have it installed already)
4. Add the Apple Trailer Download script to your greasemonkey distribution.
5. Enable greasemonkey
6. Go to an Apple trailer page, like The Slammin’ Salmon page
7. Note in the bottom right you’ll get a little window with a bunch of links to the various versions – looks like this:

Right click, copy link location for the version you want

8. Open a command prompt and fire up wget thusly:

wget -U QuickTime [url of trailer]
You need to specify the -U parameter (User-Agent) as Apple rejects any requests for trailers that are not from QuickTime.

10. Realise that this method violates the Apple.com Terms of Use and stop doing it.

My brother needed to figure out how to do a “moving average” in some code he was writing a while ago. I’d never done this before and couldn’t find any really simple code examples so ended up on Wikipedia where I found it’s actually called a cumulative moving average.

Super simple code example follows:

<?php
$numbers = array(1,2,3,4);

// A simple function to calculate averages
function average($array)
{
	return (array_sum($array) / count($array));
}

/*
 * Simple function to calculate moving average with the following parameters
 * $datapoint - the most recently acquired new datapoint
 * $average - the current average
 * $count - the total number of items we're dealing with
 */
function cumulativeAverage($datapoint, $average, $count)
{
	return $average + (($datapoint - $average) /  $count);
}

// First let's print the average calculated normally so we can compare to the final result
print "Normal average:\t\t".average($numbers)."\n";

// $lastav stores the most recently calculated average
$lastav = 0;

// Loop through all the numbers in the array and calculate the cumulative average each time
for ($i = 0; $i<sizeof($numbers);$i++)
{
	$lastav = cumulativeAverage($numbers[$i], $lastav, $i+1);
}

print "Cumulative Average:\t".$lastav."\n";

>>

wget is a simple command line-based tool that allows you to download files from the command line. It is a very powerful tool for downloading with a lot of options, but it can also be really helpful in diagnosing common downloading problems.

Unfortunately, as it is a command line tool, it can be daunting for users to use. This post is intended as a really simple reference for people who have been asked to download something via wget and want some step-by-step instructions to follow:

STEP 1: Download wget

a) Go to the wget for Windows page by Bart Puype and look for the wget download link and download it to your computer – preferably to your desktop. If you choose to download it elsewhere, you’ll need to know where you downloaded it to for the next step.

The download link looks like this:

STEP 2: Open a command prompt

wget is a command line utility, which means you can’t just double click on it and have it work. You need to open a command prompt and run it from there.

a) Hit the following key combination: Windows Key and R (that is, hold down the Windows key and hit “R”). A small “run” dialog will open in the bottom left of your screen.

b) In the run dialog, type cmd.exe and then hit the “OK” button:

c) After doing this, a black window will open on your screen looking something like the below:

STEP 3: Find and test wget

By default, your “run” command will have you sitting at your user profile directory. If you followed the instructions above, you downloaded wget.exe into your Desktop directory. Now you need to run the following commands, which you should be able to just type directly (without the quotes):

a) Type “cd desktop” and hit enter to change into the directory in which wget.exe is residing (if you did not download to your Desktop, you’ll need to figure out where you did download it to, and manually change to that directory).

b) To check if wget is working, type: “wget” and hit enter. It should look like the below:

If it does not, then something is wrong and you probably need to start again!
STEP 4: Get your download URL ready

If you’ve gotten this far, wget is correctly installed and ready to be used. All you need now is a URL to download.

A URL is the thing that you are clicking on that starts the actual file download process. When you click on a file download URL, you’ll get that little “save as” dialog in your browser which you then use to tell your browser where to put the file.

a) Instead of left-clicking on the URL you want to test to start the download, right-click on it. You will get a menu that looks like this (in Internet Explorer):

Select “copy shortcut”. Note if you are using a different browser like Firefox the text might be different (Firefox is “Copy Link Shortcut” for example).

STEP 5: Use wget to download a file

Now we have everything we need!

a) Click back to your command prompt window, which should be sitting there ready and waiting.

b) Type “wget” – but don’t hit enter!

c) Hit the space bar, so you have a space after the wget command.

d) In the top left of the command prompt window, there’s a little “C:\” icon. Click that and go through the menu that results until you find the ‘Edit’ submenu, then hit ‘Paste’:

e) If you have copied the link correctly, it should get pasted into the command line prompt, so you should end up with a something on the command line that looks like: “wget http://your.url.here.com/filename.zip”

f) Hit enter! The download will start and information will start spewing to the command prompt showing how fast it is downloading and how long it has left. If you get errors then you might not have pasted the right URL or there might be some other problem.

Over on AusGamers, we run a moderately popular download service for files. We push out around, oh, 30 terabytes a month of data (this is a lot).

Our file servers work pretty hard, but we prefer the work they do be related to just reading files off risk and throwing them down the wire at users. Unfortunately sometimes they have to do other things – like deal with bad requests from really terrible download software.

In this case, FlashGet is the bad download software. It is really annoying. Here’s a few reasons why:

• If you give it a URL that 404s or 403s (ie, a URL that doesn’t exist or is forbidden), FlashGet inexplicably wants to keep retrying that URL, over and over every two seconds.
• It incorrectly identifies itself as an IE5-based browser. This is just rude at best, and flat-out lying at worst.

I have written about this earlier, but now that I’ve seen the following data from a single month of usage on our file servers, I think the time has come to do something more:

The top entry here is FlashGet, with over 16 million hits to our server. The vast, majority of these hits are 403 or 404 errors from repeatedly trying to access files that are no longer there or that it no longer has access to.

At this stage the plan is to block FlashGet users. This is harder than it sounds because it is so stupid it ignores things like 403s and 404s and keeps retrying. What I am thinking we’ll do is detect FlashGet via the user-agent string and then redirect them to a different file. The file will be a little video file that explains why their download failed.

Ever looked at a link to a file online and wanted to find out how big the file is? Sure, if you’re hardcore you can fire up a command prompt and just do “wget –spider –debug [url]” or whatever, but surely you should just be able to right click on it and go to Properties and get a file size, just like in Windows, right?!

Well, the Extended Link Properties add-on fore Firefox is what I’ve always used to do exactly that. Unfortunately it hasn’t been updated for a billion years and doesn’t work in newer versions of Firefox.

I’ve been using the Nightly Tester Tools to force compatibility (because it’s just a version requirement) but I finally got bored with it constantly breaking every new Firefox version. I’ve dissected the add-on and re-built it into a new XPI which I have dubbed ‘Extended Link Properties v1.2.4′.

Anyone wanting to download it can do so here (md5sum: 03e3e278c2d8b921401cf1a8b68efc45), though you’ll need to manually install it at the moment by downloading the .xpi file, then in Firefox simply go to the File Menu then hit Open File and browse to the XPI. You’ll get the usual notice.

I have emailed Torisugari, the original developer of the add-on, to see if he’s still around. I’ll try and get it on the official extension site at some point if he replies.

Update: Torisugari has replied and pointed out the file is tri-licensed already under MPL/GPL/LGPL. He also points out that this extension will only useful up to version 3.5.x of Firefox – after that the Firefox developers have decided to remove support for the Properties dialog. If you use Properties and/or Extended Link Properties, you should head over and add your voice to that bug report.

As a result of that I’ve changed the v1.2.4 build of ELP that I put together to only support up to v3.5.x of Firefox.