trog's haus

I’ve been uploading stacks of files to Usenet over on AusGamers lately to try and make it easier for gamers to get them, doing it all on Linux using newspost. I’ve been struggling with an issue for a few months – an error which stops a posting session in its tracks.

WARNING: unexpected server response: 240 post
WARNING: unexpected server response: 240 post
Socket error: Broken pipe

Fortunately newspost is awesome and with some careful checking I’ve been able to resume broken uploads by manually posting parts of files, but uploading large files (like the 10gb LOTRO client I uploaded over the weekend) is a total pain in the ass, as it means I’m constantly spending time restarting it (and it’s not uploading).

Well, I finally decided to try and fix it, but Googled the error first, and of course someone already had. Debian have a patch available which fixes this error. The official newspost site hasn’t been updated since 2003, so if you’re like me and just want a source tarball, I’ve patched the original source with it and made it available right here.

Nats showed me this awesome little trick a while back – if you’ve just added a stack of new files to a directory and need to easily add them all into SVN from the command line (for example, if you’ve just unpacked a new WordPress install and there are new files in it that you need to add), you can use the following command:

svn add `svn st $DIR | grep -E '^\?' | cut -b 8-`

If you are like me, you find streaming trailers from Apple is a pain in the ass. I still prefer having the trailer locally and playing it locally, even if it means a bit of screwing around. Here’s a quick way to download trailers:

1. Install the Firefox web browser
2. Install the Greasemonkey add-on for Firefox
3. Install wget for Windows (most Linux distros will have it installed already)
4. Add the Apple Trailer Download script to your greasemonkey distribution.
5. Enable greasemonkey
6. Go to an Apple trailer page, like The Slammin’ Salmon page
7. Note in the bottom right you’ll get a little window with a bunch of links to the various versions – looks like this:

Right click, copy link location for the version you want

8. Open a command prompt and fire up wget thusly:

wget -U QuickTime [url of trailer]
You need to specify the -U parameter (User-Agent) as Apple rejects any requests for trailers that are not from QuickTime.

10. Realise that this method violates the Apple.com Terms of Use and stop doing it.

wget is a simple command line-based tool that allows you to download files from the command line. It is a very powerful tool for downloading with a lot of options, but it can also be really helpful in diagnosing common downloading problems.

Unfortunately, as it is a command line tool, it can be daunting for users to use. This post is intended as a really simple reference for people who have been asked to download something via wget and want some step-by-step instructions to follow:

STEP 1: Download wget

a) Go to the wget for Windows page by Bart Puype and look for the wget download link and download it to your computer – preferably to your desktop. If you choose to download it elsewhere, you’ll need to know where you downloaded it to for the next step.

The download link looks like this:

STEP 2: Open a command prompt

wget is a command line utility, which means you can’t just double click on it and have it work. You need to open a command prompt and run it from there.

a) Hit the following key combination: Windows Key and R (that is, hold down the Windows key and hit “R”). A small “run” dialog will open in the bottom left of your screen.

b) In the run dialog, type cmd.exe and then hit the “OK” button:

c) After doing this, a black window will open on your screen looking something like the below:

STEP 3: Find and test wget

By default, your “run” command will have you sitting at your user profile directory. If you followed the instructions above, you downloaded wget.exe into your Desktop directory. Now you need to run the following commands, which you should be able to just type directly (without the quotes):

a) Type “cd desktop” and hit enter to change into the directory in which wget.exe is residing (if you did not download to your Desktop, you’ll need to figure out where you did download it to, and manually change to that directory).

b) To check if wget is working, type: “wget” and hit enter. It should look like the below:

If it does not, then something is wrong and you probably need to start again!
STEP 4: Get your download URL ready

If you’ve gotten this far, wget is correctly installed and ready to be used. All you need now is a URL to download.

A URL is the thing that you are clicking on that starts the actual file download process. When you click on a file download URL, you’ll get that little “save as” dialog in your browser which you then use to tell your browser where to put the file.

a) Instead of left-clicking on the URL you want to test to start the download, right-click on it. You will get a menu that looks like this (in Internet Explorer):

Select “copy shortcut”. Note if you are using a different browser like Firefox the text might be different (Firefox is “Copy Link Shortcut” for example).

STEP 5: Use wget to download a file

Now we have everything we need!

a) Click back to your command prompt window, which should be sitting there ready and waiting.

b) Type “wget” – but don’t hit enter!

c) Hit the space bar, so you have a space after the wget command.

d) In the top left of the command prompt window, there’s a little “C:\” icon. Click that and go through the menu that results until you find the ‘Edit’ submenu, then hit ‘Paste’:

e) If you have copied the link correctly, it should get pasted into the command line prompt, so you should end up with a something on the command line that looks like: “wget http://your.url.here.com/filename.zip”

f) Hit enter! The download will start and information will start spewing to the command prompt showing how fast it is downloading and how long it has left. If you get errors then you might not have pasted the right URL or there might be some other problem.

Ever looked at a link to a file online and wanted to find out how big the file is? Sure, if you’re hardcore you can fire up a command prompt and just do “wget –spider –debug [url]” or whatever, but surely you should just be able to right click on it and go to Properties and get a file size, just like in Windows, right?!

Well, the Extended Link Properties add-on fore Firefox is what I’ve always used to do exactly that. Unfortunately it hasn’t been updated for a billion years and doesn’t work in newer versions of Firefox.

I’ve been using the Nightly Tester Tools to force compatibility (because it’s just a version requirement) but I finally got bored with it constantly breaking every new Firefox version. I’ve dissected the add-on and re-built it into a new XPI which I have dubbed ‘Extended Link Properties v1.2.4′.

Anyone wanting to download it can do so here (md5sum: 03e3e278c2d8b921401cf1a8b68efc45), though you’ll need to manually install it at the moment by downloading the .xpi file, then in Firefox simply go to the File Menu then hit Open File and browse to the XPI. You’ll get the usual notice.

I have emailed Torisugari, the original developer of the add-on, to see if he’s still around. I’ll try and get it on the official extension site at some point if he replies.

Update: Torisugari has replied and pointed out the file is tri-licensed already under MPL/GPL/LGPL. He also points out that this extension will only useful up to version 3.5.x of Firefox – after that the Firefox developers have decided to remove support for the Properties dialog. If you use Properties and/or Extended Link Properties, you should head over and add your voice to that bug report.

As a result of that I’ve changed the v1.2.4 build of ELP that I put together to only support up to v3.5.x of Firefox.

I have never figured out the RPM package management system. It’s mostly because I haven’t tried hard, but it still makes me feel like a moron.

Every now and then I stumble across a package that, for whatever reason, is only distributed in .rpm form. Right now I wanted to install nano on godaddy.com’s shared hosting (because I’m also too lazy to learn vi), and the easiest way I could think of was to nab the .rpm and just rip out the nano binary.

Turns out this is really easy from a Linux shell:

# rpm2cpio [rpm filename] | cpio -idv

Another victory for laziness! It spewed out a bunch of stuff, I nabbed the nano binary, threw it on my godaddy shared hosting using wget via ssh, and now I have a fully functional and awesome editor (no matter what anyone tells you).

The Giganews Accelerator is a free Windows application distributed by Usenet provider Giganews to provide compressed, encrypted header downloads. It also provides in-built speed control and some statistics about connections.

I recently used it to download the full list of headers from alt.binaries.games (I accidentally blew away all the AusGamers NZBs from our file server and needed to recreate them), which worked out to be around 9gb of data (once stuffed into a MySQL database).

My header retrieval script estimated it was going to take about 30 hours from start to finish. I recalled something about compressed headers and did some Googling and found this, and then tried it out – the new estimate was less than 10 hours, and I think it actually finished faster than that (I left it running overnight and it was done by the time I got up).

There’s a couple of annoying things about it:

1) There’s only a Windows version. A Linux version would have saved me heaps of dicking around (my header stuff usually runs on Linux on an AusGamers server, so I had to move the whole thing onto my work box and Windowsify it all).

2) The Windows version will only listen on localhost. There’s no way to make it listen on all IPs for a box, so you can’t (for example) run it on a Windows box and connect to it from some other box (which would have obviated the need for number 1, at least for me in this particular scenario, as I have a Windows server I could have used).

I tried a few things I found – notably, netsh for Windows, which looks really handy but deserves another post all on its own, if only for the fact that it’s advertised purpose doesn’t seem to actually work on Windows XP because of some bug.

Overall though, if you’re doing anything on Usenet that involves getting a lot of headers, getting this thing working is pretty handy.

After reading Hogfather’s post in this forum thread about people recording stuff off the radio it occurred to me that people probably don’t know how to do it in this new era of digital streaming radio.

Be aware that this is almost certainly not fair use and almost certainly is a violation of copyright. It is here for educational purposes only. I personally use this if there’s a song on the radio that I want to remember to look up later so I just grab a little bit of the lyrics so I can Google for it later to find out what song it is.

Here’s how to do it (you will need some familiarity with using the command line):

1) Get wget. I’ll assume you’re using Windows, so go to the wget for Windows page and grab the latest version.

2) Go the TripleJ website and look for the ‘Listen Live’ box which has the MP3 stream link.

3) Right click the link and save the file (triplej.m3u) to your computer.

4) Open that file in a text editor like notepad – it will just contain a single line, which will be a HTTP URL to the mp3 audio stream (at the time of writing that URL is http://202.6.74.107:8060/triplej.mp3).

5) Open your command prompt and invoke wget with that URL (or the URL in the file, if it’s different, obviously) as the parameter:

wget http://202.6.74.107:8060/triplej.mp3

This will start downloading an mp3 file which is a recording of what is playing on the Triple J stream.

WARNING: If you don’t stop it downloading, that file will grow to be huge. It will keep downloading as long as the stream is active. Just hit CTRL-C to stop it downloading once you’ve recorded the section of the song you want.

I recently got an email from PayPal (an actual one, not a phishing attempt) telling me that they’re changing their email address (subject: “Important Address Change Information from PayPal”).

I noticed this part:

How do I know if an email is really from PayPal?
PayPal emails only come from a ‘paypal.com.au’ or ‘paypal.com’ address. We will always address you by your first name AND last name.

It’s important to note that the first part of this answer is utterly useless from a security point of view. Anyone that knows anything about the Internet will tell you that it is completely trivial to send an email so it looks like it is coming from any email address. Email has no built-in security to stop this from happening.

It’s a little annoying that PayPal focus on that by putting it first, because it’s much, much less of a useful security measure than the second thing they propose – using your first AND your last name. Most email spam/phishing attempts simply attempt to guess your name by deriving it from your email address – for example, if your email address is david@example.com, then they’ll start their email with “Dear David”.

However, there’s (almost) no way to derive your last name in bulk mailing attempts like this – unless you already have that information, like PayPal would if you had an account with them. (I say ‘almost’ because there are fringe cases where spammers could guess your first and last name – for example, if your email address is formatted like david.harrison@example.com).

If you’re reading emails and wondering whether or not they’re from who they purport to be, bear in mind that looking at the actual email address is never a good way to do it. You’ll need to look for other clues.

Unless, of course, they’re using PGP or some other mechanism to digitally sign their emails. It boggles my mind that financial institutions aren’t offering this as a matter of course, even if only a handful of people would actually use it.

In addition to my below post, I was surprised to find the source for the latest version of MediaCoder is not available. Poking around on the forums I found a bunch of people that point out it hasn’t been open source for ages, which I didn’t realise. Articles like this one on their blog – asking for nomination in SourceForge.net’s Community Choice Awards – make this a little confusing.

It looks like you can still get the source for old (pre v0.7) versions, so I guess it still qualifies as open source in that regard, but Vic has pointed out MediaCoder is listed on ffmpeg’s Hall of Shame and there’s an ongoing ticket to try and get it removed from SourceForge.

I’ve always loved MediaCoder; it’s an awesome piece of software for video transcoding and takes a lot of the headaches. But really, all it does is provide a nice front-end to a bunch of open source tools and chains them together in a useful way. It seems the developers of MediaCoder have decided they’d rather pursue it as a commercial project now, so it’ll be interesting to see what happens from here.