Updated newspost v2.1.1-4 Source Package

I’ve been uploading stacks of files to Usenet over on AusGamers lately to try and make it easier for gamers to get them, doing it all on Linux using newspost. I’ve been struggling with an issue for a few months – an error which stops a posting session in its tracks.

WARNING: unexpected server response: 240 post
WARNING: unexpected server response: 240 post
Socket error: Broken pipe

Fortunately newspost is awesome and with some careful checking I’ve been able to resume broken uploads by manually posting parts of files, but uploading large files (like the 10gb LOTRO client I uploaded over the weekend) is a total pain in the ass, as it means I’m constantly spending time restarting it (and it’s not uploading).

Well, I finally decided to try and fix it, but Googled the error first, and of course someone already had. Debian have a patch available which fixes this error. The official newspost site hasn’t been updated since 2003, so if you’re like me and just want a source tarball, I’ve patched the original source with it and made it available right here.

Recursively add missing files to SVN

Nats showed me this awesome little trick a while back – if you’ve just added a stack of new files to a directory and need to easily add them all into SVN from the command line (for example, if you’ve just unpacked a new WordPress install and there are new files in it that you need to add), you can use the following command:

svn add `svn st $DIR | grep -E '^\?' | cut -b 8-`

How to Download Trailers from Apple.com

If you are like me, you find streaming trailers from Apple is a pain in the ass. I still prefer having the trailer locally and playing it locally, even if it means a bit of screwing around. Here’s a quick way to download trailers:

1. Install the Firefox web browser
2. Install the Greasemonkey add-on for Firefox
3. Install wget for Windows (most Linux distros will have it installed already)
4. Add the Apple Trailer Download script to your greasemonkey distribution.
5. Enable greasemonkey
6. Go to an Apple trailer page, like The Slammin’ Salmon page
7. Note in the bottom right you’ll get a little window with a bunch of links to the various versions – looks like this:

apple-greasemonkey

Right click, copy link location for the version you want

8. Open a command prompt and fire up wget thusly:

wget -U QuickTime [url of trailer]
You need to specify the -U parameter (User-Agent) as Apple rejects any requests for trailers that are not from QuickTime.

10. Realise that this method violates the Apple.com Terms of Use and stop doing it.

Really Simple wget Tutorial

wget is a simple command line-based tool that allows you to download files from the command line. It is a very powerful tool for downloading with a lot of options, but it can also be really helpful in diagnosing common downloading problems.

Unfortunately, as it is a command line tool, it can be daunting for users to use. This post is intended as a really simple reference for people who have been asked to download something via wget and want some step-by-step instructions to follow:

STEP 1: Download wget

a) Go to the wget for Windows page by Bart Puype and look for the wget download link and download it to your computer – preferably to your desktop. If you choose to download it elsewhere, you’ll need to know where you downloaded it to for the next step.

The download link looks like this:

wget-for-windows

STEP 2: Open a command prompt

wget is a command line utility, which means you can’t just double click on it and have it work. You need to open a command prompt and run it from there.

a) Hit the following key combination: Windows Key and R (that is, hold down the Windows key and hit “R”). A small “run” dialog will open in the bottom left of your screen.

run-box

b) In the run dialog, type cmd.exe and then hit the “OK” button:

run-box-cmd

c) After doing this, a black window will open on your screen looking something like the below:

cmd-prompt

STEP 3: Find and test wget

By default, your “run” command will have you sitting at your user profile directory. If you followed the instructions above, you downloaded wget.exe into your Desktop directory. Now you need to run the following commands, which you should be able to just type directly (without the quotes):

a) Type “cd desktop” and hit enter to change into the directory in which wget.exe is residing (if you did not download to your Desktop, you’ll need to figure out where you did download it to, and manually change to that directory).

b) To check if wget is working, type: “wget” and hit enter. It should look like the below:

cmd-prompt-wget

If it does not, then something is wrong and you probably need to start again!
STEP 4: Get your download URL ready

If you’ve gotten this far, wget is correctly installed and ready to be used. All you need now is a URL to download.

A URL is the thing that you are clicking on that starts the actual file download process. When you click on a file download URL, you’ll get that little “save as” dialog in your browser which you then use to tell your browser where to put the file.

a) Instead of left-clicking on the URL you want to test to start the download, right-click on it. You will get a menu that looks like this (in Internet Explorer):

ie-context

Select “copy shortcut”. Note if you are using a different browser like Firefox the text might be different (Firefox is “Copy Link Shortcut” for example).

STEP 5: Use wget to download a file

Now we have everything we need!

a) Click back to your command prompt window, which should be sitting there ready and waiting.

b) Type “wget” – but don’t hit enter!

c) Hit the space bar, so you have a space after the wget command.

d) In the top left of the command prompt window, there’s a little “C:\” icon. Click that and go through the menu that results until you find the ‘Edit’ submenu, then hit ‘Paste’:

cmd-prompt-paste

e) If you have copied the link correctly, it should get pasted into the command line prompt, so you should end up with a something on the command line that looks like: “wget http://your.url.here.com/filename.zip”

f) Hit enter! The download will start and information will start spewing to the command prompt showing how fast it is downloading and how long it has left. If you get errors then you might not have pasted the right URL or there might be some other problem.

Extended Link Properties for Firefox v3.x

Ever looked at a link to a file online and wanted to find out how big the file is? Sure, if you’re hardcore you can fire up a command prompt and just do “wget –spider –debug [url]” or whatever, but surely you should just be able to right click on it and go to Properties and get a file size, just like in Windows, right?!

Well, the Extended Link Properties add-on fore Firefox is what I’ve always used to do exactly that. Unfortunately it hasn’t been updated for a billion years and doesn’t work in newer versions of Firefox.

I’ve been using the Nightly Tester Tools to force compatibility (because it’s just a version requirement) but I finally got bored with it constantly breaking every new Firefox version. I’ve dissected the add-on and re-built it into a new XPI which I have dubbed ‘Extended Link Properties v1.2.4’.

Anyone wanting to download it can do so here (md5sum: 03e3e278c2d8b921401cf1a8b68efc45), though you’ll need to manually install it at the moment by downloading the .xpi file, then in Firefox simply go to the File Menu then hit Open File and browse to the XPI. You’ll get the usual notice.

I have emailed Torisugari, the original developer of the add-on, to see if he’s still around. I’ll try and get it on the official extension site at some point if he replies.

Update: Torisugari has replied and pointed out the file is tri-licensed already under MPL/GPL/LGPL. He also points out that this extension will only useful up to version 3.5.x of Firefox – after that the Firefox developers have decided to remove support for the Properties dialog. If you use Properties and/or Extended Link Properties, you should head over and add your voice to that bug report.

As a result of that I’ve changed the v1.2.4 build of ELP that I put together to only support up to v3.5.x of Firefox.

Unpacking / extracting a .rpm file

I have never figured out the RPM package management system. It’s mostly because I haven’t tried hard, but it still makes me feel like a moron.

Every now and then I stumble across a package that, for whatever reason, is only distributed in .rpm form. Right now I wanted to install nano on godaddy.com’s shared hosting (because I’m also too lazy to learn vi), and the easiest way I could think of was to nab the .rpm and just rip out the nano binary.

Turns out this is really easy from a Linux shell:

# rpm2cpio [rpm filename] | cpio -idv

Another victory for laziness! It spewed out a bunch of stuff, I nabbed the nano binary, threw it on my godaddy shared hosting using wget via ssh, and now I have a fully functional and awesome editor (no matter what anyone tells you).

Troubleshooting GnuPG – gpg: no ultimately trusted keys found

My GPG installation (Windows binaries, some ancient version) has worked flawlessly for several years, but I just went to run my usual mail backup script after some minor changes – I installed enigmail for Thunderbird. This act, or some related act, appeared to mess up something in my keyring.

At first I thought it was that it had unsigned my keys, but a closer look indicated it was something to do with the trust database. I thought this would be a trivial problem to solve (ie, I’d be able to Google the error message and be given a nice, simple howto to follow), but I was surprised – there was a bunch of useless stuff.

Anyway, the warning appears to be related to there being no ultimately trusted key (funnily enough). That is, you haven’t specified a “root” key that you have declared as the one that you trust to make all other decisions (I’ve had 4 beers and might not be articulating the purpose of this well).

However, the fix is pretty simple. You just need to specify your key as “ultimately trusted”.

The easiest way to do this (assuming you are using GnuPG command line like I am) is to just edit your key and make it trusted:

1) gpg –edit-key [your key id]
2) select the key (I just typed ‘1’ and hit enter; you can confirm by typing ‘list’
3) type ‘trust’ to change the ownertrust
4) select option 5, “I trust ultimately”, then say ‘yes’ to the confirmation
5) type ‘quit’

…and you’re done.

Giganews Accelerator is Pretty Neat, but Needs Work

The Giganews Accelerator is a free Windows application distributed by Usenet provider Giganews to provide compressed, encrypted header downloads. It also provides in-built speed control and some statistics about connections.

I recently used it to download the full list of headers from alt.binaries.games (I accidentally blew away all the AusGamers NZBs from our file server and needed to recreate them), which worked out to be around 9gb of data (once stuffed into a MySQL database).

My header retrieval script estimated it was going to take about 30 hours from start to finish. I recalled something about compressed headers and did some Googling and found this, and then tried it out – the new estimate was less than 10 hours, and I think it actually finished faster than that (I left it running overnight and it was done by the time I got up).

There’s a couple of annoying things about it:

1) There’s only a Windows version. A Linux version would have saved me heaps of dicking around (my header stuff usually runs on Linux on an AusGamers server, so I had to move the whole thing onto my work box and Windowsify it all).

2) The Windows version will only listen on localhost. There’s no way to make it listen on all IPs for a box, so you can’t (for example) run it on a Windows box and connect to it from some other box (which would have obviated the need for number 1, at least for me in this particular scenario, as I have a Windows server I could have used).

I tried a few things I found – notably, netsh for Windows, which looks really handy but deserves another post all on its own, if only for the fact that it’s advertised purpose doesn’t seem to actually work on Windows XP because of some bug.

Overall though, if you’re doing anything on Usenet that involves getting a lot of headers, getting this thing working is pretty handy.

ANZ on the Security of Email

A while back, ANZ offered me the opportunity to receive some of my statements as ‘e-statements’. While I fully approve of the move away from paper, I must confess I was slightly disappointed to find out that they’d be emailing me notices about these e-statements – one of the big reasons I think Australia (or at least, ANZ) has done well in the fight against phishing is because they’ve simply not ever sent any emails out, ever. Contrast this to a US bank (Wells Fargo) – within days of signing up I’d received a huge variety of emails, making it easy to see why so many US citizens get scammed so easily.

I typically ignore these emails but as part of my ever-growing interest in how email works and how people use it, I checked out my most recent one, and was interested to see the following disclaimer in the email footer:

ANZ does not guarantee the integrity of this communication, or that it is free from errors, viruses or interference. As email is transmitted via the Internet, which is an unsecure environment, ANZ cannot ensure that an email is not interfered with during transmission.

Clearly they’ve never heard of public cryptography! Of course, even if they had, and the email was encrypted and/or digitally signed, that last sentence wold probably still exist from a sheer cover-their-ass perspective.

Still, I’m looking forward to the day when my bank (and other sites) let me enter in my public key as part of my account settings so all correspondence from them can be encrypted. I’m continually surprised that so few sites do this. I’m keen to integrate something like this into AusGamers – not that we really need it, but just because I think it would be cool to do.

It should be noted though that their emails include /no/ links at all and are sent in plain text.

Event Cinemas/Birch Mobile Site (with XSS holes)

I went to www.birch.com.au the other day to look up some timetables and they’ve replaced it with a new loud glary site that I couldn’t get working instantly. I turned off Javascript and found they have a mobile site as well, which is at http://m.greaterunion.com.au – it offers a really simple interface to quickly get timetables for their cinemas all across Australia.

Except, as jadz0r points out, it appears to be subject to XSS vulnerabilities, so use at your own risk.