trog’s haus

We bought a Canon HF-10 video camera in July, mostly to take to E3 to shoot the occasional video. It’s a great camera - small, light, easy to use, nice build quality, SDHC cards, etc.

The software that comes with it - something called ImageMixer 3 SE - is surprisingly not sucky. I mean, it still sucks in the sense that almost all OEM software does, but I was impressed that it supported three export types (DVD-quality MPEG2, iPod-quality h264 MP4, and Youtube). Unfortunately since installing the software my laptop is prone to bluescreen, but I guess that happens.

Sadly, those export options aren’t enough for me - I want more control over the formats.

So, that created the still-ongoing saga of trying to find an path to convert the camera’s native .M2TS or .MTS files into any random format (preferably h264 MPEG-4 for use in a Flash video player) - using open source software.

MediaCoder

The first thing I tried was MediaCoder, which I’ve had some success with in the past. It’s under active development by a dude in China and is relatively reliable, updated frequently with the latest versions of things like ffmpeg and x264 - so it can generally read just about anything.

I was further stoked when I read on their blog that it had AVCHD support in a recent update. Unfortunately, try as a I might, I couldn’t get it working (along with just about everyone else). You can read about some of the saga here, but the short version is I could only get it working after reading some random post on the forums with some information about specific settings.

I’ve compiled those settings into a sort-of-working MediaCoder preset profile, which you can download here:

* Download HF-10/AVCHD MediaCoder preset

Note, however, that it doesn’t work reliably for me. The longer the file the more likely it is to fail, crashing randomly at some point in the file. For short files though, it seems to work fine. I’ve posted about this on the forum.

Other OSS software:

I tried just about everything else I could find in various combinations. The only thing that came close to working was installing a recent version of ffdshow-tryout and using AviSynth and VirtualDub to read via a simple DirectShowSource script - but ffdshow looks like it still isn’t reading the AVCHD format 100% cleanly; there were a lot of visual artifacts - not enough to totally ruin it but enough to make me not want to use it.

Basically everything else I tried was the same as MediaCoder - using some variant/combination of ffmpeg, x264, mplayer, mencoder - all came close but didn’t cut the mustard.

The (sadly closed source) solution

In the end I just shelled out for a copy of the CoreAVC Professional Edition. Installed that, fired up VirtualDub and AviSynth and (using DirectShowSource again) had it working perfectly. In about 10 minutes I had a beautiful looking HD xvid encoded that made the previous weeks of effort looking for an open source solution completely wasted.

The verdict

I’m by no means a video expert, so I might have missed something. I think I know more than the average bear though, so I figure if I can’t get it working, most people aren’t going to have much of a shot.

I was about to give up and buy Nero Vision, which I think is one of the few commercial products that a) doesn’t COMPLETELY suck, b) isn’t ridiculously priced, and c) actually fucking works, but the CoreAVC option integrates happily and more effectively with the rest of my OSS tools, so I think that’ll be what I do for a while.

If you’re trying to do something similar with OSS tools, I’d give up for now and drop the ~US$15 on CoreAVC and use VirtualDub and/or other solutions that can read from a DirectShow source.

I’ll be keeping an eye on it; I’m sure it’s only a matter of time before MediaCoder (or its dependent components) are updated to work effectively.

I’ve decided to make my various Windows installs boot up, automatically log in, then lock themselves. This is because I have tired of turning my PC on, waiting for it to boot up, then logging in and waiting about the same amount of time for the login process to go through.

A simple script to lock a Windows install - throw this into a .vbs file and double click it and you’re done:


Set shell = WScript.CreateObject("WScript.Shell")
shell.Run "rundll32.exe user32.dll,LockWorkStation",1,false


We’ve just set up a Zimbra server as a trial in the office to use as a groupware solution - hopefully avoiding the Microsoft tax of Exchange.

First glance seems good, but it really highlights the deficiencies of the open source world in this area - Thunderbird/Lightning/Sunbird need a lot of work to catch up to Outlook/Exchange. Looks like they’re on the right track though.

Anyway, the first thing I tried to do was connect my Thunderbird/Lightning install with the Zimbra server. The documented method on the official wiki only included reference to connecting to your calendar via ICS using the webcal:// format.

Not really sure what it’s doing on the back end there, but I immediately ran into problems deleting events - a quick Google indicates many other users have the same problem. Fortunately in some other forum post (which I can’t find any more) I saw reference to Zimbra getting CalDAV support, which seems to work fine.

I’ve updated the wiki page above, but figure it’s worth saying again because it doesn’t appear to be widely known.

The CalDAV URL for your calendar is just:

http://zimbra.url.goes.here/dav/username/Calendar

It should possibly also be noted that you cannot map your Tasks in this way - ICS seems to be the only way to do that.

Google’s LatLong blog has announced Google MapMaker, their foray into community-driven mapping for the Google Maps site.

It’s only open for editing in a few countries at the moment - places like Iceland, Jamaica, and the Barbados, which we can assume have limited coverage already. It seems safe to assume this will be expanded in the near future to include more areas.

The OpenGeoData blog has a few scathing things to say about it:Google have launched MapMaker, a kind of faux OpenStreetMap where they own all the data and you’re only allowed to map in certain Freedom Of Speech Zones.

Like Knol, the mooted ‘wikipedia killer’, Google refuse to acknowledge existing communities, trample on their hard work and lack the mindset to engage with an open project. The OpenStreetMap project is a ridiculously awesome open collaboration between people all around the world that are trying to solve the problem of outdated and expensive maps for GPS systems.

They have a Google Maps-esque system where you can look at the maps online, as well as a comprehensive Wiki that explains how to use the maps. Obviously there’s a lot of incentive for the normal consumer GPS devices to not be readily compatible with these things as they no doubt make a killing selling you maps - but hopefully this sort of data will inevitably become the norm.

Check out this animation showing how the level of available maps has grown over the last few years (it’ll look like a black page for a while, but give it a second - it’s loading).

As soon as time permits I would love to get involved; if not directly then certainly by looking at some sort of sponsorship or competition via AusGamers to get other people interested that might have a bit more spare time than I do.

In the meantime, if you’re going to spend your time working on maps that you want to make available, I’d strongly encourage you to give Google a miss on this one. Putting it on OpenStreetMap means your work will be available to be used much more freely - all work is licensed under a Creative Commons license.

Why doesn’t Google just make all their maps available under Creative Commons? Well, the most likely explanation is that they’re not Google’s maps - they presumably don’t own them outright and just have a license to use them on their site.

That said, there’s nothing stopping them from using the OpenStreetMaps system as a supplementary data system and encouraging people to contribute to it - well, other than the fact that this way they get to own all the data and use it as they see fit. Maybe they’ll open it up in other ways later down the track though.

While I was staring at the output of an Apache server-status page from a server that was getting smashed by hundreds of users simultaneously downloading the latest Ubuntu release, I was thinking about how a web server reads files off the disk.

When multiple clients are connected all downloading the same file - as often happens, given that we host many popular files - they’re all downloading at varying speeds (depending on their Internet connection and server load), so the web server is reading different parts of the file off the disk and sending it to heaps of different users, all at different rates.

This results in a lot of seeking as the hard drives spin furiously to try and keep up with the demand from the HTTP server, which in turn is trying desperately to service the requests of users. This turns into load, which turns into slower downloads.

Our experience is that - under peak usage in general conditions for a typically configured web server - users can suck down files much faster than we can send them out. Even though we have bitchin’ servers on awesome connections, it only takes so many cable users with their 30mbit connections trying to suck down at full capacity before we’re hitting the wall - bear in mind 30 such users will use around 900mbit of bandwidth!

The obvious solution to this problem is to stuff the file into RAM and serve it from there. There are many ways to do this - RAM drives, mmapping with an Apache module, etc. This is probably the best way, but requires you have a buttload of RAM available to do it for large files.

The Crazy Idea

The crazy idea I was thinking about was - instead of having the disks bottleneck up the webserver from all the different clients downloading at different speeds in different parts of the file, you come up with a way to synchronise the file reads across user downloads.

The desired outcome of this idea is to reduce the overhead on the disks by sacrificing the download speed of some specific users. Obviously, such a method would result in users with faster downloads getting worse performance - but there exists a point at which the server is loaded so heavily they’re getting slow downloads anyway.

The complicated part (at least, the only complicated part I can think of - this idea might completely suck for other low-level reasons I’m not aware of/haven’t considered) would be keeping track of all the users, their download speed, and the current “position” of their download (ie, the amount of bytes they’ve already been sent), then slowing down the faster connections to bring them in line with the slower ones - so the web server is only reading from the disk ONCE, but sending it to MULTIPLE users.

With some clever programming it seems feasible to me that you could come up with rough user groups as well - for example, users downloading between 50-75 kbytes/sec could all be lumped together and set to download at 50kbytes/sec, users downloading between 200 and 300kbytes/sec could be grouped and set at 200 kbytes/sec, and so on.

Optimally, such a system would be smart enough to only turn itself on when it detects disk activity reaching a certain threshold - so under normal circumstances everyone is downloading at full speed, but once processes start getting stuck waiting for I/O, this magical new mode would get enabled, multiple connections for the same file in the same speed categories would start syncing, and then disk wait times would go back down.

I would love to know if such a system has already been implemented in an existing web server (I’ve had a quick search and can’t find anything like it), or if there’s any reasons why such a system would be impossible to implement - or if indeed it’s just a dumb idea.

Ever got one of these? An email of the form “Person X would like to recall the message, [subject of the message goes here]“?

You’ll probably get something like this a few minutes after someone has just sent you an email you weren’t supposed to get. This happens semi-regularly - I’ll get a game press release that wasn’t supposed to be out in the wild yet, or someone has just sent a message to 300 people and put them all in the Cc: list instead of the Bcc: list… there’s a lot of great screw-ups I could refer to.

While this is, on the surface, just a hilarious artifact of newbs using computers, it actually is demonstrative of a pretty serious problem - controlling accidental information spread in an increasingly digital world.

A single mis-addressed email these days can bring down businesses and sway entire markets, and trying to control confidential information is something that companies are taking more and more seriously (but still probably not seriously enough to make a big difference).

Fortunately, open source has got you covered. Some clever students over at Carnegie Mellon University have created an extension for Thunderbird (the free, open source email client from the Mozilla team) that attempts to help control the spread of information by helping you make sure you’re sending emails to the right spot.

The extension, called Cut Once, learns who should be and shouldn’t be receiving emails that you’re sending (through some sort of document word count analysis). Once it has been trained, when you go to send an email it will check your recipients and advise you if there’s someone on there that perhaps shouldn’t be.

It will also suggest recipients that you might want to add - something which I feel would be less useful for my line of work, but possibly useful for others.

Sun are reportedly working on a new open source, royalty free codec for video.

Detailed at the Sun Labs Open House event in Menlo Park, Calif., the project is called Open Media Stack or the Open Media System. It was derived out of Sun’s Open Media Commons initiative for development of royalty-free and open solutions for digital content.

No timeframe has been announced, and the article makes some mention that it is based on “H.26x technology”.

If true, this is great news - under the condition that it is truly open. There are a few problems with the current major video codecs - most notably that they’re tied up heavily with patents. So heavily, in fact, that it will be interesting to see how they’re going to go about this without getting smashed at some point with a patent infringing suit, given how much revenue is derived from licensing of these video patent portfolios.

A lot of fuss is made about codecs like Xvid and H.264 because they’re “open”. This is a common misconception - they’re not open, they just have open source implementations. To actually use these codecs commercially (for example, building a hardware device that can play them) requires paying licensing fees (usually to the MPEG Licensing Authority, who collects the fees and pays royalties for most of the video codec-related patents).

So, a truly open video codec will certainly shake things up a bit, perhaps freeing the world from the clutches of those few companies that own the video market (two of the major codecs in use on the web today - On2 and h264, both used in Flash videos like you get on YouTube, are protected heavily by patents and require significant licensing fees to do anything with them), providing more choice and more freedom for users.

It should also be noted that the BBC tried (or are trying) a similar thing with their Dirac codec, but it never gained traction - presumably because of the availability of other “free” high quality codecs like Xvid.

One of the big complaints about Firefox amongst hardcore nerds has been the memory leaking issue (despite frequent rebuttal that it is by design, or whatever). The Mozilla guys have made this a big focus of Firefox 3 and the new beta 4 shows the improvements mentioned in this post by Pavlov, one of the developers. If you want to test beta 4 without blowing away or risking your current Firefox installation check out the portableapps.com build, which will let you run it independently.

It is noticeably faster for me and definitely is looking nice. Not sure if I’m keen on the new address bar, but I’ll give it a go.

I use the command line in Windows all the time, for a lot of different things. Back in 1998, I realised one of my really common operations was to copy a file to the clipboard (using Windows Explorer; either right-click-copy or select the file/folder and hit CTRL-C) and then paste it into a destination folder.

As I always had a command prompt open, I’d often do this by changing to the destination directory, typing ’start .’ to invoke an Explorer Window in that directory, then hitting CTRL-V. I realised it’d be awesome if I could cut out that ’start .’ step, and thought a command line tool to allow me to paste files directly would be really handy.

So, I wrote a Win32 command line tool that did exactly that. It was really hacky, but it worked - I’ve used it almost every day since to copy and move files around.

10 years later, I realise I never even put it up for people to download or even offered it to anyone. So, here it is - cpaste for Windows.

I’ve compiled up a version of Thomas Akin’s 2hash for Windows for my own nefarious deeds and thought other people might find it handy. You can download it (including source, which is mostly unmodified) right here. It can crash unexpectedly if you don’t provide proper input; I haven’t bothered carefully sanitising user input just yet. If anyone else other than me actually uses it I’ll put some more effort into it.