MySQL Server fails to upgrade in Debian/Ubuntu

Just did a relatively normal ‘apt-get upgrade’ on my simple WordPress HHVM test server and got a horrible bunch of errors, leaving it in a non-working state:

E: Sub-process /usr/bin/dpkg returned an error code (1)
A package failed to install. Trying to recover:
Setting up mysql-server-5.1 (5.1.54-1ubuntu4) …
start: Job failed to start
invoke-rc.d: initscript mysql, action “start” failed.
dpkg: error processing mysql-server-5.1 (–configure):
subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of mysql-server:
mysql-server depends on mysql-server-5.1; however:
Package mysql-server-5.1 is not configured yet.
dpkg: error processing mysql-server (–configure):
dependency problems – leaving unconfigured
Errors were encountered while processing:
mysql-server-5.1
mysql-server

There are many reports of this on various forums and blogs with many different solutions. Here’s mine.

In my case, MySQL logged the following error:

May 4 07:04:22 trog mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
May 4 07:04:22 trog mysqld: 140504 7:04:22 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
May 4 07:04:22 trog mysqld: 140504 7:04:22 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
May 4 07:04:22 trog mysqld: 140504 7:04:22 [Note] Plugin ‘FEDERATED’ is disabled.
May 4 07:04:22 trog mysqld: 140504 7:04:22 InnoDB: The InnoDB memory heap is disabled
May 4 07:04:22 trog mysqld: 140504 7:04:22 InnoDB: Mutexes and rw_locks use GCC atomic builtins
May 4 07:04:22 trog mysqld: 140504 7:04:22 InnoDB: Compressed tables use zlib 1.2.7
May 4 07:04:22 trog mysqld: 140504 7:04:22 InnoDB: Using Linux native AIO
May 4 07:04:22 trog mysqld: 140504 7:04:22 InnoDB: Initializing buffer pool, size = 128.0M
May 4 07:04:22 trog mysqld: InnoDB: mmap(137363456 bytes) failed; errno 12
May 4 07:04:22 trog mysqld: 140504 7:04:22 InnoDB: Completed initialization of buffer pool
May 4 07:04:22 trog mysqld: 140504 7:04:22 InnoDB: Fatal error: cannot allocate memory for the buffer pool
May 4 07:04:22 trog mysqld: 140504 7:04:22 [ERROR] Plugin ‘InnoDB’ init function returned error.
May 4 07:04:22 trog mysqld: 140504 7:04:22 [ERROR] Plugin ‘InnoDB’ registration as a STORAGE ENGINE failed.
May 4 07:04:22 trog mysqld: 140504 7:04:22 [ERROR] Unknown/unsupported storage engine: InnoDB
May 4 07:04:22 trog mysqld: 140504 7:04:22 [ERROR] Aborting
May 4 07:04:22 trog mysqld:
May 4 07:04:22 trog mysqld: 140504 7:04:22 [Note] /usr/sbin/mysqld: Shutdown complete
May 4 07:04:22 trog mysqld:

 

I’ve bolded the important lines – in my case, MySQL was starting and trying to allocate some memory for InnoDB, but it was failing. So when the package manager updated MySQL and restarted it, it would simply exit as failed, returning failure as part of the update process.

I solved this hackishly (but easily) by stopping a memory-hungry process on the server (HHVM), then re-running the upgrade. Without HHVM running there was plenty of spare memory, the update was able to successfully restart MySQL, then I just restarted HHVM.

This is of course indicative that this system probably doesn’t have enough memory – but it’s a small-scale test environment. So if you’re looking for a cheap easy fix when Debian/Ubuntu fail to upgrade MySQL, this might work.

Initial Binary Lane Feedback: Positive

So far the feedback for the newly launched Binary Lane has been really positive:

Feedback on Whirlpool has been similarly positive:

bl-do

bl-wp-1

… and also a good thread on LowEndTalk.com, a developer-focused community for infrastructure services.

There’s still a lot of work going on behind the scenes. New features are still be developed – most recently, a new BYO ISO system, allowing people to install their own operating systems, including things we haven’t supported before like FreeBSD.

Stay tuned.

Binary Lane – a new hosting service from Mammoth Media

I’m proud to announce that the Mammoth team has launched a new product this week – Binary Lane.

Binary Lane is our new take on virtual private server hosting. Our service Mammoth Networks has been in operation since 2010 and has grown steadily to become a respected name in VPS hosting in Australia, and the mPanel – the software that drives this service – has matured and evolved until it has become, in our humble opinion, one of the best platforms for managing virtualised servers on the planet.

Binary Lane features several significant differences that we thought justified a new brand. The main changes:

  • Binary Lane uses KVM for its virtualisation back end, instead of Xen. This gives us a bunch of great new features – including live migrations, allowing us to almost instantly move guests between host nodes.
  • We’re in a new data centre, right here in Brisbane. We’ve had a great relationship with PIPE (formerly SOUL, formerly Comindico) and have been in that facility for years, but we wanted a local presence in our own city. The new NEXTDC data centre and their DC-as-a-service model really fits what we want to do.
  • New IP transit. Again our PIPE/TPG/SOUL connectivity has been awesome, but getting transit from a single carrier has reduced our flexibility a little bit. We’re working with the unbelievably great team at APEX Networks to provide our transit.
  • All SSD storage. SSDs are the future for server systems. The price has reduced to the point where building entire storage systems out of SSDs is feasible for a huge variety of application types. While bulk data storage is still a little bit away, we feel that the performance offerings of SSDs are impossible to ignore.

The Binary Lane brand is also something that gives us a little more freedom internationally – Mammoth Networks was, for various reasons, somewhat restrictive in the United States :) Startups, let that be a lesson to you – even if your target market is Australia initially, always think about the rest of the world when you start building your brand!

Of course Mammoth Networks isn’t going anywhere – it will still continue to receive all the attention it gets currently. We’ll be doing updates to mPanel that may appear on one or the other site first, but eventually both sites will be updated with any new features that are added.

Big congratulations to our awesome development and operations teams at Mammoth for their really hard work on this project and special mad props to our technical director Nathan O’Sullivan, whose tireless efforts leading the charge have again resulted in shipping another great product.

Oh yeh – we’re so confident you’ll love Binary Lane that we’re offering a free seven day trial. Go check it out, marvel at the speed, wonder at the power of mPanel, and be amazed.

Possible Unintended Consequences of Legalising Drug Use: Faster Cryptocurrency Uptake

A few states have recently legalised marijuana use in the United States. This is weird for everyone. Not only is it a complete reversal on drug policy, but it’s still actually illegal at the federal level. So while its legal in certain states, it’s still technically a federal no-no to be using it. This is confusing as hell, although it didn’t stop Colorado’s first day from being a huge event which raked in a cool $1 million.

A great example of the confusion popped up on the New York Times today, as they covered the fascinating story of how banks in these states are basically refusing to offer banking services for these (legal) providers of marijuana. US banks, of course, have an impeccable reputation when it comes to deciding what to do with money, so this of course comes as no surprise.

The first thing that I thought of when I read this article (a little sarcastically, because I’m so sick of hearing about it) was “well, I guess they could just use BitCoin, amirite?” But as I thought about it some more, I started thinking that it’s probably not a terrible idea for these guys.

Compared with the difficulty of dealing with putting cash into brown paper bags or Tupperware to move it around and the scary issues of security logistics entailed, the idea of using a freely available and accessible cryptocurrency seems compelling to say the least.

Sure, it’s still nowhere near as widespread as Visa or Mastercard. There’s the possibility of vulnerabilities. People don’t really know how to deal with keeping worthless digital assets secure, let alone a digital asset that contains real money. But it’s here, now, and it means that these legal(ish) vendors could go about their business with one less headache.

Well, at least until tax time when they have to pay the IRS – it seems likely that they’re one entity that is not going to be accepting payments in cryptocurrencies any time soon. I guess the drug vendors will have to deal with the problems of having a massive pile of super-secret untraceable anonymous digital cash!

Except, of course, this isn’t really how cryptocurrencies work (at least, those based on the BitCoin blockchain system). Intentionally trying to hide their income from the feds is probably just as dangerous as hauling around large amounts of small bills. The instability of most of the cryptocurrencies is probably the biggest concern; storing your tax payment in a currency that might halve in value by tax time would be risky to say the least. But if they wanted to make life more difficult for the IRS, switching to cryptocurrency would probably do that in the short term, while saving them all sorts of other hassles.

In short – talk about your disruptive technologies! Banking – for what are probably good reasons – is failing to keep up with the pace of legislative change here. Whether or not cryptocurrencies actually pose a threat to

By the time I finished writing this, the following articles appeared: Colorado Marijuana Dispensary Uses Bitcoin to Evade Federal Laws, and back home, businesses in Australia are starting to accept BitCoin.

The UK Is Eating Our Lunch

The United Kingdom has a history of building reasonably decent empires.

The recent news that Australian tech company Atlassian is moving their base of operations to the UK is depressing, but completely unsurprising. Sure, Australia’s an expensive place to do business – but the UK are going out of their way to make themselves ludicrously sexy for technology companies.

I first found out about the UK tax incentives when some friendly people from the UK consulate approached me after I’d participated in a panel at the NICTA TechFest 2013 (along side Matt Barrie from Freelancer.com and Professor Stuart Feldman from Google – the guy that created ‘make’, no big deal), discussing ways to address the ICT skills shortage in Australia. At Parliament House. In Canberra. Right before Prime Minister Julia Gillard took the stage to talk about innovation in Australia.

Seriously, at an event to try to help draw attention to technology in Australia these guys came right up to me and basically started talking about all the reasons why I should be trying to move our company to the United Kingdom. I was impressed by their audacity – but I was even more impressed by what they were offering.

The people from the consulate made a good pitch. There’s a 225% tax credit for research and development, they have a thing called the Patent Box offering significant tax breaks on profits made from patents, they’ve got a tech-friendly ‘entrepreneur visa’ to make it easy to get there, decreases in corporations tax, and – oh, what else is there? Oh yeh, it’s right next to EUROPE which has like a jillion people, so a giant market.

London is great.

Shortly after that, they got in touch and asked if we’d be interested in talking to some of their people in Brisbane (I imagine they try to have tendrils everywhere, so they have well-dressed Englishmen scattered strategically around the country, ready to be dispatched to technology companies at a moments notice). A couple of nice chaps turned up and we had a chat where they went into some more detail about what is involved in moving to the UK.

They also offered to do a bit of an analysis for us to figure out what the market was like in our specific area. This is basically a free service they provide – looking up a bunch of stuff and doing some (fairly basic, but helpful) market analysis for you. Shortly afterwards they sent through a nice report with heaps of detail about telcos and data centres and that sort of thing, with a bit of a reminder about all the other perks.

A few months after that I got an invite to a breakfast put on by London and Partners, the official promotional organisation for the city of London. I went along just to check it out, but it was mostly a repeat of what I already knew – just with a bit more focus on London.

Still, I was again impressed not only by the pitch (PDF), but the simple fact that the United Kingdom is currently focused like a laser on making their country attractive to technology companies. They know the future is about technology and are displaying adaptability accordingly.

It’ll be sad to see Atlassian go. It’ll be sad to watch others follow. It’ll be sad if Australia doesn’t treat this as a wake-up call.

Didn’t they know that the only unhackable computer is one that’s running a secure operating system, welded inside a steel safe, buried under a ton of concrete at the bottom of a coal mine guarded by the SAS and a couple of armoured divisions, and switched off?

— On Secure Computers – From the Atrocity Archives

Do Not Let Your Nexus Device Fill Up the Disk

A few weeks ago my Nexus 7 tablet (the original release) started slowing down a bit. I had no idea why, but it was still mostly useful – until I stuffed heaps of video onto it for a trip overseas. After that it slowed to a crawl – almost completely unusable.

Some Googling indicated the problem was probably related to the storage being filled, and then finally I stumbled across this article which confirms many people have been suffering from this same issue.

The short version is that the original Nexus devices/Android operating systems don’t properly TRIM the SSD. This is the sort of techno-speak I would have reveled in learning all about when I was younger, but now all I care about it how to make this thing work again. Basically, it manifests itself in super-slow disk performance – you can see really high IOWAIT if you use something like ‘top’, or if you run a benchmark you’ll see really bad performance like this:

2013-12-02 19.05.10 - Copy

As of some recent Android release, it is supposed to automatically do this TRIM stuff once every 24 hours – but if it’s doing it on my device, it’s not making any difference.

It seems there are only two real options:

1) Factory reset.

2) Root the device and run the TRIM manually using something like LagFix.

Both of these options are pains in the ass to various levels, but the device is basically unusable in this state. I ended up doing the factory reset, thinking it would be less of a hassle.This seemed to fix it up and now I’m looking at the following speeds – note the increase in sequential write:

2013-12-03 18.16.02 - CopyHowever, if you can avoid it, it seems safest to not let your devices drop below ~3.5GB free (at least on a 16GB device).

I Want to Pay for Dropbox – But It Might Make Me Hate Myself

Dropbox is one of the very few applications I’ve installed that has completely changed the way I use computer systems.

Under most circumstances, I no longer have to think about having to deal with the irritating sending and receiving of files, or stuffing them onto some other system to be retrieved later. I can just save everything onto my local disk – exactly as I’d like to – and know that it will magically pop up at some later point on every other PC that I own. I take photos on my phone knowing they’ll be stuffed onto Dropbox for later retrieval on my PC – indeed, I no longer even think about “copying photos off my phone”, because it just happens.

There are, of course, a few limitations. For example, it’s hard to do this with large volumes of data, simply because the upstream on most broadband plans is woeful. In those cases typically reaching for the USB disk or stuffing bytes onto my phone is a better alternative.

Of course, the other limitation is the few paltry gigabytes of storage you get on the free plan. If you’re dedicated though, it’s pretty trivial to boost this by quite a bit – referring friends, linking devices, all that sort of stuff. At the time of writing I have 4.2GB available on my Dropbox, without spending a cent.

And now, perhaps inevitably, I find myself in the situation of wondering why the hell all my files aren’t on Dropbox. It’s almost like they had some sort of insidious plan to get me hooked on their awesome system by giving me a taste for free.

Unfortunately I don’t really want to use Dropbox. Not really because I don’t want to pay for it, but because I have never really liked their security model. I want my files to be encrypted/decrypted client side.

I suspect the main reason they don’t want to offer this is because it would remove a lot of the basic functionality that the vast majority of users take advantage of regularly – the ability to access and share files quickly and easily via the web interface in particular. Not to mention the support nightmare that would certainly ensue when those users lose their encryption keys and wonder why all their files are now a bunch of unrecoverable gibberish.

In the post-Snowden world this is possibly an even bigger deal. I don’t really have concerns that faceless government agents are going to be poring through my files – but it’s even clearer that you ultimately need to be responsible for the security of putting your data online.

I’ve tried a few of the European alternatives to Dropbox – Wuala and SpiderOak most recently. Their security policies look good, they (appear to) use client side encryption, and they’re located in Europe, so I can rest somewhat comfortably knowing they’re not subject to secret NSA orders or whatever.

With the possible exception of Google Drive (which of course is subject to the same woes as Dropbox), the other services I’ve tried I found almost completely unusable compared to the elegance, simplicity, and sheer Just Workiness of Dropbox. I tried – I really did. I wanted to like them. I’m not sure if it’s all that security stuff just getting in the way of making it a good experience, but they just feel clunky and awkward to use, painful to set up, and I was generally just thinking “why am I doing this?” the whole time.

I’m a big believer in voting with your wallet. It’s not like there aren’t other options. But Dropbox is just so damn convenient in so many different ways that I can feel myself slowly caving and abandoning any lofty principles just so I can go back to Getting Shit Done.

There are two things that Dropbox could do to get me off the fence immediately.

1) Introduce client side encryption/decryption into the Dropbox client. While it remains closed source I can imagine many would still (rightfully) be hesitant to trust it (how would you know they’re not capturing your encryption keys?), but a nod in that direction would be enough for me.

2) Introduce an option to limit storage of my files on Amazon clouds in different regions. I am not intricately familiar with how Amazon’s cloudy stuff works, but it seems that this would not be a complicated feature. Allow me to opt to have my files stored on S3 within particular geographic regions. I can imagine this would be a big deal for many government services who might want to use Dropbox but might be subject to limitations on where their data can be physically stored, and for the security nerds, getting out of the reach of the NSA (yes, yes, subject to their ability to compromise any site anyway), it would be a neat service.

What I suspect I’ll end up doing is signing up for a plan and then encrypting all my stuff locally with gnupg and treating it more like a backup archival system rather than a live working filesystem.

On the NSA, Security and Privacy on the Internet

A lot of terrifyingly smart people have written extensively about the impact Snowden revelations about the NSA spying on what appears to be basically everyone in the world, including the citizens on the United States.

My thoughts on the subject are fairly predictable, but once again I’m struck by the prescience demonstrated in Neal Stephenson’s amazing Cryptonomicon. I first read it shortly after it was published in 1999, and while I loved the techno-thriller aspects of the story, I didn’t really understand the scope of some of the more political issues.

Having just re-read the book, I thought the following paragraph was particularly relevant when considering what is happening now:

Many Net partisans are convinced that the Net is robust because its lines of communication are spread evenly across the planet. In fact, as you can see from this graphic, nearly all intercontinental Web traffic passes through a small number of choke-points. Typically these choke-points are controlled and monitored by local governments. Clearly, then, any Internet application that wants to stand free of governmental interference is undermined, from the very beginning, by a fundamental structure problem.

Image Data Only Hashing of JPEG Files

As part of a small project to verify backups, I came across a case where I had two photos that looked identical but with different EXIF data.

The backup verification system (correctly) flagged these as two different files – as the SHA1 file hashes were different. However, the actual photo was – as far as I could tell – absolutely identical, so I started looking to see if there was a way to verify JPEG files based on the image data alone (instead of the entire file, which would include meta stuff like the EXIF data).

A quick look around revealed that ImageMagick has a “signature hash” function as part of ‘identify‘, which sounded perfect. You can test it like so:

identify.exe -verbose -format “%#” test.jpg

At first glance this solved the problem, but testing on a few systems showed that I was getting different hashes for the same file – it looked like different versions of ImageMagick return a different hash. I’ve asked about this on their forum and was told that the signature algorithm has changed a few times – which makes it sort of useless if compatibility across platforms is required.

After looking around a bit more for alternative I found the (possibly Australian made?) PHP JPEG Metadata Toolkit, which (amongst many other things) includes a get_jpeg_image_data() function which (so far) seems to work reliably across systems. Pulling the data out and running it through SHA1 gives a simple usable way to hash the image-only data in a JPEG file.

1 2 3 26  Scroll to top