Do Not Let Your Nexus Device Fill Up the Disk

A few weeks ago my Nexus 7 tablet (the original release) started slowing down a bit. I had no idea why, but it was still mostly useful – until I stuffed heaps of video onto it for a trip overseas. After that it slowed to a crawl – almost completely unusable.

Some Googling indicated the problem was probably related to the storage being filled, and then finally I stumbled across this article which confirms many people have been suffering from this same issue.

The short version is that the original Nexus devices/Android operating systems don’t properly TRIM the SSD. This is the sort of techno-speak I would have reveled in learning all about when I was younger, but now all I care about it how to make this thing work again. Basically, it manifests itself in super-slow disk performance – you can see really high IOWAIT if you use something like ‘top’, or if you run a benchmark you’ll see really bad performance like this:

2013-12-02 19.05.10 - Copy

As of some recent Android release, it is supposed to automatically do this TRIM stuff once every 24 hours – but if it’s doing it on my device, it’s not making any difference.

It seems there are only two real options:

1) Factory reset.

2) Root the device and run the TRIM manually using something like LagFix.

Both of these options are pains in the ass to various levels, but the device is basically unusable in this state. I ended up doing the factory reset, thinking it would be less of a hassle.This seemed to fix it up and now I’m looking at the following speeds – note the increase in sequential write:

2013-12-03 18.16.02 - CopyHowever, if you can avoid it, it seems safest to not let your devices drop below ~3.5GB free (at least on a 16GB device).

I Want to Pay for Dropbox – But It Might Make Me Hate Myself

Dropbox is one of the very few applications I’ve installed that has completely changed the way I use computer systems.

Under most circumstances, I no longer have to think about having to deal with the irritating sending and receiving of files, or stuffing them onto some other system to be retrieved later. I can just save everything onto my local disk – exactly as I’d like to – and know that it will magically pop up at some later point on every other PC that I own. I take photos on my phone knowing they’ll be stuffed onto Dropbox for later retrieval on my PC – indeed, I no longer even think about “copying photos off my phone”, because it just happens.

There are, of course, a few limitations. For example, it’s hard to do this with large volumes of data, simply because the upstream on most broadband plans is woeful. In those cases typically reaching for the USB disk or stuffing bytes onto my phone is a better alternative.

Of course, the other limitation is the few paltry gigabytes of storage you get on the free plan. If you’re dedicated though, it’s pretty trivial to boost this by quite a bit – referring friends, linking devices, all that sort of stuff. At the time of writing I have 4.2GB available on my Dropbox, without spending a cent.

And now, perhaps inevitably, I find myself in the situation of wondering why the hell all my files aren’t on Dropbox. It’s almost like they had some sort of insidious plan to get me hooked on their awesome system by giving me a taste for free.

Unfortunately I don’t really want to use Dropbox. Not really because I don’t want to pay for it, but because I have never really liked their security model. I want my files to be encrypted/decrypted client side.

I suspect the main reason they don’t want to offer this is because it would remove a lot of the basic functionality that the vast majority of users take advantage of regularly – the ability to access and share files quickly and easily via the web interface in particular. Not to mention the support nightmare that would certainly ensue when those users lose their encryption keys and wonder why all their files are now a bunch of unrecoverable gibberish.

In the post-Snowden world this is possibly an even bigger deal. I don’t really have concerns that faceless government agents are going to be poring through my files – but it’s even clearer that you ultimately need to be responsible for the security of putting your data online.

I’ve tried a few of the European alternatives to Dropbox – Wuala and SpiderOak most recently. Their security policies look good, they (appear to) use client side encryption, and they’re located in Europe, so I can rest somewhat comfortably knowing they’re not subject to secret NSA orders or whatever.

With the possible exception of Google Drive (which of course is subject to the same woes as Dropbox), the other services I’ve tried I found almost completely unusable compared to the elegance, simplicity, and sheer Just Workiness of Dropbox. I tried – I really did. I wanted to like them. I’m not sure if it’s all that security stuff just getting in the way of making it a good experience, but they just feel clunky and awkward to use, painful to set up, and I was generally just thinking “why am I doing this?” the whole time.

I’m a big believer in voting with your wallet. It’s not like there aren’t other options. But Dropbox is just so damn convenient in so many different ways that I can feel myself slowly caving and abandoning any lofty principles just so I can go back to Getting Shit Done.

There are two things that Dropbox could do to get me off the fence immediately.

1) Introduce client side encryption/decryption into the Dropbox client. While it remains closed source I can imagine many would still (rightfully) be hesitant to trust it (how would you know they’re not capturing your encryption keys?), but a nod in that direction would be enough for me.

2) Introduce an option to limit storage of my files on Amazon clouds in different regions. I am not intricately familiar with how Amazon’s cloudy stuff works, but it seems that this would not be a complicated feature. Allow me to opt to have my files stored on S3 within particular geographic regions. I can imagine this would be a big deal for many government services who might want to use Dropbox but might be subject to limitations on where their data can be physically stored, and for the security nerds, getting out of the reach of the NSA (yes, yes, subject to their ability to compromise any site anyway), it would be a neat service.

What I suspect I’ll end up doing is signing up for a plan and then encrypting all my stuff locally with gnupg and treating it more like a backup archival system rather than a live working filesystem.

On the NSA, Security and Privacy on the Internet

A lot of terrifyingly smart people have written extensively about the impact Snowden revelations about the NSA spying on what appears to be basically everyone in the world, including the citizens on the United States.

My thoughts on the subject are fairly predictable, but once again I’m struck by the prescience demonstrated in Neal Stephenson’s amazing Cryptonomicon. I first read it shortly after it was published in 1999, and while I loved the techno-thriller aspects of the story, I didn’t really understand the scope of some of the more political issues.

Having just re-read the book, I thought the following paragraph was particularly relevant when considering what is happening now:

Many Net partisans are convinced that the Net is robust because its lines of communication are spread evenly across the planet. In fact, as you can see from this graphic, nearly all intercontinental Web traffic passes through a small number of choke-points. Typically these choke-points are controlled and monitored by local governments. Clearly, then, any Internet application that wants to stand free of governmental interference is undermined, from the very beginning, by a fundamental structure problem.

On New Entrepreneurialism

[This post is a work in progress and is not finished]

I just got back from an Entrepreneurs Lunch at PwC, the title of which was “Offshoring & Outsourcing Lunch with Scott Linden Jones & Mike O’Hagan”. I’ve been to a few PwC events; they’re always well organised and very interesting, but this is the first one I’ve walked out of thoroughly depressed.

On the surface, it might seem like it’s just because I co-founded a Brisbane-based software development company and I’m worried about overseas developers eating our lunch. That is certainly true, but it’s not the whole reason I came out of the event wanting to go straight to a pub and cry into a pint. Outsourcing software development is nothing new; we feel that the value we add by being an on shore solution more than makes up for the difference in price. (And of course, some places that offshore custom development aren’t super interested in passing on the cost savings back to their clients.)

The main reason was that I was disappointed about this ‘New Entrepreneurialism’. At an entrepreneurs lunch, the main focus on being entrepreneurial was basically stripping an existing business down to its component parts, identifying the bits that could be easily compartmentalised, and then figuring out how to make people overseas do them for a fraction of the cost.

Anyone that’s spent any time on developer/tech-heavy communities, especially American ones – like Slashdot – will have seen complaints about this sort of thing before. Developers in large companies that have been doing outsourcing for a while seem to have seen this sort of process a lot, and usually end up blaming “the MBAs” for gutting the company, sending jobs to India or China, and generally making their lives suck.

So this technique isn’t new. Mike O’Hagan, the owner of the moving company Mini Movers and one of the speakers, acknowledged this. Outsourcing is something that the big companies have been doing for ages, but thanks to the Internet removing some of those barriers and increasing access to this global employment market, things have changed and now small companies can take advantage of it as well.

This is all well and good – but it this really what entrepreneurial activity looks like now? Outsourcing some business processes to third world economies and calling it “entrepreneurship” (a word which I’ve always hated) feels like it’s missing the point to me.

[ NOTES TO FINISH WRITING UP]:

  • irony of entrepreneurs all doing the same thing, instead of something new
  • Mike’s talk
    – security not a concern? what about customer data/privacy in outsourced environments?
    – noted corruption is a problem (no solution)
    – need to know who to go with
    – “same problem in Australia” with a bag full of money
    – political issues in Australia – no plan to deal with “high minimum wage”
    – Economics of outsourcing
    – prices already increasing in some outsourcing markets (implication that this sucks and is unfair and is going to mean they’re going to have to outsource their outsourcing when it gets too much and they can find a cheaper country)

Scott’s talk
– Quality of outsourced work has not been an issue (… based on one anecdote?!)
– No real data presented (many people have good stories of outsourcing success but just as obviously, but many don’t)

Other points to note:

  • Equilibrium will be probably be restored over time as markets evolve and cheaper outsourcing markets develop & become more “first-worldy”
  • Some links about offshoring as references:
    • http://forums.whirlpool.net.au/forum-replies.cfm?t=2145233#r14
    • http://slashdot.org/comments.pl?sid=4091369&cid=44570995

Image Data Only Hashing of JPEG Files

As part of a small project to verify backups, I came across a case where I had two photos that looked identical but with different EXIF data.

The backup verification system (correctly) flagged these as two different files – as the SHA1 file hashes were different. However, the actual photo was – as far as I could tell – absolutely identical, so I started looking to see if there was a way to verify JPEG files based on the image data alone (instead of the entire file, which would include meta stuff like the EXIF data).

A quick look around revealed that ImageMagick has a “signature hash” function as part of ‘identify‘, which sounded perfect. You can test it like so:

identify.exe -verbose -format “%#” test.jpg

At first glance this solved the problem, but testing on a few systems showed that I was getting different hashes for the same file – it looked like different versions of ImageMagick return a different hash. I’ve asked about this on their forum and was told that the signature algorithm has changed a few times – which makes it sort of useless if compatibility across platforms is required.

After looking around a bit more for alternative I found the (possibly Australian made?) PHP JPEG Metadata Toolkit, which (amongst many other things) includes a get_jpeg_image_data() function which (so far) seems to work reliably across systems. Pulling the data out and running it through SHA1 gives a simple usable way to hash the image-only data in a JPEG file.

Terrible Thunderbird v15.x IMAP Performance with AVG

My PC has recently been chugging a lot more than usual – massive disk activity and high CPU utilisation. Looking into it I quickly realised that it was happening whenever Thunderbird received a large bolus of new email – more than 15-20 emails within a minute or two. When I clicked on the folder with the new email, I could see in the status bar at the bottom that Thunderbird was very slowly downloading these new emails, while my disk and CPU went crazy.

Looking further I noticed that in Filemon, AVG was doing a lot of the work. Disabling AVG’s “Resident Shield” during one of these operations almost immediately fixes the symptoms – the email comes down much faster and the disk activity and CPU returns to normal.

This seemed to happen around the same time as Thunderbird v15.x was released, but I don’t want to declare that the culprit, especially as it is probably the same thing that I noticed with Microsoft Security Essentials that started happening around v11.x. I’m curious if something fundamental changed back then – either internally in Thunderbird, or perhaps within AVG – but it’s certainly possible that I’m just getting a little bit more email now and it’s just tripped my PC over the edge. I assume it has something to do with the way AVG hooks into the disk reading/writing operations – possibly Thunderbird changed something low-level there and it is simply reacting badly with how AVG does its real-time checking.

In any case, if you are experiencing massive slowdowns and system chunkiness using Thunderbird in conjunction with AVG, you can simply temporarily disable the real-time checking when getting a large number of emails. Obviously you probably don’t want to leave it off altogether.

MongoDB Fails Updating on Debian

Every so often there’s a MongoDB update on my Debian VPS that fails. The output of ‘aptitude full-upgrade’ is:

# aptitude full-upgrade
The following partially installed packages will be configured:
mongodb-10gen
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.
Setting up mongodb-10gen (2.0.5) …
Starting database: mongodb failed!
invoke-rc.d: initscript mongodb, action “start” failed.
dpkg: error processing mongodb-10gen (–configure):
subprocess installed post-installation script returned error exit status 1
configured to not write apport reports
Errors were encountered while processing:
mongodb-10gen
E: Sub-process /usr/bin/dpkg returned an error code (1)
A package failed to install. Trying to recover:
Setting up mongodb-10gen (2.0.5) …
Starting database: mongodb failed!
invoke-rc.d: initscript mongodb, action “start” failed.
dpkg: error processing mongodb-10gen (–configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
mongodb-10gen

The update works fine, but mongo just fails to start properly.

The problem in my case is simply that there’s a /var/lib/mongodb/mongod.lock file lying around from some previous process. Deleting that file and re-running the aptitude command will start it properly. (Reminder post because I keep forgetting what the problem is.)

AVG on Linux False Positives for NSIS

As of today, we’re seeing what I’m very confident are false positives in AVG running on Linux on our file servers. This has started happening after this morning’s virus database update. The database release we’re using is:

Virus database version: 271.1.1/4927
Virus database release date: Wed, 11 Apr 2012 05:55:00 +10:00

The output of avgscan is:

utils.exe |%name%=Win32/Validace_partial.nsis3|%idn%=0bcfdae664a2c000|=Win32/Validace_partial.nsis3

Files scanned : 1(1)
Infections found : 1(1)
PUPs found : 0
Files healed : 0
Warnings reported : 0
Errors reported : 0

The ‘nsis’ in the output there is presumably referring to the excellent Nullsoft Scriptable Install System (NSIS). The files I’m testing are largely game installers; when cross-checked with a file I built using NSIS it also triggers the false positive.

We are contacting AVG to report this as a probable false positive signature.

Update 3rd May 2012: AVG recommended we update to the 2012 version to fix this issue, which we did – and it fixed the problem.

Thunderbird Freezes When Deleting or Moving Email

I recently updated to the latest Thunderbird (v11.0) and was disappointed to discover that suddenly whenever I was deleting an email or moving it into a different folder, the entire application would freeze for 1-2 seconds while it processed that command.

I am fastidious about email and spend probably more time than I should ensuring everything is filed into appropriate folders (or deleted if I’m never going to look at it again). When you’re getting hundreds of emails a day, deleting and moving needs to be an operation that consumes near zero time, otherwise you’re suddenly spending way more time “doing email” than you should be. As a result, these freezes were massively irritating and caused no end of problems.

I reinstalled Thunderbird, which seemed to fix it temporarily – but before I knew it was happening again. I tried rebuilding and compacting folders – all for naught. I tried searching the Thunderbird Bugzilla looking for similar reports, but I couldn’t see anyone else having the problem.

I put up with this for a while trying various things, but eventually gave up and fired up the incredibly handy FileMon utility from the SysInternals guys to see if anything obvious was happening on the disk side of things that would account for this freeze.

Immediate pay dirt; this chunk of output in FileMon is shows the main part of what happened when I tried to move an email into a subfolder of the Inbox:

You can see there the operation started at 4:11:37pm and then the next activity was at 4:11:39pm – two seconds was roughly how long I was seeing Thunderbird freeze for.

Next step was looking at what MsMpEng.exe was – Microsoft Security Essentials. Turns out MSE was installed on my PC as part of a general system policy update at around the same time I upgraded to Thunderbird v11.0.

I tried changing the settings to see if that was indeed the cause – in MSE you just look for the Settings tab, select Real-time protection, and uncheck the ‘Turn on real-time protection’ box. Immediately Thunderbird started behaving normally with no more freezes.

Fortunately there’s an ‘Excluded processes’ option in Microsoft Security Essentials so you can add Thunderbird.exe to the list of processes to skip. This completely fixed the problem for me and now I’m back to moving and deleting emails fast as ever.