wget from Google Drive

A quick bash script to download files from Google Drive with wget.

e.g. save as gget.sh and run like:

./gget.sh 461ZQWJ2y1-zwQbzLToQsKNsLWaLwazfB outputFilename.zip
#!/bin/bash
# https://drive.google.com/file/d/[GDRIVE FILE ID]/view?usp=sharing
fileid=$1
filename=$2
wget --save-cookies=./cookie -L "https://drive.google.com/uc?export=download&id=${fileid}" > /dev/null
wget --load-cookies=./cookie "https://drive.google.com/uc?export=download&confirm=awk '/download/ {print $NF}' ./cookie&id=${fileid}" -O ${filename}

Resolving WordPress “Post-processing of the image failed” error when uploading large images

I just got the following error trying to upload an image that was 2000 pixels wide:

“Post-processing of the image failed likely because the server is busy or does not have enough resources. Uploading a smaller image may help. Suggested maximum size is 2500 pixels.”

This was confusing because the image was clearly smaller than the maximum size in the error message.

Web search for the error reveals a million different possible causes – this page has a few of the obvious ones, but none of them were my problem.

I tried increasingly slightly smaller versions of the file, then quickly realised what the issue was when it started working once the file size got below 1MB – upload limit in PHP was set to 8MB, and this was being reported in WordPress, but it was not set properly in nginx.

If I’d thought to check the nginx error log sooner I would have seen the following error:

... client intended to send too large body: 1191539 bytes

The fix is of course to just add the appropriate directive in nginx config file to align the upload size with what you have in PHP. Something like:

http {
…
    client_max_body_size 8M;
}

Using Google Colaboratory to wget Files Directly into Google Drive from Anywhere (Dropbox, AWS, VPSs, etc)

I often find myself in the situation where I have large files on various online services like Dropbox or various VPSs that I want to backup or otherwise store in Google Drive.

It seems stupid to have to download them to my desktop and then re-upload them. Good news! There’s a way to download straight into Google Drive, using Google Colaboratory.

I stole this trick from somewhere and I use it all the time to upload files directly into Google Drive.

Basically:

  1. Create a new Colab notebook
  2. Add a new cell with the following contents:
    from google.colab import drive
    drive.mount('/content/drive')
  3. Run the code. It will prompt you to authenticate this new ‘application’ against Google Drive so it can access your files.
  4. Create a new cell with the following contents:
    !wget -P /content/drive/My\ Drive https://example.com/test.zip
  5. Run the code. It will download the file, using wget, directly from the requested URL, showing you the live progress.
  6. For bonus points, you can also check the hash of the file once it has downloaded to make sure it has the contents you’re expecting (note here that you can also reference ‘My Drive’ like ‘MyDrive’?):
    !sha1sum /content/drive/MyDrive/test.zip

WordPress SSO with Google Workspace SAML and OneLogin SAML SSO Plugin

I recently embarked on a journey to simplify logins to a client’s network of WordPress sites with Single Sign-On (SSO). I didn’t really care what identity provider was going to be used, but they also used Google Workspace which I knew had support for SAML. I also figured that, being Google Workspace, the support for it would be well established, with lots of nice and simple clicky interfaces.

Nope.

Until this point my only real experience with SSO had been as a consumer; I’ve had many accounts that have implemented some form of SSO (often based on Google accounts) and it has been relatively seamless. But trying to implement it myself from a cold start I found frustrating.

I first tried auth0’s setup with their WordPress plugin, but the auth0 WordPress guide was a little out-of-date. I found the auth0 interface super overwhelming at a glance, and gave up quickly.

Some more searching put me on to the OneLogin SAML SSO plugin (GitHub), which has zero documentation on the WordPress plugins site, in their GitHub, or anywhere else that I can find. The plugin, once installed in WordPress, yields a settings page with a billion different options. While they are mostly well-described, it’s not super-clear what you need to do in order to get up and running, especially with Google Workspace.

Some trial and error and help from this support thread and it was working pretty quickly – although I still don’t know what is going on under the hood, so it will be a while before I decide to use this in any production capacity – so just wanted to document the process as it stands for WordPress and Google Workspace (as of March 2021, anyway).

There are two sides you need to configure to make this work – OneLogin’s WordPress plugin, and the Google Workspace SAML setup.

Starting with the Workspace side:

  1. Broadly, we’re following Google’s own instructions: “Set up your own custom SAML application“.
  2. Log into Workspace Admin, go to Apps, and select “SAML Apps”.
  3. Open the “Add App” dropdown and select “Add custom SAML app”.
  4. Enter whatever for the app name & click “Continue”.
  5. Copy the “SSO URL”, “Entity ID” and “Certificate” fields, taking care to get it all and preserve formatting. (You can download the IdP metadata as well for backup purposes, but you can retrieve this information again easily at any time, so don’t stress.) Click “Continue”.
  6. Now it will ask for your “Service provider details” – “ACS URL” and “Entity ID”. The Learn More link here provides no useful information about what these are or where to get them from – but they come from your WordPress setup.

So now we switch to the WordPress side:

  1. In a new tab/browser, log into your WordPress admin panel and install the OneLogin SAML SSO plugin, and activate it.
  2. Go to Settings->SSO/SAML Settings, which is where this plugin keeps its settings.
  3. At the very top of the page, there is a link: “Go to the metadata of this SP”. Clicking this will open an XML document which has the information needed for the Google Workspaces form.
  4. Two two values we want are as follows (note: both of these values seem to be able to be customised elsewhere in the OneLogin plugin settings):
    1. ACS URL: this is in the tag that looks like this: <md:AssertionConsumerService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://example.com/wp-login.php?saml_acs" index="1"/> – we want the value in the Location field.
    2. Entity ID: this is in the very first tag that looks like: <md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" validUntil="2021-03-04T00:22:23Z" cacheDuration="PT604800S" entityID="php-saml">. Default seems to be php-saml.
  5. You can ignore the rest of the fields in the “Name ID” section & just click Continue.
  6. Now we need to configure the Attributes. Basically just replicate the below screenshot (Primary email -> email, First name -> firstname, Last name -> lastname, Primary email -> username).

  7. Click ‘Save’.

Now, back to the WordPress SSO config:

  1. Set “IdP Entity Id” to be the “Entity ID” field that we copied from the Google settings up earlier on.
  2. Set “Single Sign On Service Url” to be the “SSO URL” field.
  3. Set “X.509 Certificate” to have the certificate from the “Certificate” field.
  4. Look for the “Create user if not exists” field. Whether or not you want this checked depends on whether you already have your user accounts set up. It may be easiest, if you’re just trying to get this working at all, to check this and try with an account that doesn’t already exist in WordPress with the same email address.
  5. Look for “Match WordPress account by” and change this to “E-mail”. Google Workspace does not appear to expose any username field (maybe you can make this work with mapping but not sure).
  6. Scroll down to “ATTRIBUTE MAPPING”. As with the Google Workspace-side mapping, we do the same here:
    Username: username
    E-mail: email
    First Name: firstname
    Last Name: lastname
  7. There are tons of other things that you should look at – for example, “Prevent reset password” might be something you want to do to make sure a user can’t accidentally have their WordPress password reset to bring it out of sync with their Workspace account (I suspect in theory this should not impact things as users should not be able to login without going through the SSO, but in case of WordPress bugs or vulnerabilities in plugins or whatever it’s probably safer).
  8. Once you’re ready, scroll back up to the top and check the “Enable” checkbox.
  9. I strongly recommend opening a new private browser session and logging in as admin at this point, just in case any of this blows up access to your admin section.
  10. Then back to the bottom, hold on to your butts, and click “Save Changes”.
  11. You should now be able to log into your WordPress site with your Google Workspace credentials.

This document is a work-in-progress as I figure out more about what is going on; very interested in comments and feedback.

Other notes:

  • Single Logout (SLO) does not seem to be supported by Google Workspace at the moment so there is no easy way to log out of all services at once [that I can see].

Resources:

Copying Google Sheets into Shared Drive Removes Ability to Create Project Script Properties

If you have a Google Sheet with some associated Apps Script code in your personal drive, then you copy it into a shared drive (e.g., your company share), it will strip the project properties, including script properties that you may have (e.g.) set to hold credentials.

This seems like a good plan, to prevent information leaking when scripts are copied around.

Unfortunately, it also seems to permanently remove the ability to add script properties ever again. If you open the Project properties and check the Script properties, the “Add row” link is simply missing.

If you create a new spreadsheet on the shared drive, it will work fine – so in some cases, it might be easy to just copy/paste the spreadsheet info and the app script into an entirely new document, created from scratch.

Unfortunately this didn’t work for my spreadsheet, which had a lot of names ranges, buttons, images, and drawings, which don’t lend themselves to copy/pasting easily.

Been battling this for a while and can’t find a simple solution, although other people seem to have the same problem. There’s also several bugs in Google’s Issue Tracker that seem to be related.

The easiest fix I’ve found is:

  1. Create a new copy of the spreadsheet in my own drive
  2. Delete the scripting from the spreadsheet
  3. Copy the spreadsheet over into the shared drive
  4. Re-add the scripting into the spreadsheet
  5. Add the project properties you need

This preserves all the spreadsheet bits that are frustrating to recreate manually and (for me at least) re-creating the script is a simple copy/paste.

WordPress Shortlinks Interfering with wget Mirroring

Testing some methods to convert a WordPress site into a static site, I ran into a weird problem when converting the links (using -k or --convert-links in wget) was breaking the mirroring process, putting in the wrong links.

My mirror command was simply:

wget -m -k -nH example.com

The link conversion ended up breaking the internal links – a link like:

<a href="about/index.html">[ About ]</a>

… was being converted into

<a href="index.html?p=2">[ About ]</a>

I was a bit stumped until I noticed that the About page contained the following HTML:

<link rel='shortlink' href='https://example.com/?p=2' />

Removing the shortlink by simply adding the following line to wp-config.php (although it should probably go in the theme) fixed the problem:

remove_action('wp_head', 'wp_shortlink_wp_head');

My guess is that wget sees the shortlink declaration and tries to helpfully rename the files to match, but this ends up just breaking the link conversion, rendering the mirror useless.

On the Normalisation of Augmented Reality

AirPod Pro

The “Transparency” feature of the newly-released Apple AirPod Pro grabbed my attention. The claim from Apple is that Transparency is “for hearing what’s happening around you” — it “lets outside sound in, and allows things to sound and feel natural when you’re talking to people nearby”.

Even before the launch of AirPod Pro, I had noticed that many people would engage in conversations with their AirPods in. Maybe not long conversations, and people wouldn’t sit in meetings with them on the whole time, but (purely anecdotally) I felt like I would see a lot of people having a chat while still with their headphones in.

It has felt like the usual social stigma of talking to people with your headphones still on was fading a little bit. When I’m wandering around with headphones on, I’ll usually take at least one ear out to make it clear to whoever I’m speaking to that they have my attention.

But Transparency changes the entire game. Instead of assuming that people with headphones in are blissfully unaware of your existence, lost in their own world of music or podcasts or conference calls, we can now wonder if they’re in Transparency mode and actually hyper-conscious of what you’re saying to them, because all the background noises are being stripped out.

This sounds awesome in many contexts. As someone hitting the “middle aged” milestone, I often find it frustrating being in noisy environments and trying to have a conversation. The idea of being able to pop in headphones and have filter out the background noises so I can better hear the people near me talk is appealing.

(It should be acknowledged that Apple weren’t the first ones to come up with this idea. Bose has had conversation-enhancing technology for a while; there might be other vendors with similar technology.)

One of the big challenges though for this sort of technology, however, is the fact that people would generally be self-conscious wearing augmentation hardware in many environments, both social and professional. Think about things like hearing aids and glasses — for many people the vanity issues of these devices, despite being super common and well-established in society, have prevented them from taking them up, often to their own significant detriment.

The cost of traditional hearing aids can also be a big factor. If they’re not accessible on your insurance, they might simply be unaffordable — the top models often run into the thousands. While the AirPod Pros are expensive, they might be “good enough” for many users when compared with the expense of hearing aids — cherry picking a single example from the Bose site:

One of many positive reviews of Bose assisted hearing.

Think about Google Glass and the “glasshole” phenomenon. I was super excited when I first heard about this project. I only ever met one person who was wearing them; while it was weird talking to him wondering what was going on, I still confess to being more nerdishly fascinated by the possibilities than thinking about the implications for those around me (let alone what I’d look like wearing them). The possibilities of a great AR platform have become far more interesting to me than Virtual Reality — even from a video gaming perspective, it feels like in the foreseeable future, there are a lot more fun opportunities in AR than VR.

The Glasshole Problem (source)

But Google never really nailed Glass as a platform for the average person on the street and the project was relegated to specific commercial/industrial uses. The backlash against them blew up into all sorts of weird places as society wrestled with the Glasshole Problem, which reportedly triggered physical confrontations and resulted in businesses creating policies to deny service to customers wearing them.

With the recent report that Apple is considering Augmented Reality (AR) smart glasses, it’s easy to start thinking of the AirPod Pro as a way for Apple to test the waters in terms of normalising technological augmentations — headphones and glasses — by making them Cool.

To really drive the mass adoption of AR to the level of smartphones, it will be critical to make the experience of wearing AR hardware not only technically excellent, but also Cool enough so that people are comfortable wearing them regularly.

Apple have done more to make technology Cool than any other company. The iPod set the scene by normalising interaction of music with your computer. The iPhone transformed the world with the smartphone revolution. (I remember being asked by many people, prior to its launch in the Windows Mobile/PocketPC era, “why would you want to check your email on your phone?”, an attitude which is now so far removed from reality it’s hard to even believe it once existed).

More recently, the Apple Watch has set a new standard for fashioned-based technology. They are clearly the dominant wearable; as with many other Apple devices, despite a lot of naysayers being critical of the devices, they have had a massive impact on how people see and use hardware.

The AirPod Pro has the potential to change the way people think about other hardware augmentations that are more obviously visible in your regular interactions with other people. It seems unlikely they’ll offer a variety of colours so they’re more readily thought of as fashion accessories — the white stems poking out of the ears just seems like it has brand recognition that is too good to pass up on.

But if they can get people used to conversing with people with AirPods plugged into their ears in a variety of normal circumstances — in bars, in meetings, in conversations walking down the street —it is a powerful step along the way to adjusting the expectations of the entire planet in terms of other hardware augmentations.

And if there’s any company that can make wearing high-tech nerd computer glasses cool — it’s Apple.

Their growing, evolving knowledge on how to make consumer devices that combine fashion and technology (both software and hardware) to create a unified product that resonates with people means they are uniquely positioned to effect another paradigm shift in terms of wearable computing when it comes to AR. They might be the first company to finally make a product that people feel comfortable enough to wear enough of the time to make them genuinely useful.

First published on Medium.

Disruption: why is the US so different for startups?

AVC’s Fred Wilson wrote a post recently looking at platform monopolies and why they’re great targets for disruption.

It’s clear that almost everything about the US is different for startups. A lot of it is just its sheer size – having ten times more people than Australia changes everything. Probably the biggest complaint in Australia though is the lack of VC funding and a reduced appetite for risk.

It’s not clear to me which one of these is the chicken and which one is the egg. Is the startup scene awesome because of the VC? Or does the VC exist because of the startups?

Since moving to the US, the most striking thing I’ve noticed here is the prevalence of these large, giant “platform monopolies” – but not just in the tech space. Many aspects of daily life in the US seem to be managed by these giant institutions. Navigating these large institutions is cumbersome and tiring (especially if you’re a foreigner and have no idea how things are glued together) – and as a result, there are a lot of middlemen that try to make the process easier.

For example, if you’re looking for health insurance, trying to deal with the insurance companies is a real pain. So, there are hordes of health insurance agents and brokers that sit in between you and the insurance companies to try to make that easier. They take a small fee.

If you’re starting a company and need to pay your employees, you need to be aware of the specific tax issues in your state and federally. Even for a one person company this is challenging; I can’t imagine what a headache it’d be if you were trying to set up in multiple states. But don’t worry – there are many payroll companies that specialise in this, all for a small cut. (The first time I ran a payroll here in the US I was staggered by the fact that the company used UPS to courier me an actual payslip on an actual piece of paper. )

Near where I live there’s an entire business that appears to exist solely to cash cheques – a form of payment that is basically extinct in Australia, but because it’s so common here there are these weird cheque companies that exist. I assume that they must make a bit of money on each cheque they cash.

The list goes on. Almost anything complicated you want to do, there’s a support system that seems to sit in the middle to make your life easier.

Fred Wilson comment on this in his article:

The Internet, at its core, is a marketplace that, over time, removes the need for the middleman. That is very good news for the talent that has been giving up a fairly large part of its value to all of the toll takers in between them and their end customers.

This is where a lot of the interesting disruption is happening in the US – people tired of these old, monolithic systems looking to make a change. The payroll issue is being addressed by companies like ZenPayroll. I have no idea what people are doing about cheques but I thought it was entertaining to see that you can deposit a cheque here now by taking a photo of it. And everyone is trying to crack the healthcare nut – health industry startups abound and appear to be highly favoured as targets for funding (hey, the US spend more money per person on healthcare than any other country – it’s good business).

There are many other examples, including ones with global application – Uber and Lyft are probably the most significant and most recent examples; their impact on the transport industry is still being felt. The music industry is another one – all those poor record company executives are going to be a relic of days gone by.

Every country, every society, every community has its own entrenched systems, their own way of doing things. The bigger the environment and the longer they’ve been around, the more likely you are to find middlemen. Technology makes it possible to go back and re-evaluate the old way of doing things. Better automation, communication and integration means a lot of the old ways of doing things are ready to be swept aside – and that’s where a lot of the disruption is happening.

Ultimately I think that it’s this potential for disruption that makes the US startup scene so vibrant and interesting. It’s the fact that every opportunity to change the status quo has the potential to pack a huge punch – even capturing a small percentage of the market here can mean a big business. The bigger the established players in a sector become, the more naturally resistant they become to change – meaning a lot of opportunity for smaller players to start to carve out a niche.

When you encounter someone saying “this is just how we’ve always done it”, pay attention – you might be on the verge of something big. These are tough fights to pick, but – as is being shown by Uber – if you make the right moves at the right time with the right technology, you can revolutionise a space.

Binary Lane – a new hosting service from Mammoth Media

I’m proud to announce that the Mammoth team has launched a new product this week – Binary Lane.

Binary Lane is our new take on virtual private server hosting. Our service Mammoth Networks has been in operation since 2010 and has grown steadily to become a respected name in VPS hosting in Australia, and the mPanel – the software that drives this service – has matured and evolved until it has become, in our humble opinion, one of the best platforms for managing virtualised servers on the planet.

Binary Lane features several significant differences that we thought justified a new brand. The main changes:

  • Binary Lane uses KVM for its virtualisation back end, instead of Xen. This gives us a bunch of great new features – including live migrations, allowing us to almost instantly move guests between host nodes.
  • We’re in a new data centre, right here in Brisbane. We’ve had a great relationship with PIPE (formerly SOUL, formerly Comindico) and have been in that facility for years, but we wanted a local presence in our own city. The new NEXTDC data centre and their DC-as-a-service model really fits what we want to do.
  • New IP transit. Again our PIPE/TPG/SOUL connectivity has been awesome, but getting transit from a single carrier has reduced our flexibility a little bit. We’re working with the unbelievably great team at APEX Networks to provide our transit.
  • All SSD storage. SSDs are the future for server systems. The price has reduced to the point where building entire storage systems out of SSDs is feasible for a huge variety of application types. While bulk data storage is still a little bit away, we feel that the performance offerings of SSDs are impossible to ignore.

The Binary Lane brand is also something that gives us a little more freedom internationally – Mammoth Networks was, for various reasons, somewhat restrictive in the United States :) Startups, let that be a lesson to you – even if your target market is Australia initially, always think about the rest of the world when you start building your brand!

Of course Mammoth Networks isn’t going anywhere – it will still continue to receive all the attention it gets currently. We’ll be doing updates to mPanel that may appear on one or the other site first, but eventually both sites will be updated with any new features that are added.

Big congratulations to our awesome development and operations teams at Mammoth for their really hard work on this project and special mad props to our technical director Nathan O’Sullivan, whose tireless efforts leading the charge have again resulted in shipping another great product.

Oh yeh – we’re so confident you’ll love Binary Lane that we’re offering a free seven day trial. Go check it out, marvel at the speed, wonder at the power of mPanel, and be amazed.

Possible Unintended Consequences of Legalising Drug Use: Faster Cryptocurrency Uptake

A few states have recently legalised marijuana use in the United States. This is weird for everyone. Not only is it a complete reversal on drug policy, but it’s still actually illegal at the federal level. So while its legal in certain states, it’s still technically a federal no-no to be using it. This is confusing as hell, although it didn’t stop Colorado’s first day from being a huge event which raked in a cool $1 million.

A great example of the confusion popped up on the New York Times today, as they covered the fascinating story of how banks in these states are basically refusing to offer banking services for these (legal) providers of marijuana. US banks, of course, have an impeccable reputation when it comes to deciding what to do with money, so this of course comes as no surprise.

The first thing that I thought of when I read this article (a little sarcastically, because I’m so sick of hearing about it) was “well, I guess they could just use BitCoin, amirite?” But as I thought about it some more, I started thinking that it’s probably not a terrible idea for these guys.

Compared with the difficulty of dealing with putting cash into brown paper bags or Tupperware to move it around and the scary issues of security logistics entailed, the idea of using a freely available and accessible cryptocurrency seems compelling to say the least.

Sure, it’s still nowhere near as widespread as Visa or Mastercard. There’s the possibility of vulnerabilities. People don’t really know how to deal with keeping worthless digital assets secure, let alone a digital asset that contains real money. But it’s here, now, and it means that these legal(ish) vendors could go about their business with one less headache.

Well, at least until tax time when they have to pay the IRS – it seems likely that they’re one entity that is not going to be accepting payments in cryptocurrencies any time soon. I guess the drug vendors will have to deal with the problems of having a massive pile of super-secret untraceable anonymous digital cash!

Except, of course, this isn’t really how cryptocurrencies work (at least, those based on the BitCoin blockchain system). Intentionally trying to hide their income from the feds is probably just as dangerous as hauling around large amounts of small bills. The instability of most of the cryptocurrencies is probably the biggest concern; storing your tax payment in a currency that might halve in value by tax time would be risky to say the least. But if they wanted to make life more difficult for the IRS, switching to cryptocurrency would probably do that in the short term, while saving them all sorts of other hassles.

In short – talk about your disruptive technologies! Banking – for what are probably good reasons – is failing to keep up with the pace of legislative change here. Whether or not cryptocurrencies actually pose a threat to

By the time I finished writing this, the following articles appeared: Colorado Marijuana Dispensary Uses Bitcoin to Evade Federal Laws, and back home, businesses in Australia are starting to accept BitCoin.