deferred.io

GPG, SSH & U2F via Yubikey 4c on macOS, Linux & Android

Thu, 03 Aug 2017 00:00:00 +0000

How I use my Yubikey & GnuPG

I use the Yubikey 4c daily to improve my personal and professional security online. The two use cases for me are:

GnuPG private key store
FIDO U2F authentication

I use GnuPG daily to do the following:

signing git tags
using the AUTH key for secure shell authentication
securely store passwords with pass on macOS and using Password Store on Android

I store three 4096-bit subkeys on my Yubikey:

Signing subkey
Encryption subkey
SSH AUTH subkey

My master signing key is stored securely on paper, in a vault. I rarely use it except to create new subkeys, generate revocation certs, etc. Keeping your master signing key offline is the smart choice.

I rarely encrypt or decrypt messages using GnuPG. My use cases are about authentication and verification. Pass/Password Store both use encryption extensively, but it’s all hidden from the user behind a few simple wrapper commands.

I use the Yubikey model that supports USB Type-C. This is the essential piece that allowed me to use the same 4096-bit subkeys on both my laptop and my smartphone. Prior to this I was forced to use 2048-bit keys on the, now defuct, Sigilance card. This was a big pain because the card and card reader are clunky, and it also forced me to use dedicated 2048-bit keys just for for Pass/Password Store. I still used the 4096-bit keys for Linux kernel maintenance. Now I use only the 4096-bit keys for everything and have revoked the 2048-bit keys.

I use FIDO U2F for the websites, apps and services that support the standard. Sadly, many still do not.

A note on GnuPG versions

I use GnuPG v2.1 (aka Modern) on macOS and Debian. Migrating to this version will make backwards-incompatible changes to your keyring. It also comes with some pleasant changes to the otherwise unpleasant experience of using GnuPG, which is worth it in my opinion. Please do not take the decision to upgrade lightly.

It can be difficult to remove versions 1.x and 2.0 from your system. Dependency creep from other packages installed on your computer may bring them back in. Certain applications may only know how to talk to the older libraries and agents. For instance I had to remove all GnuPG-related features from my favorite mail editor in order to keep it working after upgrading to GnuPG v2.1 Modern. I never use GPG for email and I don’t miss the features.

Mac users that have installed GPGTools will be greeted by a native UI when plugging in their Yubikey. Other UNIX users might just do it all in the terminal via the gpg or gpg2 commands (this is what I use in my examples below). Linux desktop users may edit their cards with a GTK- or KDE-specific interface.

Many of these GUI software suites built on top of GnuPG are incompatible with v2.1 Modern. Buyer beware.

A note on Yubikey models

There are a variety of different Yubico products, differences in firmware and hardware revisions, and lots of different procedures floating around the web.

This guide focuses purely on the new Yubikey 4, which involves a simpler process compared to some of the arcane steps required with the older Yubikey Neo. CCID mode is not set by default on the Neo, so make sure you do that first if you have that model.

Additionally this guide focuses on transferring existing keys to the Yubikey 4. I will create a shiny new AUTH key however, just to illustrate how that works. I’m using the Yubikey 4c, which works on both my Macbook Pro and my Google Pixel smarthphone.

If you have a Neo, or if you want to generate new keys on the card I recommend first reading Richard North’s blog. I also recommend Trammell Hudson’s blog, which generates the keys off the card, and then transfers them.

Many of the things in this blog post are more succintly explained in this Linode article, which I recommend reading as well.

I will interchangeable refer to the Yubikey, the 4c, the card and the key card. They are different names for the same thing: the Yubikey dongle plugged into your USB port.

Make a backup

gpg --export-secret-key --armor YOURKEY > YOURKEY.gpg.armor

Leaving the armor file on-disk is dangerous. It contains your private key, which anyone could use to impersonate you. There are lots of great resources on how to make proper backups of your keys via QR codes, USB drives or CDs.

I’m a fan of printing my secret key paper and storing it in a safe or vault. Don’t forget to generate a revocation certificate. Store this in your safe alongside your private key.

Reset the Factory Pins

The admin and user pins on your Yubikey are set at the factory. They are the same for all consumer-grade Yubico devices. We’ll start by changing them to something secret. While we’re at it we will personalize the key card.

Things can go wrong while editing the card sometimes. Below I describe the process of editing the card fully. Read this first before you start typing commands. The log at the bottom of this section contains the actual commands and expected output. Simply follow along with that log and you should be fine.

In case things do go wrong I have also included steps on how to recover if you accidentally lock yourself out of your own key card. This can be caused by too many attempts to unlock the key card with the wrong pin. Resetting the card can also fix some other weird, anomolous behavior.

The factory admin pin is 12345678. The new admin pin you supply must be at least 8 characters long. After changing the admin pin you should see PIN changed. If not, try again and make sure your new pin is at least 8 characters long.

If you see an error like Error changing the PIN: Conditions of use not satisfied it typically means that your pin was too short and was not saved. Try it again with an appropriate length pin.

The factory user pin is 123456. The new user pin you supply must be at least 6 characters long. After you see PIN changed you can move on to personzliation.

It is possible to lock your key card out from any further changes by trying to enter an incorrect pin too many times. You must reset the card to factory settings and try again. Save the following to a file named reset-card-script:

/hex
scd serialno
scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
scd apdu 00 e6 00 00
scd apdu 00 44 00 00
/echo Card has been successfully reset.

And then run gpg-connect-agent -r reset-card-script. You’ll have done a factory reset on the card and can start over from the top of the log.

Personalization is an optional step. The log below details the most common parameters that users might wish to change.

Now plug in your shiny new Yubikey and follow the output of my session below:

Edit your new key

$ gpg2 --card-edit

Reader ...........: Yubico Yubikey 4 OTP U2F CCID
Application ID ...: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Version ..........: 2.1
Manufacturer .....: Yubico
Serial number ....: XXXXXXXX
Name of cardholder: [not set]
Language prefs ...: [not set]
Sex ..............: unspecified
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: not forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 0 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

gpg/card> admin
Admin commands are allowed

gpg/card> passwd
gpg: OpenPGP card no. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX detected

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 3
PIN changed.

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 1
PIN changed.

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? q

gpg/card> name
Cardholder's surname: Turquette
Cardholder's given name: Michael

gpg/card> lang
Language preferences: en

gpg/card> sex
Sex ((M)ale, (F)emale or space): M

gpg/card> url
URL to retrieve public key: http://deferred.io/3a8f3b2f5a7c9849.txt

gpg/card> login
Login data (account name): mturquette

gpg/card> quit

Transfer existing GnuPG keys

I will populate all three key slots on the Yubikey 4c with subkeys, not the primary signing key. I keep my primary key offline, on paper, in a vault.

For illustration purposes, this is what it looks like to push a primary key to the Yubikey 4c/card (I don’t do this myself):

$ gpg --edit-key
...
gpg> toggle
...
gpg> keytocard
Really move the primary key? (y/N) y
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]

Please select where to store the key:
   (1) Signature key
   (3) Authentication key
Your selection? 1

The key command select subkeys. Use this to push subkeys to the Yubikey 4c. Personally, I have created a separate signing subkey, and this is what I use to populate the signing slot on my card. In the example below I move the encryption subkey off of my hard disk and onto the key card:

gpg> key 2

pub  rsa4096/3A8F3B2F5A7C9849
     created: 2012-02-18  expires: never       usage: SC
     trust: ultimate      validity: ultimate
sub  rsa2048/04022B7507176F5D
     created: 2016-06-20  expired: 2017-06-20  usage: S
sub* rsa2048/B92E93D2387E0EF4
     created: 2016-06-20  expired: 2017-06-20  usage: E
gpg> keytocard
Signature key ....: BE4B AF4E CAB0 C33F 9235  20C4 3A8F 3B2F 5A7C 9849
Encryption key....: [none]
Authentication key: [none]

Please select where to store the key:
   (2) Encryption key
Your selection? 2
...
gpg> key 2
...

Note the final key 2 command. This deselects this particular subkey. You can repeat this process using different subkey indices for the signing subkey (if you have one) and your AUTH subkey (again, if you have one).

Generate a new SSH AUTH key

If you do not have an AUTH subkey and want to use it for SSH key authentication, then follow the steps on this Yubico page.

A few notes:

Create the AUTH key on-disk, instead of on-card
Then move it using the same steps used in the above section
Disable unwanted subkey capabilities before saving the subkey to disk

The Yubico instructions mention to select the ‘A’ capability for AUTH, but they neglect to point out that some GnuPG versions will also select S (Sign) and E (Encrypt) by default. You might want to toggle those off if you already have subkeys for these functions

Using AUTH subkey for SSH

Now that you have the AUTH subkey on-card, let’s setup SSH to use it.

First, add the following line to some sort of script that is run before SSH can be invoked. In my case I put this in ~/.config/omf/init.fish, as I use Oh My Fish to manage stuff for fish, my preferred shell.

## GPG and SSH agent configuration

gpg-connect-agent /bye
export SSH_AUTH_SOCK=(gpgconf --list-dirs agent-ssh-socket)

gpg-agent needs SSH support to be explicitly enabled. Add the following to ~/.gnupg/gpg-agent.conf:

enable-ssh-support

Kill all ssh-agent and gpg-agent instances. Typically opening a new shell instance will kick off the machinery above and you should see your new shiny AUTH key by running:

ssh-add -l
4096 SHA256:osIdSEalN4U4ib8wTqpdu1OWKvNTPzIDSZNi58s6AAs cardno:000605762380 (RSA)

You’ll still need to distribute the public key to your remote machines. Generate it with:

ssh-add -L >> ~/.ssh/authorized_keys

It will be the last RSA key in the file. Distribute that to all of the remote authorized_keys files and you’ll be in business.

If you have not already unlocked your key in gpg-agent then you might see the following:

ssh 192.168.0.13
sign_and_send_pubkey: signing failed: agent refused operation
mturquette@192.168.0.13's password:

Doing some sort of gpg operation (encrypt a file, etc) will prompt you for your passphrase, unlock the key, and then you should be able to SSH without issue. Folks not using GnuPG v2.1 (Modern) might find Keychain useful for solving this issue. For v2.1 I’m still trying to figure out the best way…

Yubikey 4c & Password Store on Android

Recent Android smartphones support USB Type-C ports that are compatible with the Yubikey 4c. This means that both Password Store and OpenKeychain can use the Yubikey 4c as a key card.

Pass relies on OpenKeychain for GnuPG support. For users without a hardware key card OpenKeychain expects them to import their GPG secret key onto the phone. In my opinion this is unsafe and terrible advice. Especically for all the cool kids running around with rooted phones and custom ROMs that they simply trust to not do nefarious things with their personal info…

Note that both projects can be a bit slow about pushing the latest versions of their code to the Google Play Store. I recommend building both from source. That’s out of scope for this post, but both build quite easily using the Android SDK and can be pushed over to your phone with a simple adb install pass.apk.

Once both are installed open the OpenKeychain application and plug in your Yubikey 4c. You’ll need to create an identity and bind the key to it. Success looks like this:

Now you’re ready to use Password Store to share encrypted passwords between your smartphone and all of your computers. I host my pass git on a private Github repository. I suggest you do keep access to the git repo private, because even if others cannot decrypt your passwords they will be able to deduce the websites, services, applications, usernames and any other cleartext information that makes up the structure of your ~/.password_store directory.

My config:

Note that I do not use ssh:// for the protocol, but https://. Why? Because SSH requires you to store you SSH private key on the phone. My goal is to never store private key data on the phone.

The only downside to this approach is that in order to pull the latest passwords from Github, I must first remember to decrypt my Github password, then select Pull from remote and finally paste my password into the field. No big deal and worth the peace of mind.

If you’ve followed along to this point you should have GPG subkeys stored on your Yubikey 4c and you should also be able to share encrypted passwords between your workstations using pass and your Android smartphone using a combination of Password Store and OpenKeychain. Congratulations!

Set up FIDO U2F for GitHub

Github Help covers this pretty well already.

I have had success interchangeably using FIDO U2F and GPG on the same key. No need to kill gpg-agent or ssh-agent or hotplug the device (so far).

This only works in Chromium so far, but hopefully mobile apps will start to adopt it in the future.

Miscellaneous bits

If you use a signing subkey like me, and you prefer to use it for most common signing operations, you might want it to be the default key. Unfortunately GnuPG documentation is simply garbage and you might be surprised to find out that setting default-key mysubkey in gpg.conf doesn’t work! Subkeys require a bang at the end, to denote that they are subkeys. My ~/.gnupg/gpg.conf file:

default-key A23A9C9BC325A4D4!

Similarly for git I prefer to use my signing subkey. My ~/.gitconfig file:

~/.gitconfig

[user]
	email = mturquette@deferred.io
	name = Michael Turquette
	signingkey = A23A9C9BC325A4D4!

The same syntax can be used from the commandline. In this example we sign the f1 file with my signing subkey:

gpg2 -u A23A9C9BC325A4D4! -s f1

Introspecting GitHub Pages version history

Sat, 30 Jan 2016 00:00:00 +0000

See that nice little blockquote right above this line? That’s what this article is about.

deferred.io is powered by GitHub Pages and its source is version controlled publicly on GitHub. For the uninitiated, GitHub Pages is a free web hosting service offered by GitHub for websites whose source is version controlled on GitHub. It’s all very GitHub-y.

Since we have access to the source and the version history, it seems like a nice idea to make it easy for readers to see how an article has changed over time. I wish more news and journalism sites did this as a matter of transparency… No more ninja edits.

The implementation is super easy. I hard-coded the URL bits, but those could probably be generalised with some nice Liquid Templates. I wanted this feature on every page, which means editing the post.html layout. One-liner solution for my blog below:

So meta! My inner Hofstadter is pleased.

Bidirectional synchronization of notmuch tags with IMAP folders and Gmail Labels

Mon, 18 Jan 2016 00:00:00 +0000

Part of a series of articles describing the state of my dotfiles, UNIX environment and workflow in 2016. See the meta-article here.

TL;DR

maildir-notmuch-sync synchronizes local notmuch tags with remote IMAP folders and Gmail Labels.

I call the script from my pre- and post-sync OfflineIMAP hooks:

Usage notes

For the initial use I recommend:

Back up your notmuch database with notmuch dump. Duh.
Disable any periodic IMAP sync such as a cron job in case things go sideways. Duh.
After your local tags are in good shape, manually run

$ maildir-notmuch-sync post $HOME/mail/foo

Where mail is the root of all IMAP accounts and foo is the particular IMAP account that you are sync’ing.
If things look good then add the hooks to .offlineimaprc, re-enable your IMAP cron job and enjoy

For Gmail users creating a new tag/Label combo:

Create your Gmail Label and filter first using the Gmail web interface
Apply that label to at least one mail
Sync IMAP so that you pull down the newly labeled mail. Don’t forget to edit folderfilter in ~/.offlineimaprc so that you’ll actually sync this new IMAP folder
After maildir-notmuch-sync runs the post-sync hook you should have the tag. Verify with: notmuch search --output=tags "*" | grep my_new_tag
Now use notmuch like you normally would. Mail that has the tag applied to it will be copied to the new maildir. The next time you sync IMAP those mails will show up with the same Label in the Gmail web client

Disclaimer

This script will eat your hard disk, email embarrassing anecdotes to everyone you love and generally destroy your life. Running it means that you accept those consequences.

The details

Bidirectional notmuch tag <-> Gmail Label/IMAP folder sync

This change to my 2013 email toolchain is by far my favorite. While this script will work with any type of IMAP server, I personally use Gmail as my backing store (and so have my last three employers), which means that this technique syncs notmuch tags with Gmail labels, and vice versa.

Credit goes to Ethan Schoonover of Solarized fame for creating the initial version of the script that does the bidirectional sync. I’ve submitted my changes to his dotfiles on GitHub.

Side note: early on I ran across this pull request from Richard Lawrence. He was nice enough to share his scripts with me, which I did not ultimately use, but deserves credit for being a nice guy and answering my questions.

How does it work?

When you sync to your Gmail account over IMAP, you will locally get an IMAP folder for every Gmail Label. This is inefficient because you might have the same email labeled many times, resulting in lots of duplicates (one for each Label/folder). But it does give you a way to reconstruct Gmail Labels locally.

The method to sync these is pretty simple to understand. It has pre-IMAP sync and post-IMAP sync stages, which in my environment are tied to OfflineIMAP pre- and post-sync hooks. Described in prose:

Before syncing local changes over IMAP

The script will check to see if any emails have notmuch tags applied to them and if those same emails are not present in the local IMAP folders (maildir in my case) corresponding to the names of those notmuch tags. For example if I mark an email locally with the spam tag in notmuch, this script will detect that the email is not present in the spam folder on disk. The script copies the tagged mail physically on disk to the corresponding folder to keep them in sync.
Likewise, if I remove a tag locally, such as the inbox tag to archive a mail (in Gmail parlance) then the script will see that the mail is in the folder but not tagged, and delete the mail from maildir on disk.

After syncing IMAP and fetching new mail

The script will look for new mail in maildir that does not have a matching tag. For instance all of the mail I receive from the linux-kernel mailing list is labeled linux-kernel by Gmail, and placed into the linux-kernel maildir locally on disk. The script sees this and tags the new mail with linux-kernel using notmuch.
Similarly the script will notice if any mail on disk was deleted as a result of the IMAP sync and remove the corresponding notmuch tag locally. A good example here is if you log into the Gmail web client and archive some mail in your inbox that had previously been sync’d to your local maildir. It will have disappeared from your local directory due to the IMAP sync, and the script will remove the inbox tag from the mail.

This allows one to sync notmuch databases across different machines. All changes to the local notmuch database are converted to IMAP folder changes, and then pulled down by the other clients whenever they sync. This is a very clean alternative compared to others methods, such as storing the notmuch database on Dropbox, abusing notmuch dump and notmuch restore to distribute changes across databases or anything else super hideous like that.

How I email, 2016 edition

Mon, 18 Jan 2016 00:00:00 +0000

Part of a series of articles describing the state of my dotfiles, UNIX environment and workflow in 2016. See the meta-article here.

Much of my 2016 email workflow looks similar to my 2013 email workflow, and that article is still a useful resource for the basics. If you are starting from scratch however I recommend that you try to install the applications from your favorite package manager. That wasn’t an option back when the original article was written.

Basic workflow

I use Gmail as my backing store and IMAP server
- I make full use of Gmail filters and Labels
OfflineIMAP syncs my mail locally using maildir format
- On my workstation I sync everything, including Gmail’s All Mail folder, aka archive
- On my Macbook I only sync inbox, which is filtered aggressively, and the needs-review Label/tag
Notmuch indexes all new mail
- Automagical tagging happens based on Gmail Labels
- Super fast full-text search!
I browse mail with alot and compose with vim
- I can tag, untag and retag as I please from within alot
- Any changes I make locally will sync back to Gmail as changes to Labels
msmtp sends mail
- msmtp-queue handles automatically saves them if they could not be sent successfully. A single command flushes the queue once the network connection is reestablished

A key point

The only interface I use to my mail is through notmuch. Any changes to IMAP flags happens by changing notmuch tags. Any changes to Gmail Labels/IMAP folders happens by manipulating notmuch tags. Notmuch is the one and only oracle facing the user

Gmail filters

The most important filter for me is needs-review, which marks patches from open source software mailing lists that require my attention. Screenshot of my Gmail configuration:

inbox and needs-review

I’m a fan of the archive behavior in Gmail, where you can remove the inbox Label from an email to remove it from your sight, but you can always search for it later thanks to the Gmail’s All mail IMAP folder, which we do not touch.

In keeping with Gmail’s web client keybindings (which are not reconfigurable), I have bound the e key in alot to removing the inbox tag. I also kept the navigation bindings the same, s toggles the IMAP flag/Gmail Star, etc:

Most of the patches I receive have multiple tags applied to them. For instance, a patch addressed to the linux-clk mailing list might also Cc the linux-kernel mailing list. Thanks to the maildir-notmuch-sync script and the above Gmail filters, when we sync that email it will have at least three tags: linux-clk, linux-kernel and needs-review.

I never want to remove the linux-clk or linux-kernel tags; this mail was actually sent to those lists, so I consider these to be permanent tags. But after I review the patch, I want to remove the needs-review tag. In alot I’ve bound the & key for this, and its operation is analogous to the e key for archiving mails in my inbox.

Patch wrangling

Managing patches sent to a mailing list is a huge topic, including automatic replies, automated patch application, automated testing. You see the trend.

For starters, a quick solution to apply individual patches is to alias alot so that the working directory is in the root of your source tree of interest, and then bind a key to pipe the contents of a single mail to git or whatever you use to apply patches. For example:

nametrans, folderfilter and syncing only what you need

I’ve tried to keep the user experience between alot and the Gmail web client familiar (e.g. Gmail key bindings in alot, identical archive behavior). But the Gmail names for their directories are CaMelCaSe hell. I translate the names of these IMAP folders using a feature in OfflineIMAP cunningly called nametrans.

The OfflineIMAP folderfilter allows you select which IMAP folders to sync, which is especially helpful when you are trying to limit what you download on a laptop. An example configuration:

That configuration results in the following local notmuch tags:

$ notmuch search --output=tags "*"
attachment
drafts
encrypted
flagged
important
inbox
needs-review
replied
signed
unread

The astute reader will notice that I am translating names for folders that I don’t bother to sync, which is just because I want the example to cover all of the Gmail built-in Label names.

Steve Losh’s excellent The Homely Mutt focuses mostly on Mutt configuration, but also touches on OfflineIMAP and notmuch configuration. I recommend a look through it if you get stuck.

Saving mails as drafts

msmtp-queue and msmtpq are available through many package distributions, often bundled with the msmtp utility itself.

Copy these into your path and edit configuration as needed. My changes to msmtpq and ~/.config/alot/config:

Now any outgoing messages that failed to send properly end up in the drafts maildir. This will sync to Gmail next time you establish network connectivity and are able to sync IMAP. You can either send it through the web client by clicking on Drafts or from the commandline with msmtp-queue -r.

Inbox Zero™ on the road

When traveling I sync a reduced set of my mail. This is helpful for saving disk space (meh), and conserving precious bandwidth (yeah!). I will typically only sync inbox and needs-review. As mentioned above I have the e and & keys bound to remove those notmuch tags from alot, which in turn removes them from my disk, but only from their respective maildir.

An interesting side effect of the above technique is that if you do not sync any of the other Labels or IMAP folders containing references to the same mail, then removing the notmuch tag will delete every reference to that mail from your disk.

This is pretty cool.

This means that achieving Inbox Zero™ using the reduced on the road setup described above will mean that you could potentially have zero mail taking up space on your disk. Or zero mail being read by the guy who stole your laptop at the airport. Or whatever.

And your mail is still safe on your IMAP server where it was not fully deleted, only the inbox or needs-review Gmail Labels were removed.

Fetching mail in the background

I use a cron job to sync with my Gmail’s IMAP server:

State of the Dotfiles, 2016

Sun, 17 Jan 2016 00:00:00 +0000

Part of an on-going series of articles describing the state of my dotfiles, UNIX environment and workflow in 2016. This meta-article serves as the Table of Contents and a short overview

The articles

Bidirectional synchronization of notmuch tags with IMAP folders and Gmail Labels
Email is kind of a big deal. How I filter, label and work
Yubikey management for 2FA, SSH, GPG and disk encryption
Ghar: consistent, rapid deployment of UNIX dotfiles
Fish, Oh-my-fish and POWERLINE ALL THE THINGS
Tricks & tips: happy cron jobs, reusing dotfiles across OS X & Linux

In 2013 I wrote an article on my email workflow. It covered installation, configuration and usage of my email toolchain, which in 2013 was: Gmail, offlineimap, notmuch, afew and alot. Since then a lot of people have asked me a lot questions about it.

So what’s different since 2013?

Fast forward to 2016: much has changed and much has stayed the same. In the intervening years I have tried different MUAs including mutt-kz, notmuch-ruby-vim, the deprecated notmuch-vim,vanilla mutt with notmuchfs and more. I don’t use emacs, but I almost switched just for notmuch.el.

In the end I came back to using alot, despite its flaws. I also fall back on vanilla mutt combined with notmuchfs for certain tasks. If I ever take the time to finish supporting thread reconstruction then I might find this option more useful.

I no longer use afew for initial tagging. #112 was a show-stopper bug for me and it looks like the code is no longer actively maintained.

My lifestyle and work life has become more mobile. This does not jive with my 2013 toolchain that required me to SSH into a single machine to have access to a rich database of full-text email search (I’m looking at you, sup users). Bidirectionally syncing these tags with IMAP folders and Gmail Labels has allowed me to sync notmuch tags across multiple machines through IMAP.

Also part of my mobile lifestyle change was the desire to not sync all of my mail, but only the relevant bits to reduce on-disk size and the amount of time spent syncing with IMAP after a long period of dormancy. This has had a profound impact on my new toolchain.

I have also cleaned up and streamlined my dotfiles. I tried over a dozen different dotfile management frameworks (see GitHub ❤ ~/). The most UNIX-y approach of just about any of them is ghar. I started using it in 2013 and never looked back.

While living in San Francisco I also became a software hipster, started using a Macbook Pro and thus had to try Z shell and oh-my-zsh. After a couple of weeks failing to grow a moustache I reverted to Bash and the shameless copycat framework bash-it, which I used for a couple of years. I finally settled on fish and oh-my-fish which has left me in a state of unending bliss.

Of course I have continued to Solarize the heck out of everything and that aesthetic compulsion expanded to Powerline the heck out of everything.

I got serious about using a hardware security token for 2FA with YubiKey. That effort expanded to improve my SSH, GPG and disk encryption practices.

And I finally started using a proper password manager to generate, store and retrieve strong passwords for all of my services. Integrating pass into a non-interactive worflow (e.g. IMAP authentication from a cron job) has resulted in me removing cleartext passwords from all my dotfiles.

I have done the hard work for you.

Follow along with the articles linked at the top of the page and enjoy your UNIX experience even more in 2016.

Visualizing Linux kernel call graphs

Tue, 27 Oct 2015 00:00:00 +0000

Visualing call graphs is a great way to familiarize oneself with a piece of code. I recently did some work to rewrite the locking in the cpufreq subsystem in the Linux kernel; understanding how all of those functions relate was not trivial using only cscope/ctags and by inspection.

What I wanted was a visualization of the call graph, either as a PDF or an image format. Egypt combined with GraphViz and an SVG viewer does this nicely.

Generating the call graph

Unlike approaches using cscope output or patched versions of gcc, Egpyt is a very lightweight approach that only requires one to build the software with the -fdump-rtl-expand GCC option set.

We don’t want to generate a call graph for the entire kernel (well, maybe you do…) but since I only want to see the cpufreq call graph I made the following change to drivers/cpufreq/Makefile:

Now build the kernel with make (obviously make sure that CONFIG_CPU_FREQ is set). You should see some new .expand files such as:

$ ls -1 drivers/cpufreq/*.expand
drivers/cpufreq/cpufreq.c.170r.expand
drivers/cpufreq/cpufreq-dt.c.170r.expand
drivers/cpufreq/cpufreq_governor.c.170r.expand
drivers/cpufreq/cpufreq_ondemand.c.170r.expand
drivers/cpufreq/cpufreq_opp.c.170r.expand
drivers/cpufreq/cpufreq_performance.c.170r.expand
drivers/cpufreq/cpufreq_stats.c.170r.expand
drivers/cpufreq/cpufreq_userspace.c.170r.expand
drivers/cpufreq/dbx500-cpufreq.c.170r.expand
drivers/cpufreq/exynos5440-cpufreq.c.170r.expand
drivers/cpufreq/freq_table.c.170r.expand
drivers/cpufreq/omap-cpufreq.c.170r.expand
drivers/cpufreq/spear-cpufreq.c.170r.expand
drivers/cpufreq/tegra124-cpufreq.c.170r.expand
drivers/cpufreq/tegra20-cpufreq.c.170r.expand

Now we need to visualize this data.

Build and install Egypt

wget http://www.gson.org/egypt/download/egypt-1.10.tar.gz
tar -xzf http://www.gson.org/egypt/download/egypt-1.10.tar.gz
cd egypt-1.10
perl Makefile.PL
make
sudo make install

Verify that we can generate call graph info from our .expand files:

$ egypt /home/mturquette/src/linux/drivers/cpufreq/cpufreq.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/cpufreq-dt.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/cpufreq_governor.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/cpufreq_ondemand.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/cpufreq_opp.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/cpufreq_performance.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/cpufreq_stats.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/cpufreq_userspace.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/freq_table.c.170r.expand
digraph callgraph {
"store_sampling_rate_gov_pol" -> "update_sampling_rate" [style=solid];
"store_io_is_busy_gov_pol" -> "store_io_is_busy" [style=solid];
"cpufreq_enable_boost_support" -> "create_boost_sysfs_file" [style=solid];
"cpufreq_register_driver" -> "create_boost_sysfs_file" [style=solid];
"cpufreq_register_driver" -> "remove_boost_sysfs_file" [style=solid];
"cpufreq_gov_userspace_exit" -> "cpufreq_unregister_governor" [style=solid];
"cpufreq_get_policy" -> "cpufreq_cpu_get" [style=solid];
"remove_boost_sysfs_file" -> "cpufreq_sysfs_remove_file" [style=solid];
"store_ignore_nice_load" -> "get_cpu_idle_time" [style=solid];
...

Great! We have the call graph data, and a tool to parse it into a meaningful format. Now to visualize our data.

Generate the SVG

If you do not have the veritable Graphviz suite installed already:

sudo apt-get install graphviz

This gives us the dot application for generating simple images from well-formated data. To create an SVG of all of the core files for the cpufreq subsystem we must feed the data into dot. Using the same command as above, we’ll pipe the data into dot:

$ egypt /home/mturquette/src/linux/drivers/cpufreq/cpufreq.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/cpufreq-dt.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/cpufreq_governor.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/cpufreq_ondemand.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/cpufreq_opp.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/cpufreq_performance.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/cpufreq_stats.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/cpufreq_userspace.c.170r.expand \
/home/mturquette/src/linux/drivers/cpufreq/freq_table.c.170r.expand | dot -Grankdir=LR -Tsvg -o callgraph.svg

Viewing it

cpufreq.c with all dead-end functions pruned:

cpufreq.c, cpufreq_governor.c, cpufreq_stats.c and freq_table.c with all dead-end functions pruned:

For large call graphs I recommend Inkscape. You can print to PDF from here, or to a dead tree. Really the sky is the limit.

Enjoy visualizing your new call graphs!

Thanks to this Stack Overflow answer and Airead Fan’s blog for making me aware of Egypt

Disabling Press-and-Hold for MacVim

Mon, 03 Aug 2015 00:00:00 +0000

Mac OS X has a nice Press and Hold feature that allows users to select accents and less commonly used characters by simply holding down a key on the keyboard for a short time. A small pop-up like the following will give users the option for that particular character,

Unfortunately these feature gets in the way for some applications like MacVim, where I want to hold j or k for scrolling through text. Some quick Google searching showed me that many users of Sublime Text 3 had managed so to solve the problem with,

I use MacVim and needed to apply the same setting to that application, as I did not want to disable the feature globally. So I went with the brute force method. Looking at the above Gist I ran defaults until I got some useful info,

$ defaults find vim | grep key
Found 1 keys in domain 'com.apple.preferences.extensions.ShareMenu': {
Found 1 keys in domain 'com.apple.Preview': {
Found 1 keys in domain 'com.apple.preferences.extensions.ServicesWithUI': {
Found 1 keys in domain 'Apple Global Domain': {
Found 1 keys in domain 'com.apple.dock': {
Found 1 keys in domain 'com.apple.recentitems': {
Found 8 keys in domain 'org.vim.MacVim': {
Found 1 keys in domain 'com.apple.ServicesMenu.Services': {
Found 1 keys in domain 'org.vim.MacVim.LSSharedFileList': {
$ defaults write org.vim.MacVim ApplePressAndHoldEnabled -bool false

The first command finds the string for editing properties/defaults for MacVim. The second command then adapts the first command in the GitHub Gist above for the org.vim.MacVim namespace.

After restarting MacVim, everything works beautifully! Note that if you use vim within a terminal emulator on Mac OS X then you likely do not see this issue, as your terminal emulator probably disabled press-and-hold as a sane default. The above is only needed if you use the GUI MacVim interface.

Suunto DM5 and MovesLink2 data transfer bug

Mon, 01 Jun 2015 00:00:00 +0000

I tried to upload my dives from a recent trip diving for abalone using the Suunto DM5 software on my MacBook Pro. It failed to transfer data from my Suunto D6i. I tried lots of fixes that I found through Google:

move ~/.config/Suunto to some place safe, restart DM5 to build fresh database
completely uninstall DM5
hold the USB connector firmly against the D6i body to ensure solid connection
close DM5 and try to connect the device through MovesLink2 (I swear someone said this worked for him on a forum somewhere…)

I inspected the logs and the failures varied greatly. The most common type of error was a failure to read the device memory:

[2015-Jun-01 11:20:00 -0700] [ERROR] loadExerciseLogs: failed with exception: failed to read device memory
[2015-Jun-01 11:20:33 -0700] [ERROR] loadExerciseLogs: failed with exception: failed to read device memory
[2015-Jun-01 11:20:59 -0700] [ERROR] loadExerciseLogs: failed with exception: failed to read device memory
[2015-Jun-01 11:22:22 -0700] [ERROR] loadExerciseLogs: failed with exception: failed to read device memory
[2015-Jun-01 11:22:46 -0700] [ERROR] loadExerciseLogs: failed with exception: failed to read device memory

But I also got other weird stuff such as not being able to classify my device correctly:

[2015-Jun-01 13:26:53 -0700] [ERROR] SerialPort::configure failed: serial_port::stop_bits = 0
[2015-Jun-01 13:26:53 -0700] [ERROR] Failed to configure port for device Suunto HelO2
[2015-Jun-01 13:26:53 -0700] [ERROR] Serial Port unavailable
[2015-Jun-01 13:26:53 -0700] [ERROR] SerialPort::configure failed: serial_port::stop_bits = 0
[2015-Jun-01 13:26:53 -0700] [ERROR] Failed to configure port for device Suunto Cobra 3
[2015-Jun-01 13:26:53 -0700] [ERROR] Serial Port unavailable
[2015-Jun-01 13:26:53 -0700] [ERROR] SerialPort::configure failed: serial_port::stop_bits = 0
[2015-Jun-01 13:26:53 -0700] [ERROR] Failed to configure port for device Suunto ACWDevice
[2015-Jun-01 13:26:53 -0700] [ERROR] Serial Port unavailable

That last line in the error log, “Serial Port unavailable” made me start to wonder if there was contention for the serial port. As a last ditch effort I closed MovesLink2 (which I use for my Suunto Ambit) and DM5 immediately completed the data transfer. The logs looked good too:

[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] validateDateTime
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-06-29T16_22_28-0.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-06-29T16_30_32-1.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-06-29T16_50_46-2.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-06-29T16_51_38-3.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-06-29T16_58_39-4.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-06-29T17_01_50-5.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-06-29T17_06_33-6.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-06-29T17_11_05-7.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-06-29T17_22_56-8.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-06-29T17_33_07-9.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-06-29T17_35_35-10.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-06-29T17_39_59-11.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T11_11_15-12.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T11_12_40-13.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T11_14_23-14.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T11_18_58-15.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T11_22_08-16.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T11_24_35-17.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T11_26_48-18.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T11_30_02-19.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T14_55_48-20.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T14_59_59-21.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T15_04_58-22.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T15_24_16-23.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T17_09_21-24.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T17_13_46-25.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T17_17_47-26.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T17_22_18-27.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T17_25_58-28.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-08T17_30_23-29.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T07_55_13-30.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T08_00_55-31.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T08_06_15-32.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T08_11_02-33.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T08_16_36-34.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T08_23_31-35.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T08_25_49-36.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T08_31_54-37.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T08_36_55-38.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T14_18_10-39.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T14_23_23-40.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T14_28_34-41.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T14_33_55-42.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T15_01_29-43.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T15_06_57-44.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T16_29_56-45.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T16_32_48-46.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T17_13_50-47.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-09T17_16_32-48.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T10_12_55-49.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T10_17_51-50.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T10_22_20-51.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T10_25_12-52.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T10_28_28-53.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T10_32_04-54.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T10_37_25-55.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T10_41_20-56.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T10_45_59-57.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T10_48_35-58.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T10_56_21-59.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T10_58_00-60.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T11_04_24-61.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T11_07_47-62.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T11_09_38-63.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T11_24_44-64.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T11_26_39-65.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T11_29_57-66.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T11_41_20-67.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T11_44_08-68.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T11_51_08-69.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T11_53_16-70.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T11_55_02-71.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2014-08-10T11_59_32-72.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-30T12_18_28-73.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-30T12_20_35-74.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-30T12_22_44-75.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-30T12_31_22-76.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-30T12_33_00-77.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-30T12_39_30-78.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-30T12_43_35-79.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-30T12_45_13-80.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-30T12_47_08-81.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-30T12_50_08-82.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-30T12_52_00-83.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-30T12_53_19-84.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-31T10_51_35-85.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-31T10_53_27-86.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-31T10_55_36-87.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-31T11_05_31-88.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-31T11_08_10-89.xml.
[2015-Jun-01 13:32:17 -0700] [INFO] LogLibrary: Reading of log /Users/mturquette/.config/Suunto/Suunto DM5/1.2.42.5696/log-30501283-2015-05-31T11_15_40-90.xml.

I hope this helps the many users that have trouble with Suunto software. I do not feel that they have good quality control on their desktop software, and certainly there seems to be little collaboration between the teams working on dive computer software and sports watch software, which is a shame.

Endless Mobile on Kickstarter

Tue, 21 Apr 2015 00:00:00 +0000

Have you seen Endless Mobile’s campaign on Kickstarter? If not you should check it out.

Endless is following in the footsteps of efforts such as One Laptop Per Child (they employ veterans of that project), with the major exception that they are for-profit. Their mission is to get low cost, geographically- and socioeconomically-relevant computers into the hands of the billions of people that do not have access to one already. The software is focused on education, wellness, communication and other applications that are meaningful to the developing world.

In mid-2014 I met Matt Dalio and Cosimo Cecchi at the Endless offices in San Francisco because I had reviewed some patches from their team and I was curious to know more about what they were doing. I saw passion and a real belief in the goal of making the world a better place. I think that the Endless team is onto something big and that they will succeed.

The patches I reviewed in 2014 were for an Exynos-based device. I was thrilled at the thought of seeding emerging markets with a device that was not only something other than Intel’s x86 architecture, but also something other than Windows. That may sound like zealotry (and it is) but I also find that a market benefits from having its incumbent leaders challenged. Variety in the marketplace is a vital component for innovation.

Before going further I should tell you that product definition, requirements and architecture are my favorite parts of the product development process. I love to let my imagination run wild during this stage of a project, and I equally love culling those wild ideas as I start to zero in on the real, achievable product.

So you can imagine how much fun I had with the idea of creating a low-cost, robust solution for emerging markets with unreliable power, sparse or low-bandwidth connectivity and a set of users that have not grown up with a keyboard and mouse in-hand. My imagination was dreaming of mesh networks to bridge Internet connections to a stable backbone and tiny Li-Ion or AAA batteries inside the computer with enough juice to hibernate to disk in the event of a power outage.

What about device lifetime? Mobile processors are often designed to fail around the two year mark⁰. This needs to be a 10-year device. A conservative process node would be needed to get there. Investors experiencing heartburn over a device with such a lengthy upgrade cycle should be reminded that the market potential is in the billions of units.

And the cost! The original OLPC XO was originally supposed to cost less than $100 and it had a display, keyboard and trackpad in addition to lots of mechanical bits for opening and closing the lid, placing it into tablet/eReader mode and for adjusting the antennas.

How to build Endless for under $100? The requirements must be managed with an iron fist. No feature creep allowed at all.

Designing this product would have been Endless fun for me.

Fast forward to 2015 where I have been watching the launch countdown at endlessm.com with anticipation. Well the wait is over, the crowd-funding campaign has begun and the product is revealed to be …

Endless disappointment.

Intel Celeron. 30W power supply. Blue-fucking-tooth. It looks and smells like an Intel reference design.

Why am I disappointed? First is the cost. $170 USD may sound like a cheap computer to those reading this blog, but it won’t be for many in emerging economies.

I don’t know what the volume pricing is for the N2807 processor used in the device but it costs $117 at retail. Ouch. There are very capable ARM and MIPS chips out there running at gigahertz speeds with full Debian/Ubuntu/whatever support that cost $20 at volume. And it has a fan on it.

The question of price was important enough to merit a response from Endless, and I completely value what they have to say about the emerging middle class and the desire for a good product. But a cheaper processor could have shaved enough dollars off the BOM to keep the same $170 price and add market-specific features.

Another point that grieves me is the 30-watt power supply. I can understand that number for a laptop powering a display and a back light while charging a battery and browsing the web over WiFi. But 30W just for a headless Linux box?

For the target market I feel like this is a loss. Power regulation is pretty bad in some parts of the world. Even in the face of stable power, the market potential for this product is so high that we can’t dismiss the carbon footprint of rolling out millions of these devices. Putting a 5-watt supply into the wall seems more reasonable to me. This would require a lower power design.

Finally there is the lack of any hardware specifically targeting the needs of these emerging markets. I don’t see mention of a battery-backed solution to intermittent power loss. Hell, use AAA batteries. But again this would require a lower power design.

There is no camera, which is fine, but all of those Cisco ads on TV tell me that smiling children in emerging countries like to teleconference into the classroom. Endless’s own video of Radolpho shows him Skyping with his daughter.

</br>

And what about all the weird stuff I haven’t thought of yet? An AM/FM radio tuner built in. A big physical button in the front that launches a “how to use your Endless” video every time you push it. LED night light in the top of the box that trips when the electrical grid goes down. An emergency hand crank that generates power over the USB port to charge up a phone. A handle on the top so you can carry it around without dropping it and a little shock-proofing with rubber around the edges.

And then there is this, from the top of the Kickstarter campaign:

A few years ago, on a trip to India, we had an epiphany: If we ran a full desktop operating system on the inexpensive processors that power smartphones, and plugged them into the TVs people already own, we could make a computer that is affordable to more people.

Yes! Do that! Do just that! A GSM modem with a general purpose OS hooked up to a TV. But where is the cellular connectivity in this product? For that matter where is the mobile processor?

Hopefully they are coming in a future product. Maybe they are coming in a future partnership where Endless licenses software out to a phone manufacturer for “TV Computer Mode”.

Motorola already tried exactly that that with their Webtop/Lapdock products; an Android smartphone with an HDMI/USB docking station. Plug the phone into the dock and it runs a custom flavor of Ubuntu in parallel to Android for the full desktop experience with a keyboard and mouse.

Motorola targeted it at the high end of their product portfolio – people that already own a Macbook and don’t need a janky Ubuntu desktop – and failed. Emerging markets are a different ballgame altogether. Convergence devices at low cost are key.

Alright, I’m done poking at the fine efforts of the people at Endless. I even retract my use of the phrase “Endless disappointment”, despite the feeling of authorial superiority that it gives me.

The truth is that Endless is a software company, while any sense of deflation I have over the Kickstarter campaign comes soley from the hardware.

Their software story appears well thought out and targeted at its users. Looking at their About Us page it is clear that Endless is investing more of its human capital into software engineering than into electrical engineering. Well-targeted, easy to use and easy to deploy software is their big differentiator.

My hope is that they eventually grow into the kind of product company that focuses equally on both sides of the coin. I also hope that they target users at the lower end of the economic scale and in more rural locales because those are such interesting problems to solve. And most of all I hope that they succeed in their efforts to connect the rest of the world.

Please check it out if you haven’t already. I hope that you like what you see and decide to pick up one of their devices or at least contribute to their fund raising goals. I know I have.

Footnotes

0: The reason for short silicon lifetimes in the mobile application processor space is complicated, and only partially a scheme to steal your dollars. Reasons include operating a chip right at the lowest voltage margin to save power and extend battery life, as well as exposure to over-current and over-heating thermal events as well. Also dropping your phone in the toilet doesn’t help.

QNAP QDK QPKG and the GPL

Wed, 11 Mar 2015 00:00:00 +0000

This is the first part of a two-part post on what it took to get Plex Media Server running on QNAP TS-219P2+ NAS device. Be sure to read the second part here.

I wanted to run Plex Media Server on my QNAP TS-219P2+, which is running Debian Wheezy (as described in a previous blog post). For my particular hardware Plex provides a binary package using the QNAP qpkg format.

This led me on a merry adventure to understand the QDK (QNAP Development Kit!) and the QPKG (QNAP package!) format. What I found was a possible GPL violation. I have rectified that mistake to a degree by hosting the code on GitHub. In addition I’ll outline my thought process during this work and the reverse engineering methods I used below.

Plex Media Server

I started by downloading the software that I wanted to run on my NAS at the Plex download page. For those following along simply run:

wget https://downloads.plex.tv/plex-media-server/0.9.11.7.803-87d0708/PlexMediaServer_0.9.11.7.803-87d0708_arm-x19.qpkg

OK, so we have this … thing. What is it?

$ file PlexMediaServer_0.9.11.7.803-87d0708_arm-x19.qpkg
PlexMediaServer_0.9.11.7.803-87d0708_arm-x19.qpkg: data

Not terribly useful. Let’s take a closer look at the first 160 bytes of this file.

$ hexdump -C PlexMediaServer_0.9.11.7.803-87d0708_arm-x19.qpkg | head
00000000  23 21 2f 62 69 6e 2f 73  68 0a 77 72 6f 6e 67 5f  |#!/bin/sh.wrong_|
00000010  61 72 63 68 28 29 7b 0a  20 6c 6f 63 61 6c 20 77  |arch(){. local w|
00000020  72 6f 6e 67 5f 61 72 63  68 5f 6d 73 67 3d 22 57  |rong_arch_msg="W|
00000030  72 6f 6e 67 20 61 72 63  68 69 74 65 63 74 75 72  |rong architectur|
00000040  65 3a 20 50 6c 65 78 4d  65 64 69 61 53 65 72 76  |e: PlexMediaServ|
00000050  65 72 20 30 2e 39 2e 31  31 2e 37 2e 38 20 69 73  |er 0.9.11.7.8 is|
00000060  20 62 75 69 6c 74 20 66  6f 72 20 61 72 6d 2d 78  | built for arm-x|
00000070  31 39 22 0a 20 65 63 68  6f 20 22 49 6e 73 74 61  |19". echo "Insta|
00000080  6c 6c 61 74 69 6f 6e 20  41 62 6f 72 74 2e 22 20  |llation Abort." |
00000090  26 26 20 65 63 68 6f 20  22 24 77 72 6f 6e 67 5f  |&& echo "$wrong_|

Now that was useful. It looks like at least the beginning of the file is a shell script. In that case the whole thing is a likely meant to be executed. Before we try running this script on our system as root and trash the hard drive, let’s take an even closer look.

$ less PlexMediaServer_0.9.11.7.803-87d0708_arm-x19.qpkg

#!/bin/sh
wrong_arch(){
 local wrong_arch_msg="Wrong architecture: PlexMediaServer 0.9.11.7.8 is built for arm-x19"
 echo "Installation Abort." && echo "$wrong_arch_msg"
 /sbin/log_tool -t2 -uSystem -p127.0.0.1 -mlocalhost -a "$wrong_arch_msg"
 echo -1 > /tmp/update_process && exit 1
}
arch_ok(){
 local cpu_arch=$(/bin/uname -m)
 [ $(/usr/bin/expr match "$cpu_arch" "armv5tel") -ne 0 ] || return 1
}
/bin/echo "Install QNAP package on TS-NAS..."
/bin/grep "/mnt/HDA_ROOT" /proc/mounts >/dev/null 2>&1 || exit 1
arch_ok || wrong_arch
_EXTRACT_DIR="/mnt/HDA_ROOT/update_pkg/tmp"
/bin/mkdir -p $_EXTRACT_DIR || exit 1
script_len=1048
/bin/dd if=${0} bs=$script_len skip=1 | /bin/tar -xO | /bin/tar -xzv -C $_EXTRACT_DIR || exit 1
offset=$(/usr/bin/expr $script_len + 20480)
/bin/dd if=${0} bs=$offset skip=1 | /bin/cat | /bin/dd bs=1024 count=96286 of=$_EXTRACT_DIR/data.tar.gz || exit 1
offset=$(/usr/bin/expr $offset + 98596375)
( cd $_EXTRACT_DIR && /bin/sh qinstall.sh || echo "Installation Abort." )
/bin/rm -fr $_EXTRACT_DIR && exit 10
exit 1
control.tar.gzLOTSOFBINARYDATA...

After control.tar.gz the file is packed with binary data. To be precise the control.tar.gz text is also part of that binary data, which I will explain later.

Inspecting the shell script clearly shows that the blob is meant to be executed and it will install Plex Media Server to some hard-coded path. That path makes sense if you run the QNAP factory software, but I run Debian. Perhaps more intriguing is this qinstall.sh script. What does it do and where can I find it?

qpkg installer and the qdk

Some Google searching quickly taught me that qinstall.sh is part of the QNAP Development Kit, aka QDK. OK, great an sdk. How nice of them. Now where can I find it?

After searching Google, the QNAP forums and the QNAP wiki for this elusive sdk I came to some conclusions:

The main place to get the QDK is as a zip file attached to this forum post
It is licensed under the GPLv2 and I quote, “making it completely open and available for anyone to use”
The only way to get the QDK source code is to install the QDK. The only way to install the QDK is to install it via your previously existing installation of the QDK.

Wait, what?

I am not a lawyer but I wonder if this constitutes a GPL violation? QNAP distributes a binary version of GPL-licensed source code but not the source in a reasonable format.

No problem! Just ask them for the source. I think that is what the FSF would want me to do.

Well that went nowhere. I think it would be faster to reverse engineer it.

h4king the Plex qpkg

The forum post I mentioned earlier has a link to a pdf document. Appendix A on page 35 shows the qpkg format and page 36 has an example header script that looks shockingly like the one prepended to the binary data in our Plex qpkg!

Appendix A tells us that both the Control section and the Data sections are gzipped tarballs wrapped in non-compresses tarballs. Genius. This jives pretty well with the header script that we looked at earlier. Maybe we can just use tar to get at this data! We’ll need to remove the header script from the binary package first. Examining the header script we can see that this is something it does already. The interesting bits are script_len and offset. Removing some QNAP-specific crap from the header script embedded into the Plex qpkg yields:

#!/bin/sh
_EXTRACT_DIR="/mnt/HDA_ROOT/update_pkg/tmp"
/bin/mkdir -p $_EXTRACT_DIR || exit 1
script_len=1048
/bin/dd if=${0} bs=$script_len skip=1 | /bin/tar -xO | /bin/tar -xzv -C $_EXTRACT_DIR || exit 1
offset=$(/usr/bin/expr $script_len + 20480)
/bin/dd if=${0} bs=$offset skip=1 | /bin/cat | /bin/dd bs=1024 count=96286 of=$_EXTRACT_DIR/data.tar.gz || exit 1
offset=$(/usr/bin/expr $offset + 98596375)
( cd $_EXTRACT_DIR && /bin/sh qinstall.sh || echo "Installation Abort." )
/bin/rm -fr $_EXTRACT_DIR && exit 10
exit 1

So script_len gets us past the script where can untar some stuff, likely the Control section of the package. Since we all have the tar file format committed to memory we can even postulate that the control.tar.gz seen at the end of the script is really the filename of the first (and only) file within the Control section tarball. It is not a plaintext portion of the header script, but instead the first bytes of the Control section.

The next section of the header script adds script_len to a magic number of bytes and we unpack an additional tarball, likely the Data section of the package since we store this data on disk as data.tar.gz. Finally qinstall.sh is run, which we don’t have. Let’s ignore qinstall.sh for now. It probably sucks anyways. The following not-very-portable script emerged from me stripping the config header, removing the QNAP-specific bits and dumping the uncompressed files:

This fun little script counts how many lines of text precede the string control.tar.gz, removes aforementioned lines and saves the modified image to a temporary file in directory ‘out’. From this point it is a simple matter of unpacking an uncompressed tarball which contains a compressed tarball, control.tar.gz, then skipping 20KB from the beginning of this temporary file and doing the same thing over again for data.tar.gz.

There is some miscellaneous data at the end of the blob which will cause tar to spit some warnings, but at the end you should have a nice control.tar.gz and a data.tar.gz. Yay! Notice that the 20KB offset for data.tar.gz is still hard-coded into my script. I think that these offsets might be in multiples of 5KB or 10KB and I’ll come back to that later.

Inspecting data.tar.gz reveals all of the files needed for Plex. Mission accomplished. Hurray!

I’ll revisit actually making Plex work on Debian in my next post. For now let’s see if the script above is generic enough to unpack any qpkg. A fun test might be trying to unpack the root cause of all of my problems: the QNAP QDK QPKG!

breaking the cycle

The short answer is it doesn’t work straight off:

$ ./unpack_qpkg.sh QDK_2.2.2.qpkg
Files generated in directory 'out'
Removing 14 lines of header script before control.tar.gz
77+1 records in
77+1 records out
39709 bytes (40 kB) copied, 0.000428835 s, 92.6 MB/s
./
./qpkg.cfg
./md5sum
./package_routines
./qinstall.sh
0+1 records in
0+1 records out
18+1 records in
18+1 records out
19229 bytes (19 kB) copied, 0.00126722 s, 15.2 MB/s
19229 bytes (19 kB) copied, 0.00814744 s, 2.4 MB/s
$ ls out/*
out/control.tar.gz  out/newQDK_2.2.2.qpkg

out/control:
md5sum  package_routines  qinstall.sh  qpkg.cfg

out/data:
data.tar.gz
$ tar -tzf out/data/data.tar.gz

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Well control.tar.gz seems to decompress and unpack just fine. Why not data.tar.gz? At a guess, maybe the offset is wrong? Inspecting the first lines of QDK_2.2.2.qpkg shows us:

#!/bin/sh
/bin/echo "Install QNAP package on TS-NAS..."
/bin/grep "/mnt/HDA_ROOT" /proc/mounts >/dev/null 2>&1 || exit 1
_EXTRACT_DIR="/mnt/HDA_ROOT/update_pkg/tmp"
/bin/mkdir -p $_EXTRACT_DIR || exit 1
script_len=627
/bin/dd if=${0} bs=$script_len skip=1 | /bin/tar -xO | /bin/tar -xzv -C $_EXTRACT_DIR || exit 1
offset=$(/usr/bin/expr $script_len + 10240)
/bin/dd if=${0} bs=$offset skip=1 | /bin/cat | /bin/dd bs=1024 count=29 of=$_EXTRACT_DIR/data.tar.gz || exit 1
offset=$(/usr/bin/expr $offset + 29369)
( cd $_EXTRACT_DIR && /bin/sh qinstall.sh || echo "Installation Abort." )
/bin/rm -fr $_EXTRACT_DIR && exit 10
exit 1

The magic offset value is 10K bytes, half of what it was for the Plex package. Replacing 20480 with 10240 in unpack_qpkg.sh gets things working:

$ ./unpack_qpkg.sh QDK_2.2.2.qpkg
Files generated in directory 'out'
Removing 14 lines of header script before control.tar.gz
77+1 records in
77+1 records out
39709 bytes (40 kB) copied, 0.000439275 s, 90.4 MB/s
./
./qpkg.cfg
./md5sum
./package_routines
./qinstall.sh
28+1 records in
28+1 records out
29469 bytes (29 kB) copied2+1 records in
2+1 records out
, 0.00689956 s, 4.3 MB/s
29469 bytes (29 kB) copied, 0.00411324 s, 7.2 MB/s
$ tar -tzf out/data/data.tar.gz
./
./.qpkg_icon.gif
./template/
./template/arm-x09/
./template/qpkg.cfg
./template/config/
./template/shared/
./template/shared/init.sh
./template/icons/
./template/x86_64/
./template/x86/
./template/package_routines
./template/arm-x19/
./qdk
./bin/
./bin/qbuild
./.qpkg_icon_80.gif
./scripts/
./scripts/qinstall.sh
./.qpkg_icon_gray.gif

gzip: stdin: decompression OK, trailing garbage ignored
tar: Child returned status 2
tar: Error is not recoverable: exiting now

Again the warnings at the end are fine. The point is that we can now access the source files to the QDK, including qinstall.sh. Do we need that crap anymore? Probably not. We already unpacked the Plex qpkg manually. However this whole GPL-code-wrapped-in-a-binary-package thing has got me irked. It feels wrong. If only there was something I could do… what about hosting the QDK sources on GitHub? Done.

While I do not plan to use these scripts ever, I don’t want someone else to run into the same boostrapping problem that I did. Now anyone that wants to use these scripts to build, modify or unpack qpkgs can do so without having to first buy a QNAP device and install the QDK qpkg.

conclusion

Honestly, the qpkg format is crap. That is no surprise as the QNAP factory firmware is also crap. I like the device. I’d buy it again. But there is no reason to reinvent something as fundamental as package management for your embedded device.

Why not use ipkg/opkg? Or just go crazy and base your Linux distro on something reasonable like Debian or Fedora? I mean we’re talking about a 2.0GHz processor with 512MB of RAM. We’re hardly embedded anymore. Stop rolling your own shit and use something standardized!

The problems are a bit more endemic than all that however. Take for example that the primary means of distribution for the QNAP QDK is either through the web interface of QNAP’s NAS management software, or through a stickied post on a forum. A freakin’ forum post! Where is the version control? How does one submit a change request and try to improve the scripts? All of that stuff is missing.

And finally comes the GPL compliance. I might be misinterpreting the copyright license but it seems clear to me that in light of a request for source code I should be able to get it. Better yet QNAP should just host it somewhere.

Le sigh.

If you made it this far then I hope you found this post helpful or at least interesting. For the next post I’ll focus on the happier issue of making Plex work on my NAS with my Chromecast and PS4.

Ciao!