phoenix felix, et tu

Making cycling slightly less boring

2023-01-13T00:00:00+00:00

A dashboard-in-the-terminal for a bike sensor.

Context

I'd rather be running and/or outside but an injury earlier this year led me to spend more time of late on a turbo trainer (this is basically a metal frame that holds a bike's back wheel while you cycle in place with some magnetic resistance) instead.

To keep track of how much work is being done while on the turbo, I got a coin-sized box with no display that attaches to bike's crank shaft or back wheel called a cadence sensor. There are many cheap sensor devices that use Bluetooth Low Energy (BLE) or the proprietary ANT protocol.

This kind of sensor is super simple. It keeps track of how many revolutions the wheel/crank has made and how many seconds have passed since the thing last switched on and broadcasts those two numbers (along with some headers) to whatever BLE/ANT transciever they've connected to each time it detects another revolution.

Unfortunately at the time of writing the available Android apps that talk to these sensors seemed to exclusively be some combination of stuffed with irrelevant ads, sketchy as fuck from a data-gathering/tracking perspective or just didn't work at all.

At the same time, I was looking for an excuse to build a terminal-user-interface (TUI) with the rich library to try it out (as opposed to the python-prompt-toolkit library that I would usually reach for).

cycling-cadence-display

So I did the (completely un)needful and wrote a TUI dashboard of info about the bike sensor showing things like how long it's been on, the instantaneous/mean revoluitions per minute (RPM). It also renders a plot of the RPM over time in ascii (via gnuplot) and trivially let's you export all the time-series data locally.

When cycling it looks like the gif above. Usually I've got a small, semi-transparent terminal-app window with this running and some TV/film thing on in the background of a wall-mounted screen just in front of the bike which is slightly more interesting than starting at the wall behind it.

Of course for using it on a phone instead you can termux which looks like this:

(As an aside, termux and it's API are a marvel for turning an Android phone into something much more useful with bash, cron, ssh, apt, access to storage/contacts etc.).

Conclusion

Overall I found the rich library intuitive to use and well documented - im my opinion it is ready to be used at work/in production to display a dashboard of information. It's not really built to for interactive interfaces though, for that you're really looking to use textual instead which is a for-profit org (who also happen to have written the rich library for use in textual).

Personally I found myself doing longer "cycles" on the bike more often since I started using the dashboard. Ironically I got over the injury not long after this project so am back to hoofing it around the local woods / thames-path most of the time but the code is on Github and available on PyPI.

Song & Spell

2020-12-27T00:00:00+00:00

A key-logger that will play an audio track when particular words are pressed and a web-app to configure the words and songs.

Introduction

The volume of information available to anyone with a screen and an internet connection is hard to comprehend; the vastness is matched only by the variety.

It's frankly miraculous that so much information is so readily available, giving anyone with a web-browser the abilities of a minor deity from Greek mythology.

And yet as a mere mortal it's also difficult to navigate without getting sucked into emotional little corners designed to illicit a response regardless of the veracity of the content.

In my humble opinion most adults are not well prepared for this, and small people with malleable minds don't stand a chance. Aside from that, the NHS guidance on screen time in general can be summarised as "nothing for those under two years old and very little for those two to five".

And yet it's great fun to sing and dance to music on your phone with a small person! Not only that but a genre I would call toddler-folk-music is actually pretty decent.

Giving a small person your phone to play something on a FAANG+ music service seems pretty dicey to me: you're trusting a commercial company whose interests are not likely to align with your own, not to mention trolls and the wider community of people you don't know. Of course, you can just use the device for the smalls, but it would be better if you didn't have to and they had the agency to choose something for themselves.

Anyway, it would be great if there was a more limited way for small people to:

play a limited selection of audio media that you can choose in advance
positively interact with technology without using a screen
learn the alphabet and encourage spelling

Song & Spell

Enter the raspberry-pi-400: a cheap, general purpose PC with on-board wifi, bluetooth and I/O pins that is enclosed in its own keyboard 🤯

Song-and-Spell is a script that turns this wonder of the information age into a way to do just this. The idea is that you can use it without a screen. When the PC powers on it will automatically connect to a bluetooth speaker, use it as an audio sink and start a web-app from which you can:

upload files with songs to be played when certain words are typed on the keyboard
download the audio track for videos directly from youtube to be played when certain words are typed on the keyboard
set the speaker-volume (or just mute it)
trigger playing the the songs directly, or remove them

The web-ui looks like this on my phone:

It does just this, no more or less. It was great to see it in action in the past few days, after ten minutes one happy little beta-tester could spell (type) words like "lizard" without assistance even though they were previously unaware of those letters.

Conclusion

I'm pretty happy with how this project turned out. Most of the coding was the product of one evening while trying to get into the Christmas spirit (Die Hard was on in the background) just after a government announcement had forbid our upcoming plans. It then took considerably more effort to correctly setup the headless bluetooth & audio sink. The code (and installation guide) is on Github and also available on PyPI.

There are some things I'm still not sure about: since there is no screen there should probably be some audio feedback after a key is pressed, or maybe just an LED flash. A beep could become annoying... someone speaking the letter was my first idea but these days phonics are taught instead. I've also ordered stickers for the keyboard with lower-case letters. If you have any insights about user-interfaces for small people then drop me a line or feel free to fork the project. Also, in case it's not obvious the web-server runs insecurely over HTTP and isn't designed to be accessible outside of your LAN. To secure it I'd recommend something simple like running nginx with SSL termination in front with .htaccess for authentication.

This is such a tiny facet of what this device can do, and I see it as just the beginning of a journey of education about technology that will of course, one day, when they're ready, include a screen and an unfettered internet connection.

okcli: a SQL command line interface

2019-05-19T00:00:00+01:00

Introduction

At work, the RDBMS de-jour for many internal apps is - for better or worse - Oracle® Database.

However, the provided tooling for Oracle (sqlplus) lacks several features that I consider to be essential to be productive (e.g. schema-aware auto-complete for SQL-statements) and since there didn't seem to be any viable open-source alternatives I started a side-project to build something better.

okcli

After being used internally for over a year the result of that project, okcli, was made publicly available under a BSD-3 license earlier this week and announced on the company tech blog. The gif below shows it in action.

Conclusion

okcli is is now available on Github and PyPI.

Adding icons to i3 workspace names

2018-11-25T00:00:00+00:00

Introduction

Without getting bogged down in the linux display-manager holy war, I use the i3 window manager every day at both home and work since... well it's a great display manager!

Additionally, its design makes it easy to extend.

A small gripe

By default i3 shows a bar at the bottom of the screen with the workspaces (the i3-bar) where each workspace is identifed by a static number. This is useful but requires you to remember what apps are running in each workspace... which is too much for my perforated memory.

And so

I wrote a little script to update the workspace name dynamically based on what apps are running in it, and since icons are much more succint than a concatenated list of app-names you can also configure it to show an icon instead of a particular app name.

Icons

The Font Awesome icons are freely available License and there are thousands of them. (If they seem familar it is because they're included in the Bootstrap library and hence all over the place).

Examples

There is an example i3-bar image at the top of this article, more screenshots are below.

Conclusion

Finally, one less thing to remember!

Installation instructions are available on the github page and the package is available on pypi.

How I use Vim

2017-11-16T00:00:00+00:00

This is my .vimrc file. 
There are many like it, but this one is mine.

Introduction

I've used vim as my primary editor/IDE for the past 10 years. Back then I was writing data-analysis code over an ssh session to hosts in the SLAC computing center in California while sitting several thousand miles away in front of a terminal in the UK and a friendly post-doc showed me this nifty looking editor that he was using...

Since then I've been tempted away by a few shiny IDEs but always give up and come back to the venerable editor (one exception to this is intellij as it's a wonderful refactoring-engine for Java. By comparison Pycharm is rubbish IMHO).

And so I thought it might be useful to post about my .vimrc file explaining what the various plugins and configuration options do. Since I've been writing a lot of python over the past year there's a bias here.

This scope of this post creeped up but it is mainly:

a walk-through of my .vimrc file (anything not explained is hopefully obvious from the comments)
a litany of the useful vim-plugins therein and how/why to use them
some general vim tips

Installation

If you want to skip to the goods you can clone my dot-files repo and run vim.sh to download and install all of the plugins below.

Updating to a more recent version of vim is usually trivial though, for Debian/Ubuntu you can do:

sudo add-apt-repository ppa:jonathonf/vim
sudo apt update
sudo apt install vim

If you don't have vim-8 installed all the plugins except ALE will work anyway.

Plugins

Vundle

Let's start with Vundle: a plugin manager for vim that also fetches plugins straight from Github. We start by setting the runtime-path (where Vundle will download the plugins files).

set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()

Plugin 'gmarik/Vundle.vim'

Navigating the local filesystem

NERDTree

NERDTree is for exploring the file-system and is sometimes useful for jumping between files in your current project. Personally I don't like this being open on-startup.

Plugin 'scrooloose/nerdtree'

" Open/close NERDTree Tabs with 
let NERDTreeDirArrows=0 
" To have NERDTree not open on startup
let g:nerdtree_tabs_open_on_console_startup = 0

A few highlights:

to open a new NERDTree tab use :NERDTree
to open a file under the cursor in a v-split pane hit s.
closes like a normal vim tab

ctrlp

ctrlp is the most useful way to navigate between files while working on a project IMHO.

Plugin 'ctrlp/ctrlp.vim'

" search ctrlp with regex by default
let g:ctrlp_regexp_search = 1
let g:ctrlp_clear_cache_on_exit = 0

Quite simply, press Ctrl+p while in normal mode to open a buffer that will match to filenames in the directory you're in (and recursively in it's sub-directories). The options I've set will:

allow you to use regex to match filenamese
cache the directory index (F5 to refresh). This is useful if like me you ever have to work with a shared NFS mounts on which the indexing operation can be sluggish

Version control with Git

vim-fugitive

The vim-fugitive repo says it better than I could:

I'm not going to lie to you; fugitive.vim may very well be the best Git wrapper of all time

As someone that uses vim-fugitive to interface with git every day I don't consider this an understatement. This is a large topic but my top highlights are:

:GCommit, :Gpush, :Gpull wrap git {commit|push|pull}
:Gblame: opens a vertical-split window with git-blame info, and you can then open a commit and see the diff in another bugger
:Gdiff: see diff output for the current file.
You can pass any command directly to git by starting a command with :Git, for example add the current file to the change-set with :Git add path/to/current/file or just :Git add %
we're using fugitive to display the git branch in the status line with the lightline plugin (see below).

Plugin 'tpope/vim-fugitive'

vim-gitgutter

vim-gitgutter complements fugitive by displaying the git sign for lines that have added/removed/updated in the current file (using the output git diff)

Plugin 'airblade/vim-gitgutter'

Bonus: configure git to use vimdiff

vimdiff is a diff-tool that opens a vertical-split window in vim. I find it much easier to see what's changed than when using git's in-built diff tool. Here we jump to my .gitconfig:

[diff]
    tool = vimdiff
[difftool]
    prompt = false
[alias]
    d = difftool

We can now look at the output of git diff in vimdiff. For example, to diff the curreent index with two commits ago you can do

git d HEAD~2

Highlights of this method is that when navigating the file in either window:

the files can be edited as if they were normally opened files in vim
"dp" will put the change under the current cursor to the other file
"do" with obtain the change from the other file and apply it to the cursor position in the current file

Status Bar

lightline

lightline allows you to configure a snazzy status bar (at the bottom of the window). The snippet below will show:

a color-scheme named wombat for the status-bar itself
the vim-mode (INSERT/NORMAL/VISUAL)
the git-branch if the file in the current buffer is part of a git repo (more on this later)
the python-virtualenv in use (more on this later).

Plugin 'itchyny/lightline.vim'

" set lightline to include git-branch
let g:lightline = {
      \ 'colorscheme': 'wombat',
      \ 'active': {
      \   'left': [ [ 'mode', 'paste' ],
      \             [ 'gitbranch', 'readonly', 'filename', 'modified' ] ,
      \             [ 'venv', 'readonly'] ]
      \ },
      \ 'component_function': {
      \   'gitbranch': 'fugitive#head',
      \   'venv': 'virtualenv#statusline'
      \ },
      \ }

" Always show statusbar
set laststatus=2

This produces the following status bar:

Linting and syntax highlighting

Many people use syntastic for syntax highlighting in vim. It has one drawback which is that it runs synchonously in the main UI thread. ALE (Asynchronous Lint Engine) is arguably an asynchronous successor.

ale

ALE let's you specify specific linters for different languages. Below I've specified flake8 for linting and autopep8 for fixing syntax "errors".

Plugin 'w0rp/ale'

" pip install flake8
let g:ale_linters = {'python': ['flake8']}
let g:ale_fixers = {'python': ['remove_trailing_lines', 'trim_whitespace', 'autopep8']}

Note:

this assumes flake8 and autopep8 are already installed
any config for either in ~/.config/pep8 or ~/.config/flake8 will be respected.
the linting is done asynchronously
to apply the fixer (in this case autopep8) use the command :ALEFix

This produces:

Python

Here are a few plugins that are great for python specifically.

jedi-vim

jedi is probably the best auto-completion library for Python that I've ever used (including Pycharm). jedi-vim requires that your vim binary was compiled with the python/python3 option but that's pretty much guaranteed unless you're compiling vim yourself.

" ------ jedi-python ------"
Plugin 'davidhalter/jedi-vim'

let g:jedi#popup_on_dot = 0
map <Leader>b Oimport pdb; pdb.set_trace() # BREAKPOINT<C-c>

Anyway, this is another large topic but here are a few highlights:

autocomplete (C-x-o)
jump-to variable/method/function/class definition (leader+g)
- this works with defined on PYTHONPATH (including in your virtual-env) but not egg files weirdly)
find-usages (leader+f)
jump-to method/class/function/variable declaration (leader+d)
rename (leader+r)
show docstring with K
As a bonus: leader+b is remapped to add a pdb-breakpoint.

Below shows a few of these in action:

vim-virtualenv

vim-virtualenv allows you to set the python-virtualenv used in vim (and by extension jedi-vim). You can set/unset it with :VirtualEnvActivate virtual_env_name or :VirtualEnvDeactivate.

tagbar

Personally I find tags of limited use, but they're handy for exploring a new or otherwise un-familiar codebase. Here we use tagbar.

" --- C-tags integration --- "
Plugin 'majutsushi/tagbar'

Note, this requires ctags or similar to be installed, so for Debian/Ubuntu:

sudo apt-get install ctags

Then we can toggle the tagbar window with :Tagbar and navigate code at a higher level of abstraction, as shown below.

Other useful plugins

Honourary mention to vim-go and rust.vim which provide great integration for those languages too.

Also, for keeping a work-log or wiki I can't recommend vimwiki enough: it is like an even-better org-mode ;-)

persistent-undo

This isn't a plugin but something I find useful for essentially caching the list of updates to a file such that if you close and re-open a file you can undo the last few edits (note, most vim distros in the wild will be compiled with the persistent_undo")

" Keep undo history across sessions by storing it in a file
if has('persistent_undo')
    let undo_dir = expand('$HOME/.vim/undo_dir')
    if !isdirectory(undo_dir)
        call mkdir(undo_dir, "", 0700)
    endif
    set undodir=$HOME/.vim/undo_dir
    set undofile
endif

Colours

It's possible to set the colour-scheme directly in vim but these days most terminals (terminator for linux and iTerm2 for mac) have solarized colour schemes baked in. Once set, vim will use these and the job is done.

tmux

On a different note, tmux is a wonderful way to multiplex teminal sessions and I find it complements vim very nicely.

I could probably fill an equally lengthy post about tmux.

vim miscellany

Again, this is a much larger topic but here are a few tricks I find useful:

:sp and :vs will split the current window horizontally/vertically
gg=G will auto-indent the entire file
e# navigates to the previously opened file in the current buffer
vim macros are pretty rad
you can use % as a shortcut for the current file-path in any command in command mode e.g. get it's md5-hash :! md5sum %
you can read from stdin via a UNIX pipe with vim -, e.g.

>> curl example.com | vim -

Motivation

Warning: this section is opinionated.

You might ask: why bother with all this? Surely [insert GUI-based IDE + whatever else] is good enough?

Well, it may be good enough. But often when working on some data-analysis or building some service what matters far more than the length of time spent working is how long I can spend in a state of flow. This usually means being focussing on one thing; being un-distracted. Using this editor is a great way of staying focussed and minimising the operational overheads of software engineering.

Things I find good for staying focussed:

Keeping my eyes fixed on one application
Having things like auto-complete and linters
Using as few keystrokes as possible for common tasks (eg. VCS commands, running tests)

Conversely, these have the opposite effect:

Navigating menus
Using a mouse at all
Switching to another terminal window or (much) worse a web-GUI

So in a way these plugins and hotkeys are a way of making it easier to stay in the flow.

vim-8 is awesome

Most package managers will distribute binary versions of vim 7.4.XXXX which is fine. There are many reasons to update to vim version 8.X is but perhaps the biggest is that it allows tasks to be performed outside the main UI thread. All of the plugins we'll discuss will work fine with either version except for the ALE plugin that requires version 8+. Also, certain commonly-enabled compilation flags (python and persistent_undo are assumed).

Wrapping up

Putting it altogether, what I focus on while working usually looks something like this.

If you've made it this far I hope found it useful.

Muting the news

2017-02-04T00:00:00+00:00

TL; DR

Mute the news when listening to the radio on a thing running Android.

Introduction

As someone who listens to a lot of BBC Radio 6, I highly recommend pretty much all of the weekend shows, they're usually full of really great music.

As part of the radio service, the BBC News butts-in with a 2-3 minute broadcast at hourly intervals, full of current events to help keep the public informed. Now, aside from the known adverse effects of consuming too much news, the tone of the broadcast is usually completely at odds with what I'm doing at the time. Why should I be made aware of what someone said in London or the latest tweet of a large-tangerine with small hands? Why must the minds of millions be made aware of minor crimes in major cities? At times during the run up to the Brexit referendum it felt like listening to the Nigel Farage hate-trumpet.

In consuming media this way a person acquiesces control to the broadcaster, allowing them to shape your reality with great subtlety, often making you morbidly aware of seemingly endless minutiae for the political or commercial gain.

But I digress.

For some time I've just reached over and muted the radio when the news jingle starts. But last weekend I had a few hours to kill in a pub before meeting Sarah to play Hive. So obviously I wrote an Android App (below) to mute the streaming-music volume at set times of the day for a configurable period of time. Note this doesn't affect the phone-ringer or alarm volumes, only streaming music like playing the radio through a web-browser and can be (de) activated with a toggle.

mute.time

The source-code is on github, and the (free, open-source, ad-free) app is published on the Android play-store.

2016 in Books (part 2)

2017-01-09T00:00:00+00:00

Introduction

Anyone looking for a book?

Below is a brief review of some books I've read in the second half of 2016.

This makes just over 40 books read in 2016, next year I'm tempted to include some graphs and maybe a few journal articles.

Non Fiction

Dreams of a Final Theory: Steven Weinberg

In the late 80s a particle-physics experiment called the Superconducting Super Collider (SSC) had it's funding cut, despite the fact that construction was well under way. This was most likely because Texas (where it was being built) also had a lot of money being spent there as part of the International Space Station project and the ensuing political difficulties that followed the concentration of resources into just one state. Over many years a gradual but consistent decline in the Department of Energy budget put them at odds and the SSC lost.

The main thrust of this book is Weinberg (who won part of the Nobel Prize for the theory of Electroweak Symmetry Breaking) arguing for the SSC over the scientifically bereft ISS, along with a now standard popular-science-level Particle Physics introduction and a lot of less-standard ranting about religion and philosophers of science.

There are many parallels between then and now and you can almost replace Tevatron/LEP with LHC and the arguments for building the next-generation of particle physics experiments after the LHC and the existential crisis facing Particle Physics experiments and theory in the absence of significant new physics from the LHC (except for a SM-like Higgs of course) are very relevant.

The Elements of Computing Systems: Building a Modern Computer from First Principles: Noam Nisson & Simon Schocken

This book/course (that also goes by the moniker 'From Nand to Tetris') took a month but what a rewarding endeavour. The first chapter engages the reader to derive logic operators (AND, NOT, OR) purely in terms of NAND (like a transistor) and gives a scaffolding to build:

an ALU that runs on (a provided) emulator;
a 16-bit assembler for a somewhat simplified instruction set that runs on said emulator;
a stack based VM that compiles to said 16-bit instruction-set;
a high level Java/.Net style compiler/language spec to run on the stack VM;
something between a light-weight OS and a library for the high level language for interacting with memory mapped input devices, allocating memory etc.

I can't recommend this book enough for understanding each level of abstraction in a running computer.

Boss Fight Books: Spelunky: Derek Yu

Spelunky is a game with which I became somewhat obsessed years ago, logging many, many hours of play. Having studied (and read quite a bit about) its procedural level generation in the past there was still plenty of things to learn in this book by its designer. Absolutely fascinating.

Surreal Numbers: Donald Knuth (Yes, that Donald Knuth.)

This is a rather strange book full of proofs about a branch of mathematics called Conway's numbers. It's also the first work of fiction to feature a new result before being published in any journal. Frankly, the love story is somewhat laughable for its lack of credibility but the ambition here is very noble: in wrapping up Number Theory derivations in a fictional narrative Knuth really, really wants to reach people that a math journal cannot. A+

Eyes on the Sky: Francis Graham-Smith

A whirlwind tour across the telescopes of the electromagnetic spectrum from Gamma-Ray bursts to Radio wavelengths and from the classic lenses of antiquity to the cutting edge of astronomy. This is probably a bit specialist but as a nerd who did an undergraduate courses on modern telescopes I loved it. A+

Weapons of Math Destruction: Cathy O'Neil

A somewhat alarmist review of the ways in which tasks formerly performed by humans are now being done by (usually quite rubbish) automation. The author is quite opinionated and makes quite a few un-substantiated claims but the basic thesis that automation tends to affect those who are poor and un-represented far more than the wealthy elite thus exacerbating wealth inequality in society is well researched and argued for. For example, when getting insurance or a bank loan if you go to an expensive, boutique service you'll probably still be seen by a human, rather than an algorithm that can make mistakes or codify the (hopefully subconscious) biases and prejudices of those who made it.

Effective Python

A good complement to 'the Hackers guide to Python' this book really helped me think more like a python-ista before starting a new role in which I get to be one.

The 59 Icosahedra: H.S.M. Coxeter, P. Du Val, H.T. Flather, J.F. Petrie

This charity shop gem is really an extended journal article exploring the maths of the Icosahedra (20-sided 3d-shapes) group members and operation, complete with a full enumeration of each many diagrams and in some cases pictures of paper models made by the great Coxeter. It's on my to-do list to hijack a 3-d printer and get it to print some of these.

Myth of a Strong Leader: Archie Brown

This was on Bill Gates winter reading-list and so far I'm loving it. It's sort of a history of politics in the 20th Century analysing the party leaders. Strangely haven't found any references to former psuedo-reality-tv-show-hosts.

Fiction

Johnny Mnemonic: William Gibson

Reading this made me appreciate the 80s Keanu Reeves film, given that it was only a short story.

The Dispossessed : Ursula LeGuin

A thinly veiled and slightly exaggerated metaphor about the differences between Capitalist and Communist societies.

Left Hand of Darkness : Ursula LeGuin

Another sci-fi classic that's set on a distant world in which the human inhabitants periodically change sex and spend most of the time as an a-sexual hemaphrodite. Not only that, but they make first contact with a more technologically advanced portion of humanity.

Zones of Thought: Vernon Vince

A Fire Upon the Deep
A Deepness in the Sky
Children of the Sky

This trilogy sort of snuck up on me. The first two are very reminiscent of the Culture novels by Iain M Banks (which I also loved) and thoroughly explore some very interesting ideas such as a feudal society of dog-like creatures that can only think coherently (and collectively) in small packs, and get screwed up when two packs get too physically close together. The second book I actually preferred, focusing on a race of meter-tall spiders in a just-post-industrial-revolution-society undergoing a world-scale conflict. Not only that, but they collectively hibernate several hundred years at a time and are set for first contact with a more technologically advanced humanity. I feel a strange affinity to spiders after reading this book. The third ties up the story nicely but is otherwise miss-able in my humble opinion.

Howls Moving Castle: Diana Wynne Jones

A cute kids book I read on Sarah's recommendation. Certainly richer than the (excellent) film.

Call of Cuthulu: H P Lovecraft

The Colour of Space: H P Lovecraft

Solaris: Stanislav Lem

A superb sci-fi great that discusses metaphysics. A lot.

The Dark Tower series: Stephen King

The Gunslinger
The Drawing of Three
The Wastelands
Wizard and Glass

What to say about this series? Clearly I can't stop reading them, but in the same way that it's difficult to stop eating junk food full of Mono-sodium-Glutemate. The writing is not superb and you have to be quite forgiving about the plot in places but I can see this making a pretty good HBO show later this year.

2016 in Books (part 1)

2016-06-14T00:00:00+01:00

Introduction

Who doesn't like books?

Below is a brief review of some books I've read in the first half of 2016.

This would probably have come out sooner but apparently I like to write book reviews during the 15-30 second crunch-time gaps between Counter-Strike (Source) rounds.

Non Fiction

No Logo: Naomi Klein

Is it a manifesto and primer for the culture jammer movement? Is it a catalogue of heart-breaking corporate mis-deeds? Yes! But what's more it's a thoroughly researched (both primary research and review) of corporate behaviour across the 80s and the 90s that leaves you feeling uncomfortable and ready to boycott something. Fascinating and well written.

Structured Interpretation of Computer Programs: Abelman, Sussman and Sussman

Ever wanted to learn computer science from scratch exclusively with a purely functional language (Lisp)? This book is for you! Requiring a lot from the reader in places, very interesting.

Designing BSD Rootkits: Joseph Kong

Ever wanted to know how vulnerable c-code is to buffer-overflow attacks?

Building Microservices: Sam Newman

I think this book should be titled "Building Services" since almost all the advice applies equally to SOA but buzzwords aside, it's a pretty decent guide to the practicalities and traps of building services and SOA instead of a monolithic, n-tier application.

Algorithm Design Manua: Steven Skiena

This was actually a re-re-reading of my favourite CS book.

Sapiens: A Brief History of Humankind: Yuval Harari

Focusing on three revolutions in human history:

cognitive ~ 70,000 years ago;
agricultural ~ 10,000 years ago;
scientific ~ 500 years ago.

A compelling read with many erudite observations and a satisfyingly high density of sources. This book is worth it alone for the insight that humanity's ability to invent abstract concepts (language, religion, money, government etc.) not only allowed it to co-exist in groups of size larger than the largest chimpanzee band, but is to a large extent responsible for civilisation. To me, this is the most interesting book on this list.

The Hackers Guide to Python: Julien Danjou

After writing python for years I decided to shore up my knowledge for a particular project. This book is a cracking guide to some corners of the language that are easily missed (package distribution, scalability and ORMs to mention but a few) from a prominent OpenStack dev including interviews with others from that project.

Fiction

Stranger in a Strange Land: Robert Heineman

Where to begin? A 500 page behemoth full of the sort of dialogue one would expect in a video game side quest. Not to mention the casual sexism through-out (is good-natured sexism a thing?) or the thinly veiled fantasies of the author. Still though, this book has a lot to say about sex, politics and religion (albeit via the proxy of fringe cults) and gave us the concept of "grokking".

Long tracts of this book are sort-of dreadful but to quote a certain Martian: "I grok it is a goodness."

The Man in the High Castle: Philip K Dick

I'll admit I came to this one after exposure to advertising for the TV Show with the same title. I found this to be a somewhat voyeuristic novel in which the author explores a San Francisco that may have been, set in a parallel universe in which the USA has been conquered and colonised by Axis powers that grow increasingly antagonistic towards each other. Most of the execution of this story was pretty meh but then out of nowhere the finale was utterly sublime.

Purity: Jonathan Franzen

With all of the same qualities of his previous work "The Corrections" including an uncanny similarity to the style of Milan Kundera, Purity tells the intimate story of a few believable characters with serious issues around (among other things) trust. This one requires a more willing suspension of disbelief in places than his previous works (that I greatly enjoyed).

Equal Rites, Wyrd Sisters and Small Gods: Terry Pratchett

Terry Pratchett was brilliant. As night-time reads, Discworld novels are perfect: there are always more of them. Not only that, they have a built-in negative feedback mechanism in which after finishing one you're always ready for another, but experience tells you reading more than one in a row will be disappointing and they will all sort of blur into each other.

Count Zero: William Gibson

More focused and compelling than it's predecessor Neuromancer and set in the same universe, this book continues the dark, noir-ey sci fi of William Gibson.

Mona Lisa Overdrive: William Gibson

Concluding the tale of Neuromancer and Count Zero with more of the same pulp-ey sci-fi goodness. Being stuck in an airport lounge for most of a day gave me the great opportunity to read this almost in one sitting.

Wool Omnibus pt. 1 & 2: Hugh Howey

The short story on which these are based is quite beautiful, but the next few parts of this post-apocalyptic-everyone-now-lives-in-a-silo-underground series are decidedly meh.

The Lady Astronauts of Mars: Mary Kowal

A charming, poignant little short story set in a parallel-world in which Mars was colonised shortly after the moon landings.

A sunrise-alarm-clock-web-service using a Raspberry-Pi

2016-01-09T00:00:00+00:00

Introduction

Waking up can be tricky. I'd heard good things about alarm clocks that light up gradually over about 30 minutes mimicking the sunrise from dull red to bright yellow light.

Many retail models are available with huge variation in price and review strength but for one with a respectable lumen count at the time of writing the price was ~ £100. What's worse, they all had garish front displays that you couldn't turn off and a very limited selection of wake-up music (or radio).

£100 seems an lot for something that's not quite right, surely?

Requirements

So, what are we building?

A powerful lamp with a full range of colours and variable output
A web-interface for setting the time, light profile, etc.
A web-server for interacting with the light

Hardware

Host

There are various options for small boards running full linux distributions. I happened to have a slightly spurned but otherwise serviceable Raspberry Pi Model B lying around (it was replaced as a media server some time ago by a beefier B+ model) so I used that, but a cheaper option would be a Pi-zero that somehow will eventually cost $5!

Lights

At first I tried using some LED arrays that were powered directly from the Pi's GPIO pins, but that limited the choice to devices like the (otherwise super cool) LedBorg which wasn't nearly bright enough to wake me up.

This had two consequences:

A more powerful led-strip was going to ~ double the budget.
It would also require power source external to the Pi which would probably mean soldering.

Eventually I found the LPD8806 led-strip for £29 which like the Pi take a 5V supply, got a super cheap usb-wifi dongle for £2.47 and decided to try and power both the Pi and the led-strip from the same power supply (see diagram) using a 5V, 2A DC supply power supply for £5.99 using this converter for £2.26.

Soldering

Of course, the led-strip didn't have any connector, just four bare wires. This necessitated getting my hands dirty (and mildly singed) to create a very simple circuit connecting the rasberry-pi and the led-strip in parallel with the power supply, as shown below.

Be super careful when soldering not to create any short circuits or burn yourself. If you don't know what you're doing here learn about that first!

Software

I've made the project to control the led array, host the webserver, web-app and runtime management available here. If you don't care about software you can skip over to the install section.

Web-server

The web-server is a super simple Flask service which stores the application state (ie. what time and days of the week to go off) as a json document. It also hosts the web-app and directly controls the led-array.

Web-app

The web-app uses React and Bootstrap to show the, uh, minimal (and responsive) web-ui shown below. Clearly web design is very important to me.

Runtime-management

Some minimal runtime-management is needed to make sure the web-server start when the raspberry-pi is powered up and restarts if it's crashed. The Raspbian distribution still uses sys-v-init (since it's based on Debian-Jessie) so we use a sys-v-init script to do exactly that.

Generally sys-v-init is much fiddlier than more modern boot systems like the now deprecated Ubuntu upstart (which was great) or the much maligned systemd that will replace it in later debian distros but whatever, this works.

Installation

Enable SPI

By default the SPI bus on a raspberry-pi is disabled. Since we'll use to control the led-strip, first enable the SPI bus in the raspi-config which can be opened by:

sudo raspi-config

Project

To install all the project software (including dependencies) run the install script passing in the user and remote host at which your raspberry-pi is available on the network. For example, I have a user "pi" and the device has the hostname "raspberrypi" so to install server etc. I would do

> source install.sh pi@raspberrypi

If you now browse to http://raspberrypi on that same network you should now see the web-ui show above.

Disclaimer: in a more formal setting I'd be using Ansible for this sort of thing but somehow it didn't feel right given how janky the rest of the project is ;-)

Result

In case you think I'm making all this up here it is.

Conclusion

I really enjoyed this Christmas project and would highly recommend it to anyone who feels like doing some hardware for a change. The total hardware cost was under £65, but using a pi-zero would reduce that by ~ £20 and now I have a sweet alarm-clock-in-a-jar!

I also believe it would serve as a good starting point for building something bigger with more lights with cooler animations since it hooks up a web-ui to the leds directly.

A JVM client for IPFS

2015-09-27T00:00:00+01:00

Introduction

The inter-planetary file-system (IPFS) is a super exciting open-source project that, among other things, fixes several problems with current data storage protocols combining ideas used in version control systems, and decentralised peer-to-peer protocols like bit-torrent. The primary IPFS implementation efforts are (so far) in Go and JavaScript and the rapid development pace aside, the IPFS HTTP client API is stable enough to be used by projects like Peergos.

However, using IPFS from the JVM would be immensely easier with a JVM client that wraps the IPFS REST API...

scala-ipfs-api

And lo, there is now an IPFS client for the JVM.

Although the client's written in Scala, it's written with Java projects in mind requiring a single jar in your Java project class-path (made possible with the sbt-assembly plugin).

Writing this client has certainly improved my understanding of the project, and using IPFS as a storage engine for Peergos is now a giant step closer.

Remote file-system browser web-app

2015-05-23T00:00:00+01:00

Introduction

Work with Ian on Peergos is progressing well, to the point where our distributed-secure-file-system finally needs a remote file-system browser (like Explorer, Finder etc.) and despite my standing revulsion for all things Javascript, I plunged in.

Requirements

Specifically, we required a web-app file-system browser which could support file-upload & download, deletion, sharing and so on. I immediately wanted to use Bootstrap to make the design responsive (look different on different sized screens) but there was still one problem...

Let's face it: web UIs are tricky. I attribute most of this to the weak-typing and general difficulty to create invariants in Javascript. For example, what do you think this expresson:

["10", "10", "10"].map(parseInt)

evaluates to? Props if you were expecting this:

[10, NaN, 2]

But I digress. For this reason, I decided to try something new: Facebook's React Javascript UI library. At first I was skeptical but after four hours I had a working prototype and after two working days (integrated over a week) the task was feature complete. Essentially within React's model you tend to write more modular, composable JS code... and that's something the world desperately needs.

A web-app file-system browser

The project is open-source and available on Github. To simulate the backend filesystem I created a simple Flask server that supports filesystem operations in a chroot environment with cookie-based session management and password-based authentication (specified as a server command-line parameter) but bear in mind this is little more than a mock for the web-ui, albeit a functional one supporting:

Grid and list views;
File upload and download;
Context menu (right / long click) with rename and delete options;
Navigation buttons (back, up etc.);
Sorting on file attributes (size, name, last-modified time);
Directory creation;
Cookie-based session management and simple password authentication;

A few screen-grabs on a large / small screen are shown below.

Desktop grid-based view

Desktop list-based view

Mobile grid-based view

Mobile list-based view

Conclusion

This web-app is now feature complete and destined to be the Peergos front-end but please feel free to fork it or submit a pull request. Also a disclaimer, although there is password authentication it is still pretty far from being "secure" without proxying it through a local HTTPS server and auditing the Flask server security.

RSS Repo: Yet another RSS aggregator

2015-04-12T00:00:00+01:00

Introduction

I have a confession to make: I still use RSS to navigate the web... even now!

Years after the fabulous RSS aggregator service Google Reader went the way of the HAL 9000 there are still those that feel its absence, myself included. Of course, many alternative services and browser plugins tried to fill the gap but frankly, I've never found one that was quite right. Some modest requirements:

No advertisements of any kind;
A simple way to import / export my data (feed-list) in standard OPML format;
No third party cookies;
No default, ever-present feeds;
A secure connection;
A single-button to fetch updates from all of my feeds;
Render HTML article content in a cross-platform (including mobile) client;
And of course, it should be open-source :-)

That sounds pretty easy, right? Well with that in mind I started building RSS Repo (HTTPS). After a few weekends it is now fairly usable, and so I thought I'd write a post about it and see if anyone else is interested... so feel free to sign-up!

Architecture

RSS Repo is a single-page application written in regular HTML and Javascript, using the Bootstrap CSS-library to render in a mobile-friendly way. On the other side is a pretty simple Flask server backed by SQL.

In this model, the server is fetching / storing the RSS updates and the browser is rendering the article-HTML directly in the single-page application.

Originally I thought this would be a web-view sitting inside a mobile app built with something like Cordova so you could read cached articles without a web-connection but I think actually browser local storage or IndexedDB would be easier of the mobile browser coverage were good enough.

For now the HTTPS certificate is self-signed since I didn't want to shell out for one from a certificate authority. Of course, once the EFF initiative Let's encrypt is available I'll switch to that.

What it looks like

The simple UI is shown in the screen-grabs below; in one table sit the added / imported RSS feeds with a count of the unread articles, and in the other a list of articles (those in bold are currently unread).

And here is what a mobile browser view looks like (although this is actually a small desktop Firefox window).

Conclusion

Well, that's it: the site is live and open to signing-up so please let me know if you have any suggestions. The project isn't yet on Github but the plan is to do so and use parts of the site as tutorials of stuff like user sessions, automated emails for sign up and best practices for password storing... watch this space.

Porting the NaCl cryptography library

2015-04-06T00:00:00+01:00

Introduction

NaCl is one of the most widely used cryptography libraries currently available. Tweet-NaCl distills it into an entirely understandable, auditable and portable 100-tweets worth of C code (no really: it is even on Twitter).

As part of Peergos, Ian and I really needed a set of crypto primitives that could be used in both a Javascript environment (ie. a browser) and in the JVM. Tweet-NaCl was already available in Javascript but we could not find a stable, working version written in pure Java...

Pure Java Tweet-NaCl

Java Tweet-NaCl is a pure Java port of the Tweet-NaCl cryptography library. The actual port was pretty easy except when it briefly turned into a regex-olympics competition to turn as much of the C to Java automatically.

To verify the correctness of our pure-Java port, we wrapped the pure-C Tweet-NaCl library in the Java Native Interface (JNI). This allowed us to directly test the various Tweet-NaCl API calls in the C-source and our Java port.

To verify that our port is compatible with the Javascript port we're using in Peergos, we also tested the entire API running the Javascript version in Nashorn.

So there you go: three implementations of the Tweet-NaCl library running in the same JVM:

The original C running in the JNI;
A Javascript port of the original C running in Nashorn;
Our new pure-Java port running directly.

And best of all, all tests are now passing!

Profiling the Raspberry PI network I/O rate on the JVM

2014-09-06T00:00:00+01:00

Introduction

It is a fact universally ackowledged that the Raspberry PI is a thing of beauty. I recently stumbled across some performance test data I collected earlier this year concerning the network performance of the little things, in particular when using the (Oracle) Java Virtual Machine (JVM) versus native.

Context

At the time we were building a distributed NoSQL database built on the JVM and the Rasbian folks had recently added the Oracle JVM to the Raspian repositories. To demonstrate how a cluster of machines running the database would fare, we needed a cluster. We used AWS to build a cluster in their Ireland availability zone but I thought it would be much more fun to demonstrate a physical cluster and designed a 64-node Raspberry PI cluster demo that we could show off with and would be reasonably portable. I was convincing enough to secure a few Raspberry PIs to test but the company didn't survive long enough to publish anything about either the database or build the cluster.

In theory, the network card on a Raspberry PI model B should be able to deliver a bandwidth of about about 100 Mbits per second. This would certainly handle many database transactions per second (or a bunch of web traffic or make for a decent media server), but sometimes there is a semantic gap between what is advertised and the reality. I'd also heard various horror stories regarding former JVM network-stack implementations. The question I wanted to know was how good the Oracle JVM would be at performing network I/O operations on a Raspberry PI.

Setup

Specifically I wanted to benchmark the network I/O performance between two Raspberry PI units and also between one Raspberry PI and a conventional server over TCP.

Our inventory for the test is as follows:

Two Raspberry PI model B units;
A beefy server with a 1 Gbps network card;
A switch capable of delivering 1 Gbps uncontended between two endpoints.

To measure the native network bandwidth (as seen by an application), I used a free-and-open-source benchmarking tool called iPerf:

For the JVM I used a simple Java I/O test-harness (available on my Github) that I had previously built to profile the network and disk performance of various cloud providers (Amazon, Rackspace etc.).

Results

The transfer rate over a few minutes for each of the four scenarios is shown in the plot.

The key features I see reading this plot are:

From RPI to the server the transfer rate is stable using both the iPerf and my JVM code and pretty close to the advertised maximum of 100 Mbps.
There is a small but non-neglibible overhead using the JVM, compared to native. This was expected but it's also of the order 1% which is pretty good, props to Oracle. I didn't try too hard to optimise the JVM tests but if you see something that can be improved let me know.
The transfer rate is less stable when the RPI is receiving the traffic, particularly on the JVM.
The RPI can received a good deal less than the Network card is specced for. I tried fiddling various buffer sizes to improve the iPerf results but they seemed fairly stable at 64KB and similarly with my JVM code.

Conclusion

None of these results are too surprising but I was satisfied that they would do the job.