Posts on Claudio Ortolina

A great music system

Fri, 08 Mar 2024 09:00:00 +0100

Over the last few months I had the opportunity to spend some time ironing out a music system that would marry analog and digital listening, log everything I play for recommendations and tracking of future releases, and collect all physical releases I own along with accompanying notes and reflections.

Music library

I own music in different formats:

Vinyl records
CDs
High-resolution files (usually in FLAC format)

Why 3 formats?

Vinyl is usually the best format for old records before the 1980s, as it was the dominant medium. For a lot of music, there’s literally no other available physical format. The format had a massive revival in the last few years, and many artists release high-quality, 180g vinyls which sound amazing.
CDs tend to be the target format for music produced in the late 1980s up to the 2010s, before streaming took off. A FLAC digital file is absolutely fine as a replacement for a CD quality-wise, but I’m a sucker for booklets and limited edition goodies. Particularly in the UK, second-hand CDs are sold in pretty much any charity shop, and more often than not you can get very good deals.
Digital files are convenient for listening on the go, and are also sometimes the only option for bands who don’t produce physical releases. Both record labels and websites like Bandcamp give the option to purchase in FLAC format.

What’s crucial here is that for music I really love, I do wanna own it for a few reasons:

For new releases, I want to compensate the artist or band, and the best way is to buy their music (and attend their gigs).
Streaming platforms can pull content anytime, and in many cases they don’t have all releases I want to listen to.
Once I have albums in my library, I can organise them how I prefer (both physically and digitally).

A priceless extra is that my 2-year old daughter loves CDs because they look like little books but they make music, and vinyls because they’re big and rotate.

Hardware

There are a few moving parts:

A Synology NAS storing all digital files.
A Ruark R5 music system (which can play CDs directly, and that has an integrated amplifier for the turntable). It’s been recently discontinued, which does make a bit uneasy, but the manufacturer still supports it and I reckon I can go on for a few years before having to evaluate a replacement.
A Pro-Ject E1 turntable, playing through the R5. Pricy, but it really just works.
A Raspberry Pi 4 with a HifiBerry DAC2 Pro hat, which allows the PI to play audio with great quality via the R5.

None of this is cheap and there’s certainly more affordable options particularly if you’re able to source a different speakers/amp/turntable combo. The big advantage of using a Raspberry Pi (or an equivalent off the shelf solution) is that you separate “brain” from “muscle” (and the Pi + DAC hat are relatively cost-effective).

Playing music

The whole system revolves around Plex: the NAS runs Plex Media Server, while the Raspberry Pi runs PlexAmp headless. My laptop and phone run PlexAmp as well.

Setting up PlexAmp headless on the Pi required minimal work thanks to the excellent bash-plexamp-installer project, which makes it a breeze to both install and update the software and correctly configure the DAC2 Pro hat. Couple of reboots and everything worked like a charm (and still does).

With this setup, I can use any device to access my music collection, and play it through the Pi if I’m at home, or on-device if I’m out.

The Plex Media Server instance is configured to scrobble to Last.fm, so everything gets logged automatically no matter where I play it from.

When I play physical releases, I use the excellent OpenScrobbler.com web application to search for the album I’m playing, and scrobble it.

I’m now trying out Tidal as a way to listen before buying, as it provides hi-fi quality, pays artists better than other platforms, and it’s deeply integrated into Plex (which means I don’t need to use a separate app, and can rely just on PlexAmp).

If the Tidal experiment is a success, I’ll stop my subscription to Apple Music.

When music is played by the Pi, the R5 system has no information on what’s currently playing. I can read that on my phone, but for other people in the house I programmed a small automation that can be run with Siri, so that anyone can ask “What’s playing?” and get a good answer. This took some lightweight reverse engineering of the PlexAmp web application, and I’m not expecting it to be rock solid as it’s based on private APIs.

In short, the PlexAmp web application polls an endpoint that returns a fairly comprehensive playing status with metadata information about the artist being played, both for local music in the Plex Server and on Tidal. As PlexAmp is written in Node.js, I wrote another small Node.js application to poll the same endpoint, and parse the information I need. I then created an iOS shortcut to hit my Node.js application, and read out the response in a human readable format. The shortcut is automatically available on all devices, and can be run via Siri by its name.

Discovery

Algorithmic recommendations are now very precise - no matter the platform. I’ve come to find them too precise in the sense that they don’t stray off the beaten path.

I prefer to monitor different sources, particularly when it comes to my core preference, progressive rock:

Prog Magazine, which is an old-school monthly zine (both printed and digital) that covers pretty much anything happening in the progressive scene. I have a digital subscription, and read it on the iPad where I can take notes and add albums to a queue.
The reviews RSS feed for progarchives.com, which is a fairly old progressive rock community website where people publish reviews of whatever album they can think of. The RSS format is great because I can read the review, see the cover art (which does a lot for me), and from that decide if I’m interested to explore more.
The occasional visit to a few subreddits, just to get a feel of the community is looking at.
Newsletters from specific record labels that tend to publish music I like (e.g. Kscope) or Karisma Records)
The Album Years podcast with Steven Wilson and Tim Bowness, which provides a very opinionated list of significant music releases roughly from the 70s till the 90s. Only downside is that it’s very UK-centric, and I wish there were similar lists for other parts of the world.

Once I’ve listened to something, and the artist is scrobbled to Last.fm, I’m able to monitor new releases via the excellent MusicHarbor iOS app. The app imports the list of artists I have on Last.fm, and keeps track of all releases by these artists. The import process is easily done manually via a few button presses in the app, and I do that once a month.

Each Friday (release day!) I get a new batch of albums to check out, and I add whatever picks my attention to my queue.

Physical collection management and notes

Managing a physical collection is a relatively solved problem: with Discogs, one can simply scan or search for the correct release, and add it to their own account.

While this is great to track value and conditions of the item, it’s not geared towards managing a collection with associated notes about the artist(s), or lyrics. It’s also not possible to arbitrarily draw connections between albums.

I’m currently trying out Obsidian with the MediaDB plugin. This lets me quickly add every album via the MusicBrainz search API (without matching to an exact release, i.e. you can’t pick the correct year/label/country/limited edition).

Each album becomes its own file with a set of metadata information I can query through a couple of plugins. On top of that, I can write notes and associate albums with simple internal links.

As the data is stored as markdown files with a YAML front matter, there’s no risk of lock-in, and if I ever need to process the data for further analysis or visualization, I can write my own program that does that.

Obsidian is available on all platforms I use, and syncs both data and settings without having to do anything special.

Too much?

I often ask myself if this system is too complicated, but I personally find that the experience of using it is simple at the expense of a reasonable amount of hidden complexity. I’m also aware that my requirements are many, and that’s because music is pretty much my only significant hobby. It still gives me the same joy I felt as a teenager, and still manages to surprise me even when I think I heard it all.

Arriving Somewhere

Fri, 23 Jun 2023 17:34:25 +0300

I’m almost 40 - and while my life has somehow the shape that I envisioned as I grew up, there are definitely big differences. Think about the same picture, but in different colors.

There’s music that accompanied me for the last 20 years, pretty much since I became an adult. It’s been long enough that I don’t exactly remember when it entered my life, and in some ways it feels like it’s always been there.

Of all that music, thousands of songs, there’s a handful that made a deep mark that I still carry with me. More than anything else, one song stands above any other: Arriving Somewhere, But Not Here by Porcupine Tree.

I turned 39 a few days ago and my wife got me as a gift a very nice edition of Deadwing, the album the song is included in. It’s a 2018 remaster, which I’m listening to on a high-quality system.

It’s a 12 minute song, with long instrumental sections, and I know every single little twist - yet this new version has some little extra details, some of the ambient parts are crisper, some of the guitar sounds pop a little bit more. I love it.

It also dawned on me that it is a song about how life turns out different than you think - with fairly tragic images - but with a growing sense of peace as notes go by.

There is relief in the understanding that some pieces of someone’s life are set - that some of the choices made are points of no return. That some dreams are over, and that’s ok.

I do have the feeling of having arrived somewhere, but not here. It is different, but it’s equally beautiful. And I’m finally ok with that.

And through the twists and turns of the last 20 years, I have a song that feels like an old friend that grew up with me, or even that was a little bit ahead, and I finally caught up.

To the next 20 years - hopefully I’ll get somewhere.

An iPad retrospective

Thu, 11 Aug 2022 06:34:25 +0300

I’ve been working on iPad full-time for a while now, and it’s always a source of surprise when I talk about it with other people. This post aims at summarising the why behind this preference of mine.

Planting a seed

I spent 2019 on the Turkish West Coast, in a small town near Izmir. It’s a lovely place where everyone knows each other, and where we now have friends and people we care about.

I was working as a backend engineer on PSPDFKit Server, a product written in Elixir and packaged via Docker, on my trusty old 2014 11” Macbook Air. A wonderful machine - powerful for its size, and featherweight.

Around April, a few things happened at the same time:

Spring came: warm, sunny, and ideal to start working outside in the shade of a café.
A set of requirements that involved way more work with Docker, and consequently a much larger use of CPU, battery, and internet bandwidth.
The need to finally increase my editor font size to 14pt.

It quickly became apparent that I was asking too much out of my laptop.

As I’ve been writing software in the shell for years, I switched to a remote setup using a powerful remote server. 32 cores, very large bandwidth, almost the same experience as working locally (if you take into account the occasional latency problem, well mitigated by mosh).

Screen size, however, became an issue: 14pt on a 11” screen is crammed, and difficult to use.

I thought about purchasing a new laptop, but doing that from Turkey proved to be difficult, because of the different keyboard layout.

I decided I would try a week long experiment using the other device I had, a 3rd generation 12.9 iPad Pro (with its keyboard) which I mainly used for entertainment. I figured that given my remote development setup, I could still get things done.

The experience was great: I loved the screen size and ratio, and the retina screen made text very easy to read (to the point that I could go back to 12pt, which worked really well in terms of information density). I could work the entire day without worrying about the battery, and enjoy the spring/summer season.

While coding was (surprisingly) pleasant considering the fiddly setup, other activities were proving to be more difficult: apps like Slack and Google Docs were (and in some cases still are) cumbersome to use, with very little support for keyboard shortcuts, and of substantial inferior quality compared to their desktop counterparts. I got by, occasionally dusting off the Macbook Air for specific tasks that wouldn’t work at all on the iPad (e.g. work that required using the Chrome Web Inspector).

Then February 2020 came, we moved back to England and the pandemic started. Like many others, we spent most of our time at home, where I could use a decent Mac Mini with a nice keyboard and external screen.

Despite this change, a seed had been planted in my head: could an iPad eventually replace a desktop computer altogether for a “hardcore” user like me?

Unexpected changes

Around spring 2020, my role changed, and I started managing a team. Coding became a past-time activity that I kept maintaining for fun, and 99% of my work now involved planning, communication with people, and some form of documents and task lists.

In December 2020, a family situation forced us to quickly pack our bags and fly to Turkey again. I ended up taking with me all devices (which sounds ridiculous - I know).

The following months are a blur - I remember working from all sorts of places: the car, parking lots, hospital cafés, the balcony table, hotel rooms. I once had a meeting in a service station.

That’s how the iPad came back in full force: always on, always charged, with data always in sync. Pair it with the Apple Pencil, and a pair of earphones, and it gave me the ability to adapt work to a complicated personal life.

I was happy to see that software had gotten better: Safari gained a desktop mode which supported all sorts of complex web apps, and all other apps improved to the point of being at least usable (Google Docs, I’m looking at you). Notable mentions go to Agenda, Mindnode, and Fantastical, which set a very high bar for quality software in general.

Later in the year, things started to calm down and even if I could use the Mac Mini, I would prefer the iPad. I invested in a desk stand which would let me prop it to a decent height, connect it to a good keyboard and mouse, and happily went about my work day.

Introspection

I started asking myself what is it about working on the iPad that I love. What started as a challenge motivated by specific constraints evolved now into a solid preference.

I don’t think I’m making a point - as in trying to be clever or different by not using a traditional computer. There’s plenty of people who do that, and if you search on YouTube you will find all sorts of tutorials on how to try and work around iPadOS’s limitations (e.g. having a Raspberry PI as a sidecar mini-server) to achieve things that are indeed way easier with a laptop.

The iPad fits my mental model of getting things done (tasks instead of files), and it doesn’t get in the way. It’s got great security defaults, it’s encrypted, and automatically backed up. If I buy a new one, I can easily transfer all my data and be up and running in a couple of hours.

The iPad lets me focus on what I want to achieve, as opposed to losing myself in endless maintenance.

I take notes of points of friction, and every once in a while, I review my workflows: if I wanna change anything, I’ll test a different application (after making sure my data is portable), and move on.

Teaching Elixir

Tue, 06 Jul 2021 08:34:25 +0300

When I joined PSPDFKit in 2018, I inherited the development and maintenance of PSPDFKit Server, the server-side component of PSPDFKit for Web.

Initially, I was the only person working on the project full-time. The team eventually expanded to a total of three people and I moved on to manage the Web Team.

In addition, other developers occasionally contributed features with differing degrees of complexity (ranging from small, incremental changes to large features over a few weeks of work).

Looking at the entire process, I can safely say that onboarding has been smooth and fast, especially for internal hires from different teams.

There are, of course, lessons learnt from this process, which is exactly what this blog post is about.

This post was sponsored by digital product consultancy DockYard to support the Elixir community and to encourage its members to share their stories.

The organization

PSPDFKit is a distributed company where everyone works remotely. The PSPDFKit Server team includes people with overlapping time zones and no fixed working hours.

The core business for the company is the sale of SDKs used to manipulate documents (predominantly in PDF format).

SDKs support a variety of platforms (iOS, Android, Windows, and web) and are embedded in client applications written and maintained by our customers.

The Project

PSPDFKit Server’s codebase is more than five years old and has grown organically over time thanks to the work of people with different expertise and backgrounds.

Domain-wise, the application manages the lifetime of a document, allowing our customers to build specific workflows on top of a small set of primitives: documents, content layers, annotations of different types, and permission sets. At a very high level, a user can open a document, perform an arbitrary set of actions (with the option of leveraging real-time collaboration with other users), and finish their session with changes automatically and efficiently stored.

Due to the way the product is sold and operated, it has a few key properties that make it different from most projects I’ve ever worked on:

It’s both CPU and memory intensive due to the amount of operations performed on PDF files. The component that interacts with PDF files is run as a binary daemon and maintained by a different team.
The application has optional components that are activated depending on specific configuration options. For example, one can enable an entire caching layer based on Redis.
Features can be turned on and off remotely via our licensing infrastructure.
The release is shipped as a Docker image to be operated on premise by the customer. This property has two implications. First, it means that we don’t have direct access to performance monitoring or visibility over runtime issues. Second, it drives us to minimise the operational complexity of the end product; for example, we don’t leverage the native Erlang/Elixir distribution because that would complicate deployment for our customers.

Onboarding results

In the last two years, we managed to onboard a good number of people:

1 external hire with extensive Elixir experience, now part of the team full time
1 internal hire with no Elixir experience, now part of the team full time
2 internal hires with no Elixir experience, in the process of joining the team full time
1 person contributing a significant piece of work over a few weeks
Occasional contributions by other people

It’s great that we can facilitate different degrees of contribution, with a ramp-up time measure between a couple of days and a week.

The people involved have completely different backgrounds: heavy frontend development, systems programming, and mobile development. Furthermore, apart from one person, everyone had very little functional programming experience.

Training structure

Learning styles

An important aspect to consider is each person’s learning style. Some people prefer pair-programming, while others would rather go through tutorials in their own time and ask questions when needed.

To accommodate such differences, we combine learning materials with targeted pairing sessions as well as follow-up calls to clarify where needed. We also keep the schedule pretty flexible, meaning that we don’t have a fixed curriculum people go through in a set period of time.

Tooling

The company uses a diverse set of languages and associated development tools. This ranges from text editors to full-fledged IDEs, as well as running on MacOS, Windows, and Linux.

Apart from work on platforms where there’s strong incentive to standardize on a specific tool (e.g. XCode for iOS development), people pick their tools based on personal preference.

A quick informal survey of the Server Team reveals that people predominantly use MacOS and favor Vim, Emacs, VSCode, and JetBrains IntelliJ.

We enforce code formatting at the CI level, but encourage people to set up their development environment to format on file save.

For code intelligence, Elixir has a solid implementation of a Language Server that provides intelligent autocompletion, inline docs, and jump to definition. Especially for people coming from a Java background, this type of tooling helps provide a consistent experience.

In terms of domain-specific tooling, we glue everything together with shell scripts and Docker-based workflows.

Productivity ramp-up time

People are usually productive in two to three days, where “being productive” means being able to set up the project, getting a basic understanding of its structure and architecture, taking on a small piece of work, and producing a relevant pull request containing implementation, tests, and documentation.

You may have heard the mantra “Make a commit on your first day.” While I agree with the sentiment behind it, I find it puts too much pressure on the contributor.

As part of the initial onboarding, it’s normal to find little issues with setup scripts (e.g. things that stop working between major OS versions) or unclear documentation. In most cases, fixes are cheap to implement and offer an opportunity to discuss the application architecture.

Proficiency

On average, it takes two to three months for an engineer to become a proficient contributor who has worked on most areas of the codebase, including some of the internal parts which are subject to very low churn over time.

Learning materials

We generally recommend going through the Elixir Lang guides: they’re well written, accurate, and split in easy-to-digest chunks.

For all libraries, making package documentation available on hexdocs tends to be sufficient.

While we do have a few books available in the company digital library, they’re rarely consulted.

Scope

We start with small, incremental features, as they have a very high completion success rate. For internal hires, we try to choose something connected to a use case the person is already familiar with from previous work.

A more experienced team member is available for general guidance, so that the resulting piece of work includes all necessary components: implementation, tests, and documentation.

On a technical level, we try to avoid pieces of work that introduce more than one key concept. For example, if the person is not familiar with immutable data structures, we avoid assigning a feature that would also introduce concurrency patterns.

Example projects

There are cases where it’s worth starting from smaller, separate projects. For concurrency and distribution, it’s better to rely on smaller applications (e.g. the examples referenced in the Elixir Lang guides) to nail the basics.

Once the basics are clear, we build up by looking at codebase components to address topics like error handling, logging, production hardening, and recovery.

Learning is complemented by specific articles (e.g. an introduction to circuit breakers).

Providing feedback

When a person starts to submit code for review, we try to provide different layers of feedback:

At first, we focus only on functionality, so that things work as expected. At this stage, the purpose is to build confidence and get the person comfortable.
We address code quality and language conventions in follow-up PRs, so that changes can be reviewed separately and it’s clearer why they’re necessary.
Reviews include links for self-learning, so that the person can explore on their own.
If a PR review becomes too long, we suggest a pairing session to discuss things directly.
We try to use proper names for all concepts, but also be conservative about introducing too many ideas in the same feedback session.
We make a clear distinction between company and community conventions, as they may differ. This way, when the developer reads other code on their own, they can compare and contrast approaches. This also often provides constructive criticism on the way we do things.
We slow down as needed during pairing sessions focused on learning. We allow ourselves to diverge from the initial session goal if the person wants to drill into a different topic. Maintaining interest and enthusiasm is more important in the early stages than sticking to a predefined plan.
We recently started recording short videos that document how to approach specific development tasks in order to facilitate independent learning.

Milestones

Every person that learned Elixir in the company followed a different process, but it’s possible to identify common milestones.

Pattern matching and control flow

Understanding pattern-matching comes in the first hour of reading existing code. It takes more time to develop a sensibility around specifics, such as when to use a single function head with an internal case statement versus multiple function heads with specific pattern matching clauses.

Immutable data and functions

For people who approach Elixir as their first functional language coming from object-oriented languages, the first adaptation step is to start reasoning in terms of data passed through functions.

Our trainees quickly embraced immutability due perceived benefits in clarity and predictability of the program. For example, one of our engineers commented that at the beginning of his learning journey, he could sprinkle print statements all over the codebase to see values changing and visualize the application flow.

In this phase, developers tended to write very explicit code, with plenty of intermediate bindings and little use of more advanced language features like the |> operator or the with special form.

Configuration

Configuration usually triggers a few questions:

Should this value be configurable?
What about changing the value at runtime?
How does configuration interact with building a Docker image and, later on, the container runtime?

This is usually a good opportunity to introduce compile-time vs. runtime configuration, along with the operational implications of relying on the Application.* functions to get and set configuration values at runtime.

Common issues include hitting some light race conditions, namely unexpected test results due to configuration values being changed non-deterministically. It’s a great opportunity to reinforce that we should strive to write deterministic code that reads configuration as early as possible and passes relevant values down the line.

Concurrency and OTP

The application has a few concurrent components, but their structure is rarely modified and developers can use public APIs without much concern for the internals.

This is a big advantage and it lets us plan when to look at the concurrency model. We normally suggest looking at the examples in the Elixir Lang guides, then scheduling a couple of pairing sessions to look at the most significant components included in our codebase.

Once we’re done with the fundamentals, we approach concurrency patterns: pools, worker queues, tasks, etc. For such cases, we train only on our application codebase.

State management

The application depends on a few stateful components: metadata store, assets store, and different layers of caching (both filesystem and memory).

These components provide the opportunity to introduce ETS tables, process lifetimes and recovery patterns, OTP guarantees around process trees, and error-handling logic.

Metaprogramming

We found very little use for meta-programming, so we don’t make it an explicit part of the onboarding process. Most of the time, people look into it by themselves when they start exploring the internals of dependencies like Phoenix and Ecto.

Bumps on the road

What follows is a list of road bumps we encountered over time. Some of them are more tied to our application and the way we ship it, so they don’t necessarily represent issues with the language or the platform.

Dialyzer

Especially for people with knowledge of programming languages with static types, having Dialyzer as a separate step from the compiler can be frustrating.

Fortunately, the aforementioned Language Server and related projects make it possible to set up the editing environment to provide almost real-time type information with a reasonable level of accuracy.

Structuring large components

One common observation is that compared to other languages, Elixir applications are built on very few architectural patterns. This applies to both ours and others I’ve worked on in the past few years, and is especially noticeable outside concurrency-related areas (which very often implement explicit behaviours).

Most of the time, we simply take care of separating modules that perform side effects (e.g. writing/reading files, http calls, database interaction) from modules which only manipulate data.

Modules are grouped according to their main “topic,” which helps exploration.

Having fewer patterns sometimes makes the conversation slightly more difficult, as you need to develop a shared, domain-specific vocabulary to quickly express more complex ideas. On the plus side, the effort of defining names helps onboarding immensely, because it forces everyone in the team to be precise and thorough in their explanations.

Error handling and recovery

As we don’t operate the software we write, we need to take extra care with error handling. As best we can, we want to provide error messages that are simple(r) to understand for our customers (who are not Elixir developers).

This creates a very mild tension between our style of error handling and the literature and community examples around applications being run directly by the developers who write them.

For example, it’s much better for us to produce a single line error log with a specific name we understand compared to a more verbose stack trace, because it facilitates communication with the customer.

Conclusion

I often hear the argument, “it’s difficult to hire Elixir developers.” In my experience, once you have two reasonably knowledgeable engineers, you can stop searching for already-trained developers and expand the search to developers willing to learn the language. With some care around the onboarding process, it’s possible to successfully train a new developer every two to three months with a very reasonable impact on the rest of the team.

Thanks to Susan Watkins for editing this article

The Perils of Large Files

Mon, 05 Apr 2021 11:00:00 +0000

I recently wrote a blog post about working with large files in Elixir and OTP at PSPDFKit's blog.

The post originated from a talk I gave last year for ElixirConf.eu, but it contains more precise analysis.

The Race for Space

Sat, 28 Nov 2020 17:55:37 +0000

Public Service Broadcasting (PBS for short) are a trio based in London, whose music usually involves historical audio documents with a mix of jazz, rock and electronic.

Each one of their album revolves around a specific theme, usually inspired by modern/contemporary history, and includes a vast amount of archival audio footage, recording, and samples, with very little vocals on top.

In 2015 they released The Race for Space, a concept album that explores the first 15 years of the Space Race missions, from Sputnik 1's flight in 1957 to the last Apollo mission in 1972.

The general tone celebrates achievements of both US and Soviet Union, emphasising each step as a human endeavour that transcends borders and politics.

Between music, vocals and engineering, the album involved another 35 people, which is something you can only perceive after listening to it a few times: some of the instruments are particularly subtle and deserve special attention.

Each song has a different personality and it's difficult to attribute a general genre to the entire album, but I can feel a general underlying cohesion which makes it that I rarely listen to one song and prefer to just play the entire record.

1. The Race for Space

The album opens with rearranged fragments of US President John F. Kennedy's speech given at Rice University, Houston, Texas, on September 12, 1962. The speech sets the tone of the entire album, celebrating mankind's efforts to reach beyond the stars:

Many years ago, Great British explorer George Mallory
Who was to die on Mount Everest
Was asked 'why did he want to climb it?'
He said 'because it is there'
Well space is there and we're going to climb it
And the moon and the planets are there
And new hopes for knowledge and peace are there
And therefore as we set sail we ask God's blessing
On the most hazardous, and dangerous, and greatest adventure
On which man has ever embarked – John F. Kennedy

It's difficult to listen to this speech without anticipation, especially knowing what would happen 7 years later.

2. Sputnik

The second track, Sputnik, takes us to the Soviet Union in 1957, to tell the story of the first object ever sent into space, the low Earth orbit satellite Sputnik 1.

In Sputnik, the original sounds of the unmanned spacecraft gradually become part of the main beat and gradually build excitement for the beginning of "man's cosmic existence".

The man made celestial body, for the first time in history
Overcame terrestrial gravity and flew into space
All men of all nations recognise this as a great achievement
In an age where the race to conquer space has become an all-absorbing factor

The era of man's cosmic existence

Once again, the accent is on how people can be united by such an important event:

All over the world, people are tuning in to the 'bleep bleep bleep' of the satellite

A dream cherished by men for many centuries comes true on October the 4th, 1957

3. Gagarin

If Sputnik makes you tap your feet, then Gagarin will make you dance: to tell the story of the first ever cosmonaut, Yuri Gagarin, PBS wrote an explosive song, a mix of funk, brass, marching band, you-name-it tune that captures the euphoria of such an important moment.

The jump into the unknown is not something to be scared of:

The World's first cosmonaut
The first to open the door into the unknown
The first to step over the threshold of our homeland

Gagarin himself appreciated his new vantage point:

Astronaut to Earth: I can see forests, rivers, [?] all around
Everything's so beautiful, it's wonderful, it wonderful…

4. Fire in the Cockpit

After Gagarin, Fire in the Cockpit is a cold shower and a stark reminder of the risks and cost of exploration, telling the tragedy of Apollo 1, where three astronauts lost their lives due to a cabin fire during a test.

The entire song revolves around the sound of static and deep keyboard tones, with a cello entering midway as we listen to the words of NASA's description of the events. It's a sad song, but exhibits the composure and respect owed to people who lost their lives while trying to advance the frontiers of human knowledge.

5. E.V.A.

In E.V.A.¹ we hear the story of Alexei Leonov, Alexei Leonov completed the first spacewalk in 1965, spending 10 minutes outside in open space.

It's interesting how the music almost stops when the astronaut leaves the spacecraft, mimicking the silence of the vacuum of space. The few piano notes we can hear really complement the marvel of Leonov's words:

I'm on the edge of the opening
Of the airlock chamber
I feel excellent
I see clouds and the sea
I am beginning to move away

6. The Other Side

The Other Side takes us one step closer to the moon landing, focusing on the Apollo 8 mission, where for the first time a manned spacecraft completed an orbit of the moon, but from the unusual point of view of ground control.

We hear voices and recordings from the control room, where ground control monitors the spacecraft as it's about to reach the blind side of the moon. The excitement and anxiety are palpable: to complete a lunar orbit, Apollo will temporarily lose signal with earth.

Once again, the music tells the story without words, getting quiet during loss of signal and exploding into a liberating instrumental when Apollo finally replies back to Houston. The event is incredibly significant:

The unmanned lunar spacecraft traversed the moon perhaps over 10, 000 times
But this is the first that a man aboard reported to his compatriots here on Earth

7. Valentina

Valentina is a celebration of Valentina Tereshkova, the first woman to ever go to space n 1963 (and to this date, the only one having ever been in a solo mission).

The song, which features choruses from The Smoke Fairies, is a graceful instrumental without any other vocals. J. Willgoose, Esq. writes on the matter:

One of the biggest problems with the material we use, from the period we address, is that it almost totally devoid of any female voice.
It is often said that history is written by the winners, but it would be equally if not more apt to say that it has overwhelmingly been written by men. Of the footage I obtained of the first woman in space, all of it featured her voice being translated by male voices.
Rather than yet more men - us, in this case - attempting to speak on her behalf, it seemed more appropriate to ask a guest singer to provide a female voice, so we tried a different approach with 'Valentina' and I'm very glad we did.

8. Go!

The story of the Apollo 11 and the first crew to land on the moon in 1969 represents one of the most important moments of human history. Once again, PBS decides to focus on the point of view of the people on the ground, to celebrate the often unseen work of preparation, monitoring, incredible engineering that made the whole thing possible.

Go! is giant checklist, where we hear the flight director Gene Kranz go through all the checks needed to make sure that the descent on the moon will be successful. It's so interesting that the landing itself is just a couple of verses in the middle:

Houston, uh
Tranquility base here, The Eagle has landed

The repetition in the dialogues provides the rhythm of the song, which matches the excitement of the mission with upbeat percussions, synth and keyboard.

Go! is a reminder that we can achieve the impossible if we work together.

9. Tomorrow

The Apollo 17 mission, the last in the Apollo program, represents the end of an era and the last time we landed on the moon.

Tomorrow reflects on its significance: as a species, we managed to leave our own planet, albeit temporarily, and look beyond to a completely unexplored universe.

While it's not possible to separate the space race from the politics that fueled it in the first place, it's also a testament to the effort of thousands of people over decades, to literally take us where no one has ever been before.

As an outro, Tomorrow tempers the excitement of the previous songs and focuses more on choruses and keyboard, painting a picture of anticipation of what's gonna come next.

the name is a shorthand for extravehicular activity, which indicates a spacesuit designed for usage outside of a vehicle.

Building a Custom Page for Phoenix Live Dashboard

Sat, 21 Nov 2020 09:29:01 +0000

One of the most interesting features provided by Phoenix Live Dashboard is the ability to define your own pages, so that you can quickly and reliably extend a Live Dashboard instance with sections that are tailored to your application domain.

While working on Tune, I found a use case suitable for a custom live dashboard page: a debugging view where I can check open sessions and inspect the underlying processes.

On the use case

I would encourage you to read Tune's README to understand the use case in more detail, but I'll quote the relevant architectural section:

Tune assumes multiple browser sessions for the same user, which is why it defines a Tune.Spotify.Session behaviour with Tune.Spotify.Session.HTTP as its main runtime implementation.

Each worker is responsible to proxy interaction with the Spotify API, periodically poll for data changes, and broadcast corresponding events.

When a user opens a browser session, TuneWeb.ExplorerLive either starts or simply reuses a worker named with the same session ID.

Each worker monitors its subscribers, so that it can shutdown when a user closes their last browser window.

This architecture ensures that:

The amount of automatic API calls against the Spotify API for a given user is constant and independent from the number of user sessions for the same user.

Credential renewal happens in the background

The explorer implementation remains entirely focused on UI interaction

In other words:

For each Spotify account connected, there can only be one session (a Tune.Spotify.Session.HTTP process named with the session ID).
For each session, there can be many open clients (i.e. browser windows or TuneWeb.ExplorerLive process).

Requirements

Our dashboard page will include:

A table with session IDs, PIDs and count of open clients
Ability to sort by session ID or clients count
Search by session ID
Support multiple nodes

Gathering the necessary data

To populate the dashboard table, we first need to find a way to get a list of all active sessions, along with their clients count.

The simplest way is to leverage the fact that each Tune.Spotify.Session.HTTP process is started with a name managed via a Registry, with the session ID as a key. Registration is in place to guarantee that there can only be one session process with the same ID on each node.

We can use Registry.select/2 to query the registry and receive back all session IDs and PIDs:

Registry.select(
 Tune.Spotify.SessionRegistry,
 [{{:"$1", :"$2", :_}, [], [{{:"$1", :"$2"}}]}]
)

Which returns:

[{"claudio.ortolina", #PID<0.565.0>}]

In the example above, we use a match specification to capture the registry key (the session ID) and the registered PID.

It's important to understand straight away the constraints associated with this approach:

A Registry is normally split into a variable number of partitions, so this query has to visit all partitions to return its results. While this is not a problem at this stage (the application has very little load), it can become a bottleneck once the number of registered processes grows.
As data is partitioned, it's not possible to apply sort order or limit the results without concatenating them all first, which means that both operations will need to be done by the caller.
Results only apply to the current node, which works well with Phoenix Live Dashboard's general structure, which always operates on one node at a time.

Given the registry query above, we can implement a function that provides the data necessary to populate an unfiltered, unsorted version of the table:

defmodule Tune.Spotify.Supervisor do
 # omitted
 def sessions do
 Tune.Spotify.SessionRegistry
 |> Registry.select([{{:"$1", :"$2", :_}, [], [{{:"$1", :"$2"}}]}])
 |> Enum.map(fn {id, pid} ->
 subscribers_count = Tune.Spotify.Session.HTTP.subscribers_count(id)
 %{id: id, pid: pid, clients_count: subscribers_count}
 end)
 end
end

The resulting data structure is a map with the necessary data:

[%{clients_count: 1, id: "claudio.ortolina", pid: #PID<0.565.0>}]

Dashboard page structure

To build a dashboard page, we need to:

Create a module that implements the use Phoenix.LiveDashboard.PageBuilder behaviour.
Mount that module into the Live Dashboard configuration defined into our application router.

What follows is a minimal implementation that shows the data we need, with the following limitations:

no searching, sorting or limiting capabilities
works only on a single node

defmodule TuneWeb.LiveDashboard.SpotifySessionsPage do
 @moduledoc false
 use Phoenix.LiveDashboard.PageBuilder

 @impl true
 def menu_link(_, _) do
 {:ok, "Spotify Sessions"}
 end

 @impl true
 def render_page(_assigns) do
 table(
 columns: columns(),
 id: :spotify_sessions,
 row_attrs: &row_attrs/1,
 row_fetcher: &fetch_sessions/2,
 rows_name: "sessions",
 title: "Spotify Sessions"
 )
 end

 defp fetch_sessions(_params, _node) do
 # returns [%{clients_count: 1, id: "claudio.ortolina", pid: #PID<0.565.0>}]
 sessions = Tune.Spotify.Supervisor.sessions()

 {sessions, length(sessions)}
 end

 defp columns do
 [
 %{field: :id, header: "Session ID", sortable: :asc},
 %{
 field: :pid,
 header: "Worker PID",
 format: &(&1 |> encode_pid() |> String.replace_prefix("PID", ""))
 },
 %{field: :clients_count, header: "Clients count", sortable: :asc}
 ]
 end

 defp row_attrs(session) do
 [
 {"phx-click", "show_info"},
 {"phx-value-info", encode_pid(session[:pid])},
 {"phx-page-loading", true}
 ]
 end
end

The main ingredients of this implementation are:

The use Phoenix.LiveDashboard.PageBuilder directive, which adopts the behaviour with the same name and imports some convenience functions useful for building pages (e.g. encode_pid/1).
The menu_link/2 callback, which is used to define the name of the page and its label in the top navigation bar.
The render_page/2 callback, which has to return a valid component - in this case via the table/1 function.

The table definition has a few moving parts:

An id (unique among other Live Dashboard pages).
A title, shown in the page.
A rows_name, interpolated in the short text blurb that details the total amount of results.
A columns attribute, which is a list of maps detailing the properties of each column. For each column, the id property has to map to a key in the data we will use to populate the table. The sortable property defines which column can be used for sorting (by clicking on the header chevron). Note that unless you specify a default_sort_by attribute for the entire table, you have to have at least one column with the sortable property defined, otherwise you will get a compile error. The format function takes the raw value for a cell in the column and transforms it to a string. It's useful to provide a string representation of the value that is suitable for an HTML table. In the code above, we copy the format function defined in the Processes Live Dashboard page.
A row_attrs function, which takes the data for each row and has to return a list of tuples representing the Phoenix LiveView attributes to apply to the table row itself. Defining attribute is necessary to enable functionality activated by clicking on the row itself. The implementation in this example lets you inspect the session PID in a modal overlay. Similar to the format function, we leverage encode_pid/1 to format the PID as string compatible with the show_info LiveView event.
A row_fetcher function, which takes the current params (search query, limit, sort key, sort direction) and the current node, and returns the data used to populate the table. The return value has to conform to a tuple shape where the first value is a list of sessions (in the shape of maps with the same keys used for column ids) and the second value is the total number of results (irrespectively of the limit). As we implemented Tune.Spotify.Supervisor.sessions/0 taking care of using the same key names, its return value perfectly fits the expectations of the row_fetcher function.

Mounting the dashboard page

To have the page up and running, we need to modify the live_dashboard/2 function inside the application router:

live_dashboard "/dashboard",
 metrics: TuneWeb.Telemetry,
 metrics_history: {TuneWeb.Telemetry.Storage, :metrics_history, []},
 additional_pages: [
 spotify_sessions: TuneWeb.LiveDashboard.SpotifySessionsPage
 ]

Filters and limits

We can now focus on implementing search, sorting and limits. Conceptually, we need to:

If specified, apply the search filter.
Always apply sort order.
Count the sorted elements, to return the correct total.
Always apply the limit clause to the sorted elements.

All of these operations have to be handled by the implementation of the row_fetcher function.

The params map has the following keys:

:search: the string representing the contents of the search input (or nil when empty).
:sort_by: the id of the column to sort by.
:sort_dir: the sort direction, expressed with the atoms :asc and :desc.
:limit: the integer value representing the amount of max items requested by the user.

The params map is very well thought out, as it has a fixed structure, applied defaults where available and values that play well with functions from the Enum module.

We can extend the fetch_sessions/2 function as follows:

defmodule TuneWeb.LiveDashboard.SpotifySessionsPage do
 # omitted

 defp fetch_sessions(params, _node) do
 sessions =
 Tune.Spotify.Supervisor.sessions()
 |> filter(params)

 {Enum.take(sessions, params[:limit]), length(sessions)}
 end

 defp filter(sessions, params) do
 sessions
 |> Enum.filter(fn session -> session_match?(session, params[:search]) end)
 |> Enum.sort_by(fn session -> session[params[:sort_by]] end, params[:sort_dir])
 end

 defp session_match?(_session, nil), do: true
 defp session_match?(session, search_string), do: String.contains?(session[:id], search_string)
end

As outlined above, we start by filtering by search, using a very simple logic that just checks if the session ID contains the searched string.

After search, we apply the sorting logic: the values of the :sort_by and :sort_dir perfectly fit using Enum.sort_by/3 (a really appreciated API design choice), making the implementation short and sweet.

When defining the returning tuple, we take care of applying the limit and returning the correct total count.

With these changes in place, the generated table behaves as expected:

Supporting multiple nodes

The last piece of the puzzle is making sure that we take into account the currently selected node.

Fortunately, we just need to make a very small change to fetch_sessions/2:

defp fetch_sessions(params, node) do
 sessions =
 node
 |> :rpc.call(Tune.Spotify.Supervisor, :sessions, [])
 |> filter(params)

 {Enum.take(sessions, params[:limit]), length(sessions)}
end

The OTP rpc module conveniently provides a call/4 function that takes a node name, module, function, and arguments, returning the exact same value of the remotely executed function.

Conclusions

To see the final version of TuneWeb.LiveDashboard.SpotifySessionsPage, you can open the file in the repo.

Tips for Finch, Telemetry, and Phoenix Live Dashboard

Tue, 17 Nov 2020 17:27:05 +0000

While working on Tune, I needed to collect performance metrics related to the interaction with the Spotify API.

The Finch HTTP client exposes Telemetry metrics, which made it very easy to display them via Phoenix Live Dashboard.

Starting from the stock TuneWeb.Telemetry file generated by Phoenix (see the official guides for an explanation), I just added two new summary metrics to the metrics/0 function:

 summary("vm.total_run_queue_lengths.io"),

 # HTTP
 summary("finch.request.stop.duration", unit: {:native, :millisecond}, tags: [:path]),
 summary("finch.response.stop.duration", unit: {:native, :millisecond}, tags: [:path])
]

With this change in place (commit), I had all metrics being visualized in the dashboard, grouped by the Spotify API path. I wanted to make some improvements:

The :path tag includes query string parameters, so calls like search?q=marillion and search?q=fish would be aggregated in different groups. Instead, I would want them to be part of the same group, ignoring query string parameters.
Since I setup Sentry to use Finch as a client, I wanted to exclude calls made to Sentry and only have charts reporting metrics about the interaction with Spotify

Aggregating by normalized path

To aggregate metrics by normalized path, we can apply a transformation function to the metric tag values, generate a normalized path tag and use that to aggregate metrics. As shown in the telemetry_metrics docs, the option we need is tag_values:

def metrics do
 [
 # omitted
 summary("finch.request.stop.duration",
 unit: {:native, :millisecond},
 tags: [:normalized_path],
 tag_values: &add_normalized_path/1
 )
 ]
end

defp add_normalized_path(metadata) do
 Map.put(metadata, :normalized_path, URI.parse(metadata.path).path)
end

We can use the built-in URI module to parse normalized path out of the Finch metric metadata and add it to the metadata itself. With that in place, we can update the tags option to reference :normalized_path. With this change, metrics are aggregated on the endpoint only, without any query string. For reference, here's the relevant commit.

Filtering only Spotify calls

To filter for Spotify calls only, we can use the keep option, which specifies a predicate function that can be used to define which metrics should be kept and which ones should be discarded. Discarded metrics will not appear in the dashboard chart.

def metrics do
 [
 # omitted
 summary("finch.response.stop.duration",
 unit: {:native, :millisecond},
 tags: [:normalized_path],
 tag_values: &add_normalized_path/1,
 keep: &keep_spotify/1,
 reporter_options: [
 nav: "HTTP - Spotify"
 ]
 )
 ]
end

defp keep_spotify(meta) do
 meta.host =~ "spotify"
end

As the meta information already includes a host, we can compare it with the spotify string. The =~ operator makes the comparison a bit more resilient, so that we don't have to worry about the exact hostname, rather a hostname related to Spotify. This choice might need to be revised if we ever end up interacting via HTTP with another service with "spotify" in their host name (unlikely, but possible).

For some additional clarity, we can also use the nav reporter option (see Phoenix LiveDashboard documentation for more details) to make sure that the navigation header displays a name that details the additional filter applied to the HTTP metrics. For reference, see the relevant commit.

Conclusion

Both improvements required very small updates. Here's the final result, showing the custom Nav title ("HTTP - Spotify") to hint at the filter used to only show Spotify calls, and aggregation by normalized path (without query string).

All in all, I was pleased to see that it was straightforward to customise the charts I needed. One thing I haven't worked on yet is aggregating metrics by logical path, i.e. by route (GET /artist/:id) instead of individual paths (GET /artist/123), but I have some ideas and will come back on it in a future post.

Using Finch With Sentry

Tue, 10 Nov 2020 08:41:30 +0000

A few weeks ago I added enabled support for Sentry inside Tune, my Spotify browser/client. Even if I'm pretty much the only user (I built it for myself after all), having exception tracking has already proved to be useful - band and song names can really create all sorts of issues.

The official Sentry package works as advertised and by default it communicates using Hackney as a http client. As I've been using Finch in the project, I was pleased to see that Sentry exposed a client configuration option that allowed using your own module, as long as it implemented the Sentry.HTTPClient behaviour.

The advantage in swapping the http client library (on top of uniforming the building blocks of the application) is that Finch has support for Telemetry metrics.

Update #1: Thanks to Wojtek Mach for a more streamlined implementation.

The module I wrote is quite short:

defmodule Sentry.FinchClient do
 @moduledoc """
 Defines a small shim to use `Finch` as a `Sentry.HTTPClient`.
 """

 @behaviour Sentry.HTTPClient

 @impl true
 def child_spec do
 Finch.child_spec(name: Sentry.Finch)
 end

 @impl true
 def post(url, headers, body) do
 request = Finch.build(:post, url, headers, body)

 case Finch.request(request, Sentry.Finch) do
 {:ok, response} ->
 {:ok, response.status, response.headers, response.body}

 error ->
 error
 end
 end
end

The trickiest bit was to get the child_spec/0 callback right while keeping dialyzer happy. The first implementation I wrote was simply {Finch, name: Sentry.Finch}, but that would fail to satisfy the typespec defined for child_spec/0. I then switched to a more verbose version:

 def child_spec do
 opts = [name: Sentry.Finch]

 Supervisor.child_spec(
 %{
 id: __MODULE__,
 start: {Finch, :start_link, [opts]},
 type: :supervisor
 },
 []
 )
 end

This version satisfied dialyzer, but turns out there's a simpler way. After publishing this blog post, Wojtek Mach reached out and submitted a PR to streamline the specification to the version shown in the full example above.

I also updated my production configuration:

config :sentry,
 dsn: {:system, "SENTRY_DSN"},
 environment_name: :prod,
 enable_source_code_context: true,
 root_source_code_path: File.cwd!(),
 client: Sentry.FinchClient,
 included_environments: [:prod]

You can of course expand on this implementation if you need to pass more options to the Finch child specification - I found that for my use case, defaults are fine, so for now I don't need to add any configuration hooks.

To see the change in context, this is the original PR, with the follow-up by Wojtek Mach.

Fish

Fri, 06 Nov 2020 12:28:16 +0000

Prelude

I have very early memories of rock music in my life. Since I was a toddler, my parents (and particularly my dad) got me used to listen to 80s rock music. Bands like Queen, Guns N' Roses, AC/DC, Van Halen, and Bon Jovi are ingrained in my memories as sounds of my childhood.

It's not surprising that my taste in music has developed from there, branching out over the years in metal and progressive rock. Music is a constant companion of my daily life and contributed to the formation of my identity.

My taste changed and adapted: around ten years ago I was dismissing Operation: Mindcrime by Queensrÿche as a mediocre album, but two years ago I picked it up again and it was a revelation. Something similar happened while listening to Rage Against the Machine, Tool or A Perfect Circle, didn't like them years ago, absolutely love them now.

This change got me thinking - why do these bands resonate with me now?

I think the answer lies primarily in the integrity of the message connected to these bands: their perspective, political stance, perspective on life really struck a cord with me in my 30s. They take an explicit position around issues I care about.

Enter Marillion and Fish

Until a year ago, I had a very cursory knowledge of Marillion's body of work. I kney they existed, that they have a substantial discography and that they represent an important page in the history of 80s and 90s prog.

One morning, Spotify's suggestions algorithm decided to play Incommunicado and Sugar Mice in the space of an hour. I found myself stopping what I was doing in both instances - there was something about those songs that really dragged my attention. I loved the music - but the lyrics stopped me in my tracks.

Incommunicado is about ambition, the drive to make it big without taking responsibilities. In the context of the entire album, it's both a metaphor for the difficult life of Torch, the main character whose life is falling to pieces, but also a direct description of the music business.

Sugar Mice is a song about the devastating effects of unemployment and economic recession. The title refers to a popular english sweet made of sugar. The main refrain "We're just sugar mice in the rain" gives a very vivid representation of how fragile life can be.

The words, references, and metaphors included in the lyrics stood out as something completely out of the ordinary - it sounded like poetry.

The more I explored other Marillion's songs, the more I realized that what really resonated with me was Fish, their initial singer and lyricist. Following his body of work after he left Marillion, I kept finding incredible songs.

Love and relationships

Do you remember?
Barefoot on the lawn with shooting stars
Do you remember?
The loving on the floor in Belsize Park
Do you remember?
Dancing in stilettoes in the snow
Do you remember?
You never understood I had to go
By the way, didn't I break your heart?
Please excuse me, I never meant to break your heart
So sorry, I never meant to break your heart
But you broke mine — Kayleigh, 1985

By far the most succesful Marillion song, Kayleigh is often cited as Fish's apology to different women for the failure of their romantic relationships, a collage of vivid images and melancholic moments.

To me, this song sows the seeds of a theme that goes through a few songs written by Fish: work/life balance. Relationships fail when you divert your attention away from them, when you don't put in everyday work to keep them going.

In Zoë 25 it's the aftermath of another relationship that didn't go well:

When you're looking for somebody, you might not even see them,
When they're standing there in front of you, right before your eyes,
If you're looking for somebody you're gonna need some help,
You know you'll never find her when you're looking for yourself. — Zoë 25, 2007

In Garden of Remembrance, it's Alzheimer's disease, something completely outside our control:

He's lost between the here and now
Somewhere that he can't be found
She's still here
Her love a ghost of memory
She'll wait for an eternity
He's still here — Garden of Remembrance, 2020

In cliche, the song lyrics use estabilished cliches to acknowledge that no matter how much hard you try, sometimes the simplest thing you can say is what matters.

That's why I'm trying to say with my deepest sincerity
That's why I'm finding it comes down to the basic simplicities
The best way is with an old cliche
It's simply the best way is with an old cliche
Always the best way is with an old cliche
I'll leave it to the best way, it's an old cliche
I love you. — Cliche, 1990

In Punch and Judy¹ it's the progressive deterioration of a relationship, escalating in murdering fantasies:

Whatever happened to morning smiles,
Whatever happened to wicked wiles, permissive styles,
Whatever happened to twinkling eyes,
Whatever happened to hard fast drives,
Complements on unnatural size

Punch, Punch, Punch And Judy, Punch, Punch, Punch And Judy
Punch, Punch, Punch And Judy.

Propping up a bar, family car,
Sweating out a mortgage as a balding clerk,
Punch And Judy, [Judy]
World war three, suburbanshee,
Just slip her these pills and I'll be free.

No more Judy, Judy. Judy no more! Goodbye Judy! — Punch and Judy, 1984

Living on your own terms

Another recurring topic is the idea of living on your own terms. From Tongues, where Fish lets out the frustration of dealing with lawyers during a very long lawsuit with music publisher EMI:

Your entrenched opinions,
On the border of arrogance,
Dug in against the compromise.
A position indefensible, your actions illogical
You're speaking in tongues

You swear contradictions
Your tedious monologues, wielding authority,
Demanding subservience, demanding
I make your sense.
Demanding speaking in tongues. — Tongues, 1991

The inability to speak the same language becomes a massive blocker. The aforementioned lawsuit with EMI ended up being a fundamental event in Fish's life, pushing him to pursue the ownership of all rights of his solo albums (except the first one, Vigil in a Wilderness of Mirrors).

In Circle Line², it's the awareness of the 9-to-5 grind that is imposed on the majority of us:

9 to 5's the only time I try to kid myself that I'm still alive,
That I'm living out the dream to earn my freedom from this rat race
Where all I do's survive, I live the lie, I serve my time.

The circle line.

Just another day, just another day, just another day,
Just another day, just another day, just another day on the circle line.

The circle line, on the circle line. — Circle Line, 2007

Lost Plot, on losing track of what matters:

I was blinded by light but the vision had died, I'd forgotten
In time just what I was fighting for
I'd forgotten who's side I was on, the difference between
Right and wrong
I was out of my depth, going out of my mind, going down in
A field where no prisoners are taken, no quarter is given
The writing was small, it burned on the wall, I'd sold out
My soul for what it was worth
I'd lost the plot, my number was up, the game was over
Snakes and ladders, a world of snakes and ladders, snakes
And ladders — Lost Plot, 2004

View from the Hill, where we the hill is an endless collection of things that don't matter and just keep us imprisoned.

Artwork for the album of Vigil in the Wilderness of Mirrors, by Mark Wilkinson

You sit and think that everything is coming up roses
But you can't see the weeds that entangle your feet
You can't see the wood for the trees 'cause the forest is burning
And you say it's the smoke in your eyes that's making you cry

They sold you the view from a hill
They told you that the view from the hill would be
Further than you have ever seen before
They sold you a view from a hill
They sold you a view from a hill — View from the Hill, 1990

The system has failed (us)

Fish has never shied away from commenting on politics: from Market Square Heroes (Marillion's first single) to Weltschmerz, the ending track of his latest (and last) album.

Market Square Heroes³ is once again an anthem of an angry generation that suffers the consequences of austerity and recession:

I give peace signs when I wage war in the disco
I'm the warrior in the ultra violet haze
Armed with antisocial insecurity
I plan the path of destiny from this maze

Cause I'm a Market Square hero gathering the storms to troop
Cause I'm a Market Square hero speeding the beat of the street pulse
Are you following me, are you following me?
Well suffer my fallen angels and follow me
I'm the Market Square hero, I'm the Market Square hero
We are Market Square Heroes, to be Market Square Heroes — Market Square Heroes, 1982

Weltschmerz is a summary of all fights worth fighting - from climate change, to poverty, to the general failure of a political system that emphasized polarisation and division:

I am a grey bearded warrior, a poet of no mean acclaim
My words are my weapons that I proffer with disdain
My melancholy aspect is something you can’t disregard
My motives you cannot question nor my strong sense of right and wrong\\

I’ve formed the opinion that things can’t stay as they are
My anger and my fury trapped like a wasp in a jar
It’s never too late to make a brave new start
When the revolution is called I will play my part — Weltschmerz, 2020

Perfume River

Perfume River⁴ deserves a mention on its own: in this song, Fish looks at the consequences of the Vietnam War, whose images are burned in his childhood memory.

Artwork for the album Feast of Consequences by Mark Wilkinson

Fire breathing dragons swarm in sweltering skies, spewing flame on innocents below
Charred and brittle corpses, blackened evidence, I am enraged, I am afraid, I am forlorn
The ashes from wise pages fly from libraries, tumble in the clouds of smoke and flies
To lie as dust in corners of dark palaces, the fetid smell of revolution haunts the air.

Take me away to the Perfume River; carry me down to the perfume river
Set me adrift on a well-stocked open boat
Show me the way to the Perfume River, send me away down the perfume river
Pour that sweet, sweet liquor down my throat; pour it down my throat — Perfume River, 2013

Once again, the images evoked are incredibly strong, full of colour - red flames and rage, smoke, dust and death.

Visuals

A powerful ingredient in Fish's artistic work has always been his collaboration with Mark Wilkinson, who illustrated all Marillion's albums (until Fish left the band) and all of Fish's work. His surrealistic style is unmistakable and perfectly complements the "visual" nature of a lot of Fish's songs.

The title refers to Punch and Judy, the main characters of a puppet show popular in British culture. See the Wikipedia page for more information.

The Circle Line is one of London's Underground lines.

The title is both a reference to a market square in Ailesbury, an english town, and Nietzsche's Parable of the Madman.

⁴

The title refers to a river running through the city of Huế in Vietnam, one of the cities deeply affected by the Vietnam War.

Remote 1-on-1 meetings

Wed, 04 Nov 2020 09:10:38 +0000

In the last two years I've done hundreds of remote 1-on-1 meetings, both as a contributor talking to my manager(s) and as a manager talking to people in my team.

As a manager, I consider 1-on-1 meetings the most important responsibility I have: empowering other people to do their best work can greatly outmeasure any contribution I can give on my own.

What follows is some notes on what seems to work for me. It does not represent by any means a proper research on the matter, so please take it with a grain of salt.

Long term objectives

Regular 1-on-1 meetings can certainly be used to solve immediate, short-term blockers or clarify some questions, but the long term objective is building a working professional relationship where:

Both parties can safely be vulnerable and address deep issues that make day to day work more difficult. This can be anything from anxiety about the world at large, to having to homeschool kids, to difficulties in multitasking different responsibilities. Note that as a manager, you're allowed to show that you're struggling to the other person. We're human after all and that helps building transparent relationships. Needless to say, confidentiality and respect are paramount.
Trust takes time and effort. It may take months before people open up and start addressing important topics: what motivates them, problems they see in the team and in the company, their own aspirations and ultimately work on their professional growth.
Lead by example: if you ask a complex question (e.g. what do you wanna focus on this year?), answer it yourself first, so that they understand better.
Always prepare something to talk about: even if it seems artificial, it will gradually tune your perception to be on the lookout between meetings for topics to talk about and it will get easier. At the same time, if the other party doesn't provide topics over multiple meetings, make sure you address that. There's always something to talk about.

Preparation

Regular 1-on-1 meetings require preparation: the easiest thing to do to ease preparation is to keep an agenda document shared between the two participants.

I can recommend a shared Google Doc, with sections titled by date and sorted in reverse chronological order (most recent first).

This way you can:

Predictably add topics to discuss between meetings (it's always at the top).
Have implicit time tracking, i.e. you know when something was discussed.
Easily collaborate in real time during the meeting to capture ideas in a form that correctly reflects the other person's thoughts.

This system is by no means perfect (especially if you end up having dozens of these documents), but it's flexible enough that you can adapt it to individual preferences. For example, I find that with some people we can use the agenda to discuss topic asynchronously in writing beforehand (and draw some conclusions in the meeting), while with other we defer discussion to the meeting itself. With other ones, we end up adding topics during the meeting as the conversation naturally goes in different directions.

In addition, it's important to prepare for the conversation, even for just 5-10 minutes before the meeting. Having such time buffer not only helps clarifying thoughts beforehand, but also gives you an opportunity to stop thinking about what you were doing before and what you're gonna be doing after the meeting.

During the meeting

Respect the agenda as much as you can, making sure that you either address all points on it or explicitly say "We'll need to address this in a separate call. Can it wait our next planned session or do you wanna schedule an earlier follow-up?".
If the conversation drifts to a completely different direction and time is running tight, make sure both parties are happy to discuss that direction at the expense of other topics. You always have the option to capture the new topic and address it at a later stage.
Be at ease in saying "I don't know, but here's how I plan to find out and here's when I'm gonna report back about it". At the end of the day, trust is built on transparency and predictability.
When the other person explains something to you and you wanna make sure you got it right, repeat it back and ask for confirmation. By doing that, you both check your understanding and implicitly ask the other person to check their own explanation. I've used this technique in all sorts of other conversations in my professional and personal life and it does absolute wonders in clarifying expectations on both sides.
If you take notes, learn how to do that quickly by using keywords, then fill in the blanks later (and ask for confirmation to the other person once done). Writing can break the flow of the conversation, so it needs to be done carefully in order to minimize its impact.
If a conversation topic implies someone taking an action, explicitly state that in the form of "I will" or "You will" or "Someone else will", with an indication of when that would happen.
If the meeting resulted in some actions, recap them at the end of the call.
Video conversations have a different pace - let people speak, listen carefully, slow down, repeat a few times if needed.
Explicitly ask the other person if they think a topic has been exhausted before moving on.
Respectfully ask the other person how they're doing, leaving them space to decide what they can comfortably share with you. Do no pry and always qualify your questions with the reason why you're asking them. For example, I've been recently asking people how they're dealing with the COVID-19 pandemic, because I noticed erratic working patterns that suggest they may be working too much (for many people, work is much simpler to deal with, so they end up using it as a safe haven - I'm not a psychologist though so this is another thing to take with a grain of salt).
Provide context: while this is important in any company, I believe it's fundamental in a remote company because people have more limited opportunities to gather context by casually taking part to unscheduled conversations. So if I'm discussing a specific project that I think it's connected to other projects, I'll share that. More often than not, the person on the other end will appreciate the additional information and will make good use of it.

After the meeting

If you have any action, just do it as early as possible. Your ability to follow up is by far the most important factor in building trust. If the other person asks you to do something, you agree to it and you don't, they will not ask you again.

If at any point you realize you didn't do something you promised to do, acknowledge your shortcoming, apologize and make up for it. It happens, and if you're transparent about it usually the other person will understand.

Topics

1-on-1 meetings are structured around the people involved: while you can definitely start from some guideline questions, they should over time develop into a unique conversation.

That said, over the course of multiple meetings you should aim at:

Unblocking specific issue related to current streams of work, e.g. "I'm undecided on how to build X, if using this or that."
Clarify responsibilities, e.g. "Yes, you need to take care of X, while Alice can take care of Y."
Provide feedback on work done, e.g. "I really liked how you did X because…" or "I'd like to speak about Y, as there's an opportunity to improve Z."
Useful things to learn about, e.g. "As you're working on X, you might enjoy learning about Y."
Connect the dots with other projects, e.g. "As you're working on X, you might be interested to speak to Alice, as she's working on Y, which is related to X as…"
Happiness, satisfaction, and future work e.g. "If we look at the roadmap, there's X, Y, Z. Do they interest you? Which one would be your initial preference to work on?"

One important aspect is balance: too often 1-on-1 meetings are focused on the day to day work and don't cover the larger picture. This is why there should be scheduled Feedback Sessions where you go through some meta-questions that allow expanding scope.

These are some examples of questions useful for those sessions:

Are you happy about the work you're doing? Is it satisfactory?
Looking at X time period, can you point out a piece of your work you're proud of?
Looking at the same time period, can you point out 3 team achievements you're proud of?
What should the team focus on in this quarter?
If you had a magic wand and could instantly change anything in the team, what would that be?

In general, Feedback Sessions are an opportunity to look at the larger picture and think about the future. For more inspiration, you can consult the Small Improvements guide to 1-on-1 meetings.

Conclusions

As mentioned before, this is by no means an exhaustive guide, but a collection on thoughts based on my experience. At the end of the day, if you always focus on listening to the other person and acting swiftly on their feedback, you will get good results.

A Short Profiling Story

Tue, 03 Nov 2020 11:42:42 +0000

While transcribing the talk I gave at the last ElixirConf.eu conference, one of my colleagues pointed out that I glossed over the details of one of the examples. This prompted me to do some digging and I want to share what I found.

The problem

The example in question is a module responsible to fetch a file from a remote source and write it at the specified path.

The implementation is very simplistic and lacks both error handling and retry logic.

 defmodule Perils.Examples.Store do
 def write(file_name, url) do
 with {:ok, data} <- get(url) do
 File.write!(file_name, data)
 end
 end

 defp get(url) do
 :httpc.request(:get, {String.to_charlist(url), []}, [], [])
 |> case do
 {:ok, result} ->
 {{_, 200, _}, _headers, body} = result
 {:ok, body}

 error ->
 error
 end
 end
 end

Looking at the code, we can see that it relies on :httpc, the http client that ships with Erlang/OTP.

Both in my talk and in the initial transcription draft, I pointed out that running this code with a 12MB file would result in a memory usage peak at around 350/375MB, but didn't really look into why.

Such delta between the file size and peak memory usage is suspicious and worth investigating.

The investigation

I started by setting up an exprof test, so that I could profile resource usage associated with the problematic function.

 defmodule A do
 import ExProf.Macro

 def run(url) do
 profile do
 Perils.Examples.Store.write("magazine.pdf", url)
 end
 end
 end

 url = "https://web-examples.pspdfkit.com/magazine/example.pdf" #12MB
 {records, _block_result} = A.run(url)
 total_percent = Enum.reduce(records, 0.0, &(&1.percent + &2))
 IO.inspect("total = #{total_percent}")

The result (with some lines omitted) shows that most of the time (51.74%) is spent converting the binary response body to a list inside the maybe_format_body/2 function:

 FUNCTION CALLS % TIME [uS / CALLS]
 -------- ----- ------- ---- [----------]
 <omitted>
 erlang:iolist_to_binary/1 1 20.46 49705 [ 49705.00]
 erlang:binary_to_list/1 1 27.54 66887 [ 66887.00]
 httpc:maybe_format_body/2 1 51.74 125664 [ 125664.00]

While this is not an indication of higher memory usage per se, it's a good lead: binary to list conversion can be memory intensive.

I then looked at the source for maybe_format_body/2, making sure to match on the OTP version I tested against (23.1.1).

 maybe_format_body(BinBody, Options) ->
 case proplists:get_value(body_format, Options, string) of
 string ->
 binary_to_list(BinBody);
 _ ->
 BinBody
 end.

As expected, the function uses binary_to_list/1 to transform the response binary body into a list. Luckily, this behaviour can be tweaked via the body_format option, which defaults to string (as in Erlang string, which maps to a character list in Elixir).

Searching for body_format in the Erlang docs for request/5 shows that indeed it's possible to tweak our problematic implementation to:

 :httpc.request(:get, {String.to_charlist(url), []}, [], body_format: :binary)

With this change, memory usage decreases dramatically, showing a delta only slightly larger than the file size.

Conclusion

This whole investigation got me thinking, as the body_format option had been in the docs all along, yet I hadn't seen it. I can find three reasons:

The overall logic in the example doesn't really care about the response body contents, as it just writes them to a file. Without seeing that response, there was no way for me to even notice its type.
File.write/2 accepts binaries, strings and character lists - again I didn't have a reason to even wonder about the type used to represent that returned response body.
Working primarily in Elixir, everything tends to be either a string or a binary. I just "forget" that character lists exist, which lead to the implicit assumption that this would be the default for :httpc.

In other words, I didn't know what to search in the docs. Profiling tools helped me understand the problem space and pointed me in the right direction.

Posts on Claudio Ortolina

A great music system

Music library #

Hardware #

Playing music #

Discovery #

Physical collection management and notes #

Too much? #

Arriving Somewhere

An iPad retrospective

Planting a seed #

Unexpected changes #

Introspection #

Teaching Elixir

The organization #

The Project #

Onboarding results #

Training structure #

Learning styles #

Tooling #

Productivity ramp-up time #

Proficiency #

Learning materials #

Scope #

Example projects #

Providing feedback #

Milestones #

Pattern matching and control flow #

Immutable data and functions #

Configuration #

Concurrency and OTP #

State management #

Metaprogramming #

Bumps on the road #

Dialyzer #

Structuring large components #

Error handling and recovery #

Conclusion #

The Perils of Large Files

The Race for Space

1. The Race for Space

2. Sputnik

3. Gagarin

4. Fire in the Cockpit

5. E.V.A.

6. The Other Side

7. Valentina

8. Go!

9. Tomorrow

Building a Custom Page for Phoenix Live Dashboard

On the use case

Requirements

Gathering the necessary data

Dashboard page structure

Mounting the dashboard page

Filters and limits

Supporting multiple nodes

Conclusions

Tips for Finch, Telemetry, and Phoenix Live Dashboard

Aggregating by normalized path

Filtering only Spotify calls

Conclusion

Using Finch With Sentry

Fish

Prelude

Enter Marillion and Fish

Love and relationships

Living on your own terms

The system has failed (us)

Perfume River

Visuals

Remote 1-on-1 meetings

Long term objectives

Preparation

During the meeting

After the meeting

Topics

Conclusions

A Short Profiling Story

The problem

Music library

Hardware

Playing music

Discovery

Physical collection management and notes

Too much?

Planting a seed

Unexpected changes

Introspection

The organization

The Project

Onboarding results

Training structure

Learning styles

Tooling

Productivity ramp-up time

Proficiency

Learning materials

Scope

Example projects

Providing feedback

Milestones

Pattern matching and control flow

Immutable data and functions

Configuration

Concurrency and OTP

State management

Metaprogramming

Bumps on the road

Dialyzer

Structuring large components

Error handling and recovery

Conclusion