Edoardo Tenani

Improving LUKS performances on SSDs

Wed, 14 Feb 2024 14:52:25 +0200

Full disk encryption is a security measure that involves encrypting the entire contents of a computer’s hard drive or storage device. This includes the operating system, system files, application data, and user files.

Especially at work, having such protection in place provides so many benefits it’s a no-brainer wanting to have it enabled everywhere! Full disk encryption ensures that even if someone gains unauthorized access to the physical device or its storage media, they won’t be able to read the stored data without the correct decryption key.

Think of GDPR or HIPAA regulations and their strict privacy requirements and think of hybrid working environments (or full remote) where having access to a device is easier that in other settings, Full Disk Encryption make your device as usuful as a brick.

But is has a major disadvantage: performances.

In case you are using LUKS on a Solid State Drive, there is a nice trick that could be used to improve performances: adding no-read-workqueue and/or no-write-workqueue to /etc/crypttab flag section.

These flags disable forcing each read and write operation going through Kernel queues. Originally on spinning hard drives having ordered reads or writes was a performance optimization but nowdays SSDs are snappier when doing random accesses to the disk, making these optimization actually hurt performances. If you have I/O intensive workloads you can benefit by disabling the queues.

Source for this is the ever awesome Arch Linux Wiki, which provides further links to learn more about where those flags come from.

To make or not to make

Mon, 18 Sep 2023 23:11:00 +0200

I had this post in the making for more than one year 😰 but I got re-inspired by an article from my colleague Giuseppe Santoro about it.

Why writing about make? make is a renowned tool and you can find it in a lot of projects. You can see it used for a wide range of tasks, involved in all kinds of developer tooling gimmicks. One of them is using it as a task runner, something that it is not, nor is ergonomic for.

It’s purpose is to build your program from its source files.

In its post Giuseppe claims that Make is too old for the present day and that we could do better:

As anyone that writes or interacts with software these days, I need to run automation scripts every day but I am still stuck with a tool that (according to Wikipedia) has been written 47 years ago. Don’t get me wrong, Makefile was really useful 20 years ago but I think we can do better than that in 2023.

I agree we could do better, but for slightly different reasons:

software that ages that much exhibit some of these proprieties: it’s useful, it’s reliable, has a community, is actively developed, has no replacement. Those qualities are good qualities and we should aim at more software like that!
it may also be that is affected by “we always do it like that” or “no one has ever been fired for buying IBM” and this is a culture I’m more than happy to abandon.

I have full respect for a software that has been around 4.7x the time I’ve been in the trade 😁 But Make is far from perfect when used as a task runner, something it has not been designed for.

Pitfalls

make is defined in the first page of it’s manual as “[…] the GNU make utility, which determines automatically which pieces of a large program need to be recompiled, and issues the commands to recompile them.”

Make documentation is for sure extensive, but searching it is complex as is referencing to it. This is a huge pain for learning how to properly use it or referencing documentation for someone else. Some documentation formats allow for text search but, personal opinion, that is not enough in 2023. It feels like a lot is given for granted, like if you (the dev) where already using it and the doc is just your reference. You know what you’re looking for, just need to refresh it.

(as a partial solution, Make documentation is searchable through https://devdocs.io 🥳).

Now that we can search the documentation, let’s start learning as I want to change the automation in my repo! Looking at the first “straightforward” (the doc says it!) Makefile I admit I’m confused:

edit : main.o kbd.o command.o display.o \
insert.o search.o files.o utils.o
cc -o edit main.o kbd.o command.o display.o \
insert.o search.o files.o utils.o
main.o : main.c defs.h
cc -c main.c
kbd.o : kbd.c defs.h command.h
cc -c kbd.c
command.o : command.c defs.h command.h
cc -c command.c
display.o : display.c defs.h buffer.h
cc -c display.c
insert.o : insert.c defs.h buffer.h
cc -c insert.c
search.o : search.c defs.h buffer.h
cc -c search.c
files.o : files.c defs.h buffer.h command.h
cc -c files.c
utils.o : utils.c defs.h
cc -c utils.c
clean :
rm edit main.o kbd.o command.o display.o \
insert.o search.o files.o utils.o

I have little knowledge around building C code, but what about automation? I would not define this example as straightforward in the context of a task runner, is not even relevant.

Well let’s move on, I just needed to add a new variable to gather the git SHA commit of my build, so let’s search what the manual says about variables. As the docs suggests:

[…] duplication is error-prone […]. We can eliminate the risk and simplify the makefile by using a variable. Variables allow a text string to be defined once and substituted in multiple places later.

That’s great! It links to How to Use Variables which does not explain how, so we proceed. The first link goes to Basics of Variable References which states:

To substitute a variable’s value, write a dollar sign followed by the name of the variable in parentheses or braces: either ‘$(foo)’ or ‘${foo}’ is a valid reference to the variable foo. This special significance of ‘$’ is why you must write ‘$$’ to have the effect of a single dollar sign in a file name or recipe.

I got it, now I know how to reference variables!

Variable references can be used in any context: targets, prerequisites, recipes, most directives, and new variable values.

Great, flexibility!

A dollar sign followed by a character other than a dollar sign, open-parenthesis or open-brace treats that single character as the variable name. Thus, you could reference the variable x with ‘$x’. However, this practice can lead to confusion (e.g., ‘$foo’ refers to the variable f followed by the string oo) so we recommend using parentheses or braces around all variables, even single-letter variables, unless omitting them gives significant readability improvements.
Emphasis mine

That’s fine, once you know about it? Wrong, this is bad developer experience, we should do better.

Custom variables are important but maybe there are built-in variables that you can use? Sure, Automatic Variables! This is useful and variants are useful too, until they aren’t.
Without looking, what is the difference between $(@F) and $(*F)? Again, this is bad developer experience, we should do better!

Make also supports some other Special Variables but which again are not useful in the context of a task runner.

The last bits that I struggle with Make are:

.PHONY. Ever heard of it? From the docs: “A phony target is one that is not really the name of a file” (which means all other targets are file names). If you’re not using it for all targets that do not create a file you may end up introducing file caching in your automation. Good luck searching for it the first time you encounter it though, as the docs don’t have search built-in.
there is no “help” or target description in make (yes, some workaround exists, but still are workarounds, is not built-in nor documented).
errors can be obscure and searching the web often don’t solves your problem.
breaking a single Makefile into different ones is a pain;
ironing out cross platform difference makes things worse.

I know that you can use other search engines or there are other ways to get these information, but I think a good developer experience requires great documentation.

Note that these are not all Make’s faults, as the issue is in using it way beyond the original scope. Writing great Makefiles is possible, this blog post is an awesome example of it, but that requires high discipline, deep knowledge and (having tried to modify it) trial and errors. It also not friendly for newcomers. makeis an awesome build tool, but when used as a task runner you lose a lot of features and get back a lot of unnecessary complexity.

Funny note while writing this post I discovered that someone implemented FizzBuzz in GNU Make 🤯. Mind-boggling 🤣

Can we do better?

Alternatives are available that do better on the developer experience and are:

easier to use;
kinder to your contributors;
have a smaller learning curve;
simpler internals, in case you need to troubleshoot them;
support advanced use cases (cross-platform specificity, import, overrides, help text).

Of all the available alternatives there is a set that stands out to me: rake, task and just.

Why?

I want this tool to be ergonomic: it should be intuitive (or at worst easy) to write task definition, dependencies and leverage external tools.
I want this tool to be powerful: I need advanced features, like cross-platform overrides or imports. I want this tool to be welcoming: documentation should be good and details how to do things, allowing to iterate fast without deep knowledge of the internals.

Rake

Rake is a make-like build utility for Ruby. It uses a DSL to write terse and concise tasks in Ruby code, automating the toil.

To me this is the best tool out there, its major drawback is relying on Ruby. This makes adding it to any pipeline for other languages a chore and require contributors to know Ruby. Not to mention having to install Ruby in your CI just for this. I would not recommend to use this unless you’re in a Ruby project.

Task

Task is a task runner / simpler Make alternative written in Go. It’s a single binary, which makes it incredibly convenient to use and install and works (in most cases) as you expect.

This is my preferred choice as of today, it’s packed with a lot of features, easy to start with and iterate. Personally I prefer to keep things simple and move logic to external scripts when needed than making it a spaghetti-YAML mess. This is a list from it’s great feature set: allows to use Go template engine, has a cross-platform shell interpreter (works on Windows, for real!), Taskfile import, .env support, variables, dry run mode and a file watcher (and there are more!).

And yes, it uses YAML. I’m not happy about it but given how pervasive it has become I consider it a decent compromise (but I suggest to keep things simple).

Just

Just is a handy way to save and run project-specific commands. It’s a single binary like Task. It’s DSL is as good as Rake, feels like a Makefile with a feature set appropriate for a task runner.

Features include: supports different languages, task parameters, .env support, running recipes in different folders.

I don’t like the fallback to parent feature that feels a bit unpredictable or the need to write recipes in different languages within the same file (I generally put them in separate files and run them) thus is the one that gets my least preferred, but is impressive and has some interesting patterns.

Conclusions

Since some years ago I found my task runner in task, it evolved since I started using it (current Taskfile version is 3) and packs all the feature I need (plus some more). But most of all, I can use it without even looking at the documentation for basic use cases and I can jump to the docs when I want to make things more complex.

In the end no DSL will ever be perfect. I think Make DSL is great for it’s intended purpose (building C code) as much as it’s painful for the task runner use case.
A task runner goal is to run a “task”, a command or a set of commands to achieve something. That’s why alternative for this specific use case exists.

An example is 2ami Taskfile that helped me streamline the tool release pipeline where I build it, compute checksum and create a GitHub release for 3 different Operating Systems.

It doesn’t matter which tool you end up using, but we must ask and use better tooling, not because the one we have are old but because we need tooling to be ergonomic and fit-for-purpose.

We have the right to more ergonomic tooling, like you have the right to an ergonomic chair to prevent long term damage of repeated actions.

Smartphone Ecodesign labelling comes to EU

Sun, 17 Sep 2023 19:18:25 +0200

Ecodesign comes to smartphones!

I suggest to read the excellet, as usual, Right To Repair blog post for further details.

The TL;DR is that from June 2025 those devices will have to factor in repairability, forcing them to provide hardware spare parts and information for 8at least 7 year* after stopping distributing the product (and 5 years for software updates).

This is huge really! I’ve been fixing smartphones for “passion” since at least 10 years ago. I fondly remember the first time I pried open by Samsung Galaxy SII 😱

During the last years most smartphones have come so much repair unfriendly (both on costs, spare parts & manufacturing “complexity” - like using a lot of plastic connections or glue) that repairing them is a real challenge, and is quite difficult to make them last more than 5 years. To me, 5 years is the minimum target life for a smartphone given the abundance of scarce metals and resources needed in its manufacturing process.

So I’m more than happy!

And in case you want the future now, there are two active manufacturing companies that got 10/10 score on iFixit for all their devices:

Fairphone have been doing this for some years alreay for smartphones (and just announced their last flagship device) - iFixit
framework laptops, their started a couple of years ago and it looks awesome (plus they got excellent reviews) - iFixit

Per project LSP settings in LunarVim

Thu, 07 Sep 2023 12:32:15 +0200

I’ve been using LunarVim successfully for years now and I just discover you can set specific LSP settings per project!

I’ve been developing with go and one thing that happens frequently is using build tags. Once a build tag is added, gopls and other tooling will ignore the file where is specified, unless told not to do so.

To edit local configuration you have to run :LspSettings local gopls

This will open a file in a local .nlsp-settings/ folder where you can add you configuration. In my case, adding the integration build tag:

{
 "gopls": {
 "build.buildFlags": [
 "-tags=integration"
 ]
 }
}

Bonus trick

If you add this snippet

{
 "gopls": {
 "ui.diagnostic.analyses": {
 "fieldalignment": true,
 "nilness": true,
 "unusedparams": true,
 "unusedwrite": true
 }
 }
}

to your global gopls LSP settings (open it with :LspSettings gopls) you get autocompletion for settings like in other editors. 🌟

Old blog, new blog

Wed, 06 Sep 2023 01:04:20 +0000

Small changes happened!

Recently Google Domains announced is going to be acquired by Squarespace. So much for a good Google product :/ Squarespace has some really dauting prices, so I started evaluating a strategy.

So far nothing major happened, but I decided meanwhile to move away from GitHub pages for hosting the website, and trying out Netlify.
The main reason are avoid the DNS coupling GitHub pages require and being able to preview the content online and not having to rely on a local editor.
Hopefully I can setup a nice blogging environment with a devcontainer and have PRs acting as a draft system!

Another change is the blog repo no longer being public on GitHub. This repo didn’t have a license, so wasn’t really intended to be public; it was due to a GitHub limit, which allows using Pages on a private repo only on the paid plan.

So long and thanks for all the fish GitHub! The transition have been quite smooth, if not for the fact that changing DNS brought the site down for some days :( Opsfail, sorry.

I also decided to remove any links point to “the site formerly known as Twitter”, as I’m really not comfortable linking to that platform anymore. I’ll keep an account just to prevent squatting the username but I will not advertise it anymore.

Inspecting TLS traffic golang

Tue, 09 May 2023 18:55:24 +0200

While debugging a Go application, I needed to troubleshoot its connections to a remote HTTPS endpoint.

Without fear, I started the mighty Wireshark and prepared to inspect some TCP packets!

The application I needed to debug is a small sample application I wrote, so I was in complete control of the source code. Quite handy, as we will demonstrate. The application is part of my study on the Elastic APM Agents communication protocol. Is a sample Go HTTP server with a couple of endpoints that emit all possible Elastic APM data, using simple instrumentation of HTTP function handlers.

The first step were to determine the IP address and port/s the application was connecting to. After starting the application, in this case named eapm, I used nestat and grep to check which connection were opened.

$ netstat -alpn | grep
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 10.10.10.10:55932 1.2.3.4:443 ESTABLISHED 658594/./eapm
tcp 0 0 10.10.10.10:35122 1.2.3.4:443 ESTABLISHED 658594/./eapm
tcp6 0 0 :::8080 :::* LISTEN 658594/./eapm

IPs have been redacted, but so far so good, we have 2 outgoing connection to the same IP address and a listening server by our small service.

With this information, we can start searching around in Wireshark. As connections go to port 443 I was expecting encrypted traffic, which meant looking at the content would be impossible without a decryption key.

As this is a known problem, there are workarounds! Being in control of the client or the server of a TLS connection, is possible to get the decryption keys needed for inspecting the traffic. In this case we are in control of the client, as is a library in a program we control the source of.

Wireshark allow decrypting traffic, as long as the Pre-Master-Secret is known.

SSLKEYLOGFILE is an environment variable that can be set tell browsers to save SSL decryption keys to a specific path on the file system. With can then pass that file to Wireshark and decrypt any TLS traffic.

So far so good, but Golang does not support it out of the box. But it supports (at least as for Golang 1.20.4) the KeyLogWriter option in the tls.Config struct (docs). It accepts a io.Writer, which is handily returned by os.Create.

To print the keys the only thing left was to find out where in the code to add this code:

fi, _ := os.Create("/tmp/foo")
cfg := tls.Config{KeyLogWriter: fi}

In Elastic APM Go Agent, the HTTP client is configured in transport/http.go. The function creates a tls.Config and returns it after some initialization.

The complete code for this little mischief was:

func newEnvTLSClientConfig() (*tls.Config, error) {
 verifyServerCert, err := configutil.ParseBoolEnv(envVerifyServerCert, true)
 if err != nil {
 return nil, err
 }
 tlsClientConfig := &tls.Config{InsecureSkipVerify: !verifyServerCert}
 // [...] no changes here
 fi, _ := os.Create("/tmp/foo")
 tlsClientConfig.KeyLogWriter = fi
 return tlsClientConfig, nil
}

With this done, the only thing left was to add the /tmp/foo file in Wireshark, under Edit > Preferences > Protocols > TLS > (Pre)-Master-Secret log filename and I were done.

Filtering for destination IP allowed to view all interesting traffic, which was now fully decrypted. Horray 🎉

Better detection of Golang wrapped multiple errors

Mon, 27 Mar 2023 19:32:22 +0200

Golang 1.20 is out and one the most interesting features for me has been multiple errors. I wrote about them in Golang 1.20 multiple errors.

I was recently researching some Go idioms and patterns for a lightning talk and stumbled upon what I would defined as “anonymous interface” (the correct term would be “anonymous type assertion” probably).

I would defined it as a one-off interface declaration used to answer the question: “does this struct implement this method?”

This helps solving one of the open questions of my previous blog post on errors: how to assess if an error is a “joined error”? We know that for an error to wrap multiple errors it has to implement the Unwrap method with this signature:

Unwrap() []error

Go does not provide an out of the box interface for this case, but using an anonymous interface allow us to overcome this limitation easily:

err := errors.Join(err1, err2, err3)

if e, ok := err.(interface{ Unwrap() []error }), ok {
 for _, ue := range e.Unwrap() {
 fmt.Println("Unwrapped: ", ue)
 }
}

This is not as neat as a i.e. errors.Multiple interface out of the box (but implementing it is straightforward and also helps with inversion of dependency principle) but I still find it pretty readable.

It also allowed me to discover a powerful Golang pattern, as the same pattern can be used to asses behaviour of a struct without knowing anything of the interfaces the same struct implements.

I also found a thread on in on Reddit, with this and other use cases! Common complain, to which I agree is: why don’t make it an exported interface? (The answer is probably to keep the public API small and help with forward compatibility)

A philosophy of CI design

Wed, 01 Mar 2023 11:51:53 +0100

When designing a Continuous Integration system, there are a lot of possible challenges and choices. Like the Blank page syndrome, we are faced with (almost) infinite possibilities. What is the right choice? Which system should we choose? How much should we invest? How to train developers/users?

To navigate complex choices I tend to resort to ignore all the practical details and use principles and values as North Star to guide me through the mess. The risk here is to end up with a flame war kind of debate where the discussion goes around “what is the best tool™”. Such a discussion is wrong. Is pointless, time consuming and will nowhere bring you near the solution you need.

The discussion to have is which trade offs you are willing to make today. Not tomorrow, when you don’t know what will be the context around you. Today.
Choosing a CI system is a revocable decision, but the costs associated with changing it will make it happen only when it had become really detrimental (precondition: the willness to change). Due to this factor, I consider this choice as based on a simple and long lasting trade off: the more aligned to your current needs, the less complex, the less expensive (money, time, people) in the long run. There is no “best” here, but “better in the long run for us”.

The CI problem

Before stating the principles, let me take a detour on my analysis of the “CI problem”.

I consider this discussion as composed by these separate problems:

developer toolchain; answers to “how do I install all the needed tools & requirements?”
reproducibility across environments; answers to “how to I run the same tools that others are running?”
command discovery; answers to “which is the command to perform this task?” and “how do I make sure I’m using the same command everyone is using?”
troubleshooting; answers to “this is not working for me, who may I get help from?”

All of them are intertwined, but not so much that can’t be considered each one by itself.

Developer toolchain

This issue complexity grows with the number of OSs and system architectures to support. This is a hard problem to solve. Being able to install tools reliably across all 3 of them is hard, evaluating cost/benefit is difficult and setting up your working environment is not something that happens frequently.

There are promising tools that may help here and there (looking at you Nix OS!) but then you have opinions. Vim or Emacs? Maybe VSCode or IntelliJ? What about Docker or Podman? goenv or asdf? make or mage? What about task? This list can go on forever.

We don’t care how tools are installed, we care those tools are available and at the version we expect. So let’s strat from here: provide guidance to check for tool availability and check the version you expect is installed. Installation is left to the reader.

Doing this does not mean having to write documentation that no one maintains. Write a small scripts that is capable of checking tools & versions, have your devs run it regularly. Scripts to run them all is the perfect example.

Reproducibility across environments

Keep in mind how deep the rabbit hole is. Depending on the depth you want to achieve there are various tools.

Why not start from keeping things simple? Just start by providing tools and versions informations. Are we running the linter@v1.2.3? This information should be added to your (Git?) repo. Make it part of your codebase. Write it in a file. Write small tooling to double check this. Aliging your local environment and the CI is just a matter of keeping this in sync, not the easiest task but the involved complexity is low. Being it a file is possible to cat <file> | grep <tool> for comparing versions. Tools like asdf may prove very helpful here.

Growing on the complexity scale, Docker images are the next step, but this heavily depends on what does your CI system of choice integrates (mostly will support Docker). Tools like localstack or testcontainers may be handy here. Be mindful of disk space thouh, as Docker images may ehaust disk space quickly.

Command discovery

Command discovery is the ability to know which command everybody else (and the CI runners) run to perform a task. How do you build the binary? The dev one? The relase one? How do you lint the code?

This is where I think tools like make do not really shine. They bundle too much complexity already, but their primary goal is not being simple, is being flexible. Not that simplicity and flexibility must be antagonists, but they usually are. Allowing for creating complex things, they bring in more cognitive load. They also tend to fail around debugging and being easy for newcomers (i.e. support for “dry-run modes”).

This is also where documentation tends to fail in the long term. It becomes stale and out of date.

To solve this, is great to write down commands in a task runner (a software whose goal is to run tasks, more or less integrated with your development environment) that developers use regularly.

Troubleshooting

Linter failed on CI for someone else, but you can’t reproduce. These issues can take an incredible amount of time to be solved, can be flaky, can happen for unrelated reasons and terribly frustrate developers and maintainers.

In my experience the only way to address these is to make all steps explicit enough that providing help does not require prior or deep knowledge of the toolchain. If you resort to always asking to the same person or team for “CI problems”, if you need guides or documentation, the system may be too complex and frustrating.

This is the area where I think the Unix philosophy of small single purpose tools shines and make our life easier. Instead of trying to have a single tool to do everything, make it small and precise, but with a high degree of reliability.

For any setup you choose there will always be something that do not work, but are devs able to unblock themselves or will they get stuck and need help from someone else? Will they be able to get help by anyone or just very specific (busy) people?

Principles

Detour done, here are the principles:

Ease of use
The choosen system should be the easiest fulfilling the requirements. Additional complexity “for future use” should be avoided at all costs. The more complex the tool, the more cognitive demanding its maintenance, and as our focus is on the product having heavy maintenance toolchain is a big risk in the long run in my opinion.
Needs to be compatible with the build tool of choice
There is no point in having tools that do not integrate well together. Look for constraints in your setup and build with them in mind.
CI is just another dev computer
Whatever runs on the CI needs to be reproducible locally or in a clean env. There must be no need to run things in CI to ensure the CI will pass. If you consider the CI just another development computer (with added security requirements), your setup will be reproducible in another CI system or on the new dev laptop for example.
Manage complex dependencies (or don’t)
Are you a multinational company with millions of lines of code, thousands of developers working on it and complex dependencies across project? If the answer is no, don’t think about managing complex dependencies (or tools to do that) and take the opposite route: aim for 0 dependencies. Don’t try to handled them, remove them.
Delegation over complication
Instead of trying to cram everything into a single tool, delegate to specialised tools where needed, and keep the integration point simple.

All these principles together aims at a single goal: in 6 months when you will be looking at your local dev environment and CI integration you will be able to undertand it and modify it.

Shortening the critical path

Tue, 14 Feb 2023 21:50:33 +0100

After the recent post Remove yourself from the critical path my friend and ex-colleague Fabrizio Mirabito gave me an insightful review: my definition of critical path is a synonym for dependency and while removing yourself from the critical path is a great operational technique, the strategic and tactical approach should be to remove all dependencies altogether. Will you be able to remove all of them? Maybe not, but to him this process is part of kaizen, the “change for the better” Japanese business philosophy which I would roughly translate as continuous improvement.

I totally agree with his review, so here I am delving into the topic. This post is called Shortening the critical path for a reason: while you must tend to remove all dependencies, this will likely be an enormous task to undertake. In respect to kaizen what we want is to aiming at removing all dependencies, thus shortening the critical path. The shortest critical path possible is zero: the only thing to do is doing the thing to do. No overhead.

If it’s impossible, does it make sense trying?

Short answer is yes.

Long answer starts from understanding the goal that I think we, as engineers working for a product company, have: build great products that customer love to use, solving real needs, in an efficient and sustainable way. Is my opinion that this cannot be achieved just by focusing on making money, it requires an active focus on the main keywords of the previous sentence: great product, customer, needs, efficient, sustainable. In my experience making money is a side effect of a great product, and a great product is a side effect of focusing on customer needs.

Is not just about the goal, a long critical path (or a lot of dependencies) will bring in some relevant risks:

partial done work: when you don’t finish something that partial work will be left in an uncompleted state; you may work on it again soon enough to complete it or it will become waste, as a raw food will become waste;
over production: if you are at constant risk of interruption you will naturally try to foresee possible future requests, baking them into your current product, hoping for them to prevent future work; most of the time this overproduction will become waste;
extra processing: handling dependencies (upstream or downstream) will require more work: more coordination, more planning, more meetings, more documents; Steve Jobs said “less is more” for a reason: you can’t cope with too much overhead, in the end it will just become a bloat that no one likes;
hand-offs: nothing kills velocity like hand-offs; have you ever watched a relay race? There is a clear additional challenge in passing the baton in a relay race, with require to slow down and great synchronisation.

It’s curious that 4×100m relay race world records are less than 4 times the 100m records. Why? In the 4×100 only 1 runner start from 0 speed, so that “penalty” does not impact that much the overall time.
It’s also curios that 4×100m world records are less than 400m records of ~5 seconds. They are requiring 4 times the people involved for a ~12% increase.

waiting: a key risk is waiting times and delay that reflect across the entire chain; if you’re part of a chain, any upstream delay will affect you and your delay will affect your downstream dependencies; waiting also hides in plain sight in some of the other risks, like hand-offs, extra processing or over production;
context switching: all the above risks sum up to generate the Megazorg of velocity killers; context switch also plays a bug part in rework (when something needs to be reworked because is not working as intended);
defects: what do you expect to happen if you never finish work, over produce, add additional processing, more hand-off and wait times and context switch across multiple domains? Defects. And in a downward spiral that will increase other risks.

The bad parts

While trying to mitigate this risks there are some anti-patterns that I suggest to look for and avoid doing.

🟨 Waterfall development

This post is not about Agile per se, even if is heavily inspired by it. Doing waterfall development (or any “agile” framework that package waterfall in shorter time frames) is in my opinion an anti-pattern. Waterfall implies hand-offs, waiting and context switching.

It may be I’ve never been exposed to a working waterfall development cycle, but I don’t think is the correct approach for building great software products. At least not in this present days with the fast changing landscape of requirements, needs and user expectations.

🟨 Layered development

When deciding how to work on a feature, fight the tendency to split it across infrastructure boundaries: there is no DB/Back-end/Front-end in a feature. No customer will ever see your Back-end. This will be looking at your application and judge it in its entirety.

Splitting across boundaries seems a great solution but tend to bring in partial done work (feature implemented in Back-end but not in Front-end), over production, hand-offs and waiting.

🟨 100% utilisation

You never want to be running at 100% utilisation. I spent a lot of time as SysAdmin and a 100% utilised resource (say CPU or disk space) always signal troubles.

The reason is intuitive: if you are running at 100% utilisation there is no room for error, no room for any unexpected occurrences. Anything different than “planned” will mean troubles, leading to partial done work, extra processing, context switch, defects.

🟨 Bus factor 1

Bus factor is a risk measurement that aims to surface risks associated with information silos: what is going to happen if - put here key person in your team, gets hit by a bus? We really hope this is just speculative, as being hit by a bus is bad.

The bus factor is “the minimum number of team members that have to suddenly disappear from a project before the project stalls due to lack of knowledgeable or competent personnel.” (Thanks Wikipedia! 💗)

How would your team/org/company cope with such an occurrence? If your bus factor is 1, all risks will be amplified but in particular waiting can become an infinite wait time.

The good parts

There are some corrections that could be applied to mitigate these risks.

🟦 Self-organised and self-managed teams

The first countermeasure for the risks above is having self-organised teams that can self-manage themselves. Given a purpose, the right cross functional skills and a strategy to execute, use self-organisation and self-management as empowering tools for your teams.

What do I mean by self-organised and self-managed team? Is a team where the work to be done is not assigned by the manager. The team members find their own work and manage the associated responsibilities and timelines. Is a team that has authority to choose the most effective way to organise their work. Is a team that is constantly looking around for ways to improve.

There are some key principles for this to happen:

ownership and responsibility: the core, having full ownership of a domain (it may be a specific software, or system, or library, …); this full ownership is balanced by full responsibilities on that domain.
trust: no team can be a great team without trust; trust is based on psychological safety.
communication: they require high bandwidth communication channels within the team itself; managing communications is probably one of the key challenges of creating such teams.

In the long term this kind of team will increase their efficiency and problem-solving capabilities, becoming true experts of the domain they have ownership on.

A key strategy here for managers of such a team is to build bridges with other teams. Your team will be focused on delivery, understanding the domain and becoming experts. They will probably struggle with keeping in touch with other teams or other part of the organisation. There is a huge part to play here in building bridges for your team, as bridges will be the communication channels the team can use.

This is also where some of the strategies highlighted in Remove yourself from the critical path may come handy.

🟦 Own your tools

Tool ownership is paramount. By this I don’t mean “they should own their own bare metal servers”, I mean they need to own the tool directly involving their day to day work. All the DevOps principles, self-servicing internal platforms of sort are the foundation for tool ownership.

Think of it like this: if you were an artisan (any craft applies) would you borrow your tools? Maybe that one that you use not often, costs a lot and you need only for a brief time. You would never borrow the entire toolbox each time. So why do we allow or force our teams to work like that?

🟦 Kaizen

To become better you can’t leave improvements to chance. You need discipline and a process. Kaizen is the Japanese term but you could just use “continuous improvement”. Invest part of your time in sharpening your tools, improving them, fixing them, revisiting processes and looking around constantly for things that may be improved.

This attitude, more than process, will be groundbreaking. We always underestimate the power of consistent small wins. Is the same concept as “compound interest”: each small win will be on top of other small wins. Each win stake on top and after some iteration you will end up with big wins that would have been otherwise out of reach.

A key strategy here is to always question the process: does this process/step still provides value? Be obsessive in asking this question and ruthless in pursuing improvements. Be mindful of dependencies though; in a fully self-organised team there may be 0 dependencies affected by a process, but in case the number of dependencies is not 0 keep in mind the effort required for others to adjust to the new process.

🟦 “Stop starting, start finishing”

How many things are you trying to do at once? If the answer is “too many”, you are failing prey of an anti-pattern. Context switching and high wait times leads to starting many things at once. You are waiting, “wasting” precious time, so the next obvious step is to start something else. Wrong!

When you are blocked, stuck in a queue or waiting for any reason the only logical next step is to remove the blocker that is making you wait. This idea builds on top of Owning your tools and Kaizen. Instead of context switching to a new task, think how you could improve the current situation. Is your CI taking 1 hour to run? How could you make it run in 10 minutes? Do you need to get answers from outside your team? How can you arrange your communication channels to reduce wait?

Stop starting new things, focus on finishing what you have on your plate. 100 in progress and 0 done tasks provide 0 value to your company and your customers.

A key strategy here is limiting work in progress. Like for real, don’t do more.

🟦 Is done really done?

One of the key elements in Scrum is the definition of done. I think this concept is powerful and well worth exploring.

Define what does it mean to be “done” with a task. And make it complete enough so that once done it creates value. “Implement a feature” does mean nothing if it’s not deployed and customers are not using it. It will just be another “In review” (or worst, “Done”) item that will hunt your future self. If you happen to work with long release cycles you know this feeling, and is not great when 3 months after you merge a PR you discover there was a bug, but no one noticed because it slipped through testing (and no, no amount of testing will ever replace real user behaviour).

This strategy may be impractical to apply, but shortening your release cycles and focusing on providing short and effective feedback loops so that “done is really done” and you can move forward without worries is a superpower.

Let it be

I don’t think it’s ever possible to condense all the information useful for such a journey in a blog post. Maybe in some hundreds books, but still, there is so much to say, to explore, to try.

Don’t make yourself overwhelmed by the challenge though. Use kaizen as your guide, start looking for things to improve and allocate time to improve them. I’m quite sure you will never reach the perfection, as I’m sure than in 1 year from now looking back you will see a huge leap forward.

“The longest journey begins with a single step.” - Pantanjali

SSH config for ec2 machines

Tue, 07 Feb 2023 18:45:12 +0100

Recently I had the need to configure ssh access for some AWS EC2 machines. One annoying element of EC2 machines is that they default to user ec2-user, which is easy to forget.

Public routable EC2 instances hostnames is in the form of: ec2-<IP>.compute-1.amazonaws.com. Due to this is easy to create a SSH config file entry to set the default user to ec2-user for these hostnames:

Host *.amazonaws.com
User ec2-user

From this point onward all connections to instances whose hostname ends in .amazonaws.com will default to the correct value.

What if I have a non-publicly routable instance? Depending on the hostname, you may be able to slightly change the configuration above to suit your needs. If you’re using IP addresses though, you should either do it manually for each host or find a way to automating the manual process.

Golang 1.20 - multiple errors

Thu, 02 Feb 2023 19:02:58 +0100

Golang 1.20 has been released 🎉

One of the key changes that I noticed reading the release notes, is that it adds support for handling multiple errors natively. This is awesome, and I wanted to dig deeper into how to leverage this.

This is what release notes say:

Wrapping multiple errors

Go 1.20 expands support for error wrapping to permit an error to wrap multiple other errors.
An error e can wrap more than one error by providing an Unwrap method that returns a []error.
The errors.Is and errors.As functions have been updated to inspect multiply wrapped errors.
The fmt.Errorf function now supports multiple occurrences of the %w format verb, which will cause it to return an error that wraps all of those error operands.
The new function errors.Join returns an error wrapping a list of errors.

The implementation is lean as usual, no cruft attached.

Let’s review the changes.

First, an error is considered wrapping multiple errors when it has a method with the signature:

Unwrap() []error

All relevant functions for error checking error.Is() and error.As() has been updated, so they can be used to infer error presence in a chain even with multiple errors (neat!).
Formatting function has been updated too, so is possible to write:

fmt.Errorf("%w and %w", err1, err2)

Last but not least, we get a function to wrap multiple errors in a single one, errors.Join().

It seems all the needed building blocks are there, so let’s look at how to use them.

Caveats

`error` does not implement `Unwrap() []error`

The first thing to notice here is that the builtin error interface, does not mention Unwrap() []error at all (docs), so is not possible to run Unwrap() on errors joined with errors.Join() or fmt.Errorf.

This is a bit confusing at first, as you can join errors very easily but then is not possible to unwrap them.

`errors.Unwrap()` will return `nil` for multiple errors

As specified in the proposal, errors.Unwrap() will returns nil if the Unwrap method returns []error.

This decision prevent this change to break compatibility with previous Golang versions, but is a caveat to keep in mind.

Examples

Using `error`s

err1 := errors.New("err1")
err2 := errors.New("err2")

errs := errors.Join(err1, err2)

errs will be report true to errors.Is for both err1 and err2.

But errs does not implement .Unwrap(), so there is no way to unwrap this error and reuse single errors.

Playground code

Formatting `error`s

err1 := errors.New("err1")
err2 := errors.New("err2")

errs := fmt.Errorf("%w | %w", err1, err2)

As in the previous example, errs will be report true to errors.Is for both err1 and err2, but does not implement .Unwrap().

Playground code

Custom error

If we build a custom type implementing error and the new Unwrap() function, we can range over errors, as I initially expected.

type CustomError struct {
 msg string
 errs []error
}

func (f CustomError) Error() string { return f.msg }
func (e CustomError) Unwrap() []error {
 return e.errs
}

...

errs := CustomError{"My Error", []error{err1, err2}}

if errors.Is(errs, err1) {
 fmt.Println(" err is err1")
}
if errors.Is(errs, err2) {
 fmt.Println(" err is err2")
}

for _, e := range errs.Unwrap() {
 fmt.Printf(" unwrapped err: %s\n", e)
}

Playground code

This is great, and I find it as a nice improvement over what I’m doing in my tool endorama/devid.

Admittedly the solution was decent, using 2 return values ([]err, err) where the usual error would signal the possibility for other errors to be present. But was lacking integrated support for checking the error chain, so that will make a great improvement!

Remove yourself from the critical path

Thu, 26 Jan 2023 22:14:33 +0100

During one of my recent mentoring sessions, an interesting discussion started with my mentee. How do you manage being a leader with being hands-on too?

If you have some experience in leading a group I suspect you faced the same issue at some point, maybe not with coding but for sure around the “I can’t be one of the hands-on people anymore”. I don’t think this is necessarily true, but being hands-on and a great leader is a tough challenge and its practicality is heavily influenced by the environment around it.

During our session the main issue were interruptions and meetings driving away time from focused work.

I there a way out of this? I don’t know, but in my experience you can help yourself. The goal in this situation is to create space for deep work, chunks of time in which you expect no interruptions and no meetings. In which you can focus and do deep work.

There are calendar blocking techniques that can be tried out, but that’s like curing the symptoms and not the causes: if everybody needs you, you will never be able to get that focus time, because there will always be something requiring you attention.

We can now agree that improving on this situation is hard, requires dedication and a lot of learning. Especially if you are moving from an individual contributor role to a team leading role, as you need to adjust your thinking and approach; you are now the one everybody else count on.

My suggestion here would be: remove yourself from the critical path. Stop being the one everybody needs to go to.

What’s the critical path?

Doing something, in a generic way, involves going from a Point A to a Point B. To move from A to B a series of steps are required. The critical path is that path from A to B that you cannot avoid taking; is composed of those steps that are mandatory to move from A to B.

Let’s visualise this with some example. Let’s say we are travelling from City A to City B. It doesn’t matter which transport are we using, we will need to go through some physical places. And we hope there will be no accident, delay or impediment along our route, because that would slow us down. Or maybe we are taking a taxi, we hope there will be no slow downs as that will make the run more costly. Or maybe you need to study for an exam, so you will want to avoid getting sick, as that would reduce the time you have available for studying, making you risk to fail the exam.

The critical path is the series of steps that needs to be performed to achieve an objective. Any impediment in the critical path will make achieving the objective be more time consuming, more costly and/or not be ready for the deadline.

We can say that this critical path is the most important element you want to watch.

Am I in the critical path?

This is a worthy question! How do you identify if you are in the critical path?

Does any of this happen to you:

other come to you for answers that they need to do their work
only you have the knowledge needed by others
you get interrupted frequently (by your reports, your peers or your manager)
you need to take decisions and people depends on them, so they are blocked waiting for your decision

If yes you are in the critical path. And now what?

There are some strategies you may try to move away from it. Those strategies require time and trust as their main ingredients, as removing yourself from the critical path is all about delegation. You need to learn how to delegate (a skill that you generally don’t need as individual contributor), and the more effective you will be at it, the more effective your work as a manager.
The hard part of delegation is that is not about doing something, is about not doing something. But that something has to happen anyway. Delegation is hard. Delegation involves trust. Trust is even harder.

So how do you build trust? Here are some techniques. In the next examples I’ll be using teacher and student as “the one that knows how to perform a task right now” and “the one that should learn how to perform a task in the future”.

Shadowing

Is there a task you do that someone else could do with some training? How do you deliver that training? Shadowing may be a solution. The student follows (or “shadows”) the teacher while they perform the task. The goal is for the student to be independent in executing the task after a set period.

Other literature is available, but I want to give some hints:

define a deadline after which the shadowing period ends
apply to task that may benefit from observation (es interviewing, meetings) but do not apply to heavy concept base tasks where a lot of prior knowledge is required

Pairing

Do you have a knowledge-heavy task at hand that you would like to delegate but don’t know how as it requires a lot of prior knowledge? In this case shadowing is probably not enough. Do pairing sessions, as in “code pairing” sessions, but for other topics.

During those sessions select a driver and a writer. The driver direct how to do things, the writer executes. Exchange roles after some time. This is a great way to get to know the other person and build trust, while sharing knowledge.

Proposal documents

Do you have a task that needs some thinking, where multiple solutions needs to be explored and there is no clear solution? Delegate writing a proposal document and review it together. Going through a document has many benefits: it allows asynchronous communications, will last for future people (“verba volant, scripta manent” Latins used to say) and will provide for time and space to have an insightful conversation.

Do remember to appoint a driver for the document, who will be in charge of moving from draft to the final decision. The driver does not have to be the one taking the decision too, on this subject there is a whole lot of literature, so go surfing!

Pitfalls

Delegation does not come without its risks. Watch out for things that you don’t want to happen, some of which I summarise below.

Not all tasks are the right candidate for delegation

That task the CEO is really looking for and keeps you accountable for? Probably that’s not the best task to delegate.

Not all tasks can be delegated, but fortunately not all tasks have the same importance (as impact * complexity * urgency). Knowing which task to delegate is one of the hidden skills of great leaders. I think that for being great at it you need to remember that half of it is your judgement but the other half is the environment around you.

Some tasks that may be good candidates are: any decision that can be reverted (from less expensive to most expensive to revert), any task outside of the critical path, any “nice to have”.

Another judgement method you can apply here is what you are aiming for: are you aiming for more personal focus time? Are you aiming to making sure the critical path does not have any impediment? Are you trying to upskill someone? Keep in mind your goal when delegating, as it affects your decisions.

One framework to help here is the Eisenhower Matrix.

Don’t setup people for failure

If you remove yourself from critical paths but is not safe to do so or is not appropriate given your team health and competence, you are setting up people for failure.

Failure brings pressure and frustration. Frustration leads to fear or anger. And we know how this ends, fear leads to the dark side.

Over time people will get disengaged, they may even get mentally or physically sick due to the excessive burden and in the end leave the group (or company). Don’t be that leader.

Be okay with people failing!

Have you ever done something for the first time? Like riding a bike, singing, studying a subject? Where you good at it? I bet not that much.

When someone else will start doing something, in particular something you are used to do frequently, they will probably not be good at it. They are building their way of doing it, and the only way for this to happen and to make them grow is to allow them to fail in a safe space. Your goal is to build the safe space, not to prevent them from failing.

Be okay with having hard talks

People you delegate to will not do everything right. As you will be responsible for the outcome, you need to learn to have hard conversations.

You need to learn how to talk about issues, missteps and errors. My advise is to learn how to be emphatetic, respectful and transparent. We tend to associate negative feedback with something bad, but learning how to give negative feedback in a human and constructive way is a golden skill, it build trust.

If you think this is hard, consider the alternative: everything will be great until it won’t be anymore, the number of wrong things will probably be too much and you will have way worst conversations to have.

Stop thinking about the effort, start thinking about the outcome

When people in your team feel safe and empowered they will do awesome things together. A team is more than the sum of its components. Your team will be more resilient, more effective. You will be more resilient and more effective.

Whew, this was a long one! This advice does not applies to managers/team-lead only, I found it useful in other roles too. As always, the goal is not to be perfect but to understand how to improve and acquire some new tools for your tool belt.

As leaders, leading other people is a challenge but will give back many rewards if you’re committed to improve and clearly show that you care for people.

My note taking in 2022

Wed, 11 Jan 2023 18:52:01 +0100

During 2021 I completely restructured the way I was taking notes. The goal was to up the game and be more effective and get more proficient at it.

At that time I was using Standard Notes, an e2e encrypted note taking app with rich features. I still think is a great app, but after using it for some years I noticed it was coming short: note search was missing full text search and organising them through tags was not enough.
I would still recommend it if you take linear notes as it’s packed with features and developed by a passionate team set up for the long run.

When I started searching for alternatives, I got exposed to the idea of “non-linear note taking”.

1 Minute history of note taking

We take notes to free our mind from having to recall all of them.

Since the invention of writing, note taking has played an important role in human history and scientific development. Quoting Wikipedia:

The Ancient Greeks developed hypomnema, personal records on important subjects. In the Renaissance and early modern period, students learned to take notes in schools, academies and universities, often producing beautiful volumes that served as reference works after they finished their studies. In pre-digital times, people used many kinds of notebooks, including commonplace books, accounting waste books, and marginalia.

Note taking is complex, as it requires a moltitude of cognitive function to produce a working result. Is also a requirement to manage large amount of information.

Note taking takes also different shapes and forms, in the effort to structure the content so it’s easier to find and consume at a later moment. We can cluster different methods in 2 main categoris: linear and non-linear note taking.

Linear note taking

Linear notes take the form of outlines or sentences. Outlines can be bullet points or phrases, organised and structured in a logical way from top to bottom, separated by headings. Sentences are short phrases that summarise a concept, and is losely organised.

Non-linear note taking

If you ever created a mind map you used a non-linear note taking method. In non-linear note taking notes are split as much as possible to bite-sized information that are then clustered.

Some of these methods come from long time ago: Cornell Note system is from 1950, Zettelkasten system is from mid 16th Century.

Welcome to the digital age

So it’s 2022 and looking at way to take notes I start digging deep into non-linear note taking. That’s the moment in which my fellow Standard Notes does not keep up. It was also the moment in which Roam Research started becoming popular, as its (freemium + file based) alternative Obsidian.md.

Obsidian is a great and extensible tool, powerful and file-based. This means I’m in control of the content, which is in a standard (and common for me) format, Markdown with frontmatter.

I started using Obsidian, breaking down my notes experimenting with Zettelkasten. That proved incredibly useful for content I wanted to learn and remember.

I was facing a new issue though, as I came to Obsidian early enough that there were still some rough edges. One of them was the ability to easily sync across devices (without paying) and collecting thoughs on the go. Another was using it to take linear notes in a fast way while following talks, videos or courses.

To fill this gap I discovered Workflowy, which is an infinite nestable bullet list. Workflowy power lies in its flexibility. You can basically do whatever you want and build whatever system you can think of using it. It also allows to tag bullet, link and mirror them. It behaves likes an hybrid linear/non-linear On top of all this, its pricing tag is amazing compared to the competitors.

They both served me well and I’m happy with the setup, even if there are still rough edges that I plan to iron out in the future.

2022 ends

This is my setup at the end of 2022:

In Workflowy:
- all things to follow up go to Workflowy, as child of the Inbox bullet; untagged, rough, mostly links or spare thoughs. Goal is to insert them as fast as possible.
- under a Personal bullet I have:
  - an Eisenhover TODO matrix
  - ideas I have, tagged; generally moved from Inbox after reflection
  - info on Projects I work on
  - spare notes on meetings I have
  - a collection of interesting quotes
  - a collection of possible presents for my family and friends
  - an Archive page where I move information that is not relevant anymore
- under a Work bullet I have:
  - an Eisenhover TODO matrix
  - other information about day to day things organised with the PARA method.
In Obsidian:
- a Zettelkasten system where I add information
- small notes, each linked to a single Topic and tagged with cross-note or topic
- a series of pages to collect related pages; for example: listing all conference notes, all people notes, all topics notes.

Obsidian has a nice little feature, is able to graph the relation between notes. This is how my graph looks like:

The power of this structure is that I’m creating links between notes, which translates to link between concepts or people. These links are powerful for creating new content and refining ideas on past content.

What’s next?

I’m happy with the current process, but everything can be improved and there are some rough edges that I’d like to smooth.

I’m currently running 3 main experiements: a reMarkable tablet, Logseq and Voiceliner.

As an unexpected present I recently received a reMarkable 2 tablet. Taking note by hand again, but in a digitalised fashion, is refreshing and something I was really looking forward. I’ve not used it enough to fully understand where to put it in my workflow, if not for highlighting technical books or articles I read. I plan to extend its usage and integrate it with other tools, how is still unclear.
As there are scientific studies (like The Pen Is Mightier Than the Keyboard) that point that taking notes by hand is better that typing them for learning, what I expect to be doing in the future is add an additional first step by hand that then gets digitalised and moved to its digital storage.
The other experiment I’m running is with Logseq. A thing that I’ve not been able to fit in all this so far has been journaling. Workflowy makes it too manual, Obsidian requires plugins or manual toil; overall it seems is neither provides a first hand experience on this. Logseq does. Another element that I’m eager to try are flash cards, but adding another tool seems to outweight the value, and it really does not fit in Workflowy and partially fits in Obsidian. Logseq does that to! Additionally Logseq has a bullet point approch like Workflowy and is file based like Obsidian. It’s promising, albeit still in early stage (but the community around it is great).
The last experiment is with Voiceliner, a speech to text recognizer bundled in a mobile app that produces Markdown files. By syncing them to Obsidian or Logseq I foresee some interesting integration there!

All of this has been quite a journey, but I’m extremely pleased at how much value I get from it. The next step will be to consolidate and integrate tools, form better note taking habits and better define processes around getting value from this personal knowledge base. In the end, that’s why we take note in the first place.

What I'm doing Now

Mon, 28 Nov 2022 22:37:52 +0200

This is a now page, inspired by Derek Sivers.

Work

Since 2021, I’m working at Elastic, the Search, Observe and Protect company, as a software engineer in the team responsible for data intake for Elastic Observability.

Open Source

This is a list of project I’m actively involved:

endorama/2ami - Your easy 2FA companion that keep the secrets secret
ansible-collections/community.sops - Simple and flexible tool for managing secrets
endorama/asdf-parse-tool-versions - A tool to parse asdf .tool-versions in a GitHub action and reuse the specified values

Life

Recently married, facilitator for Climate Fresk workshops. Serial note taker.

Making your shell scripts bulletproof pt.2

Mon, 28 Nov 2022 21:31:39 +0100

I posted some suggestion to improve robustness of shell scripts in https://endorama.dev/2022/making-your-shell-scripts-bulletproof/, but there are a few more that I usually use that I think may be useful, so here is part 2 of it.

Don’t split words on space

The Internal Field Separator that is used for word splitting after expansion and to split lines into words with the read builtin command. The default value is <space><tab><newline>.

From BASH manual, under SHELL BUILTIN COMMANDS > Shell Variables.

The default value of this variable means BASH split lines on spaces too. Most of the time this is a very confusing behaviour, as we are not used to consider a space a sentence terminator or split point (compared to newline or tab character).

To change this to only newlines and tabs use:

IFS=$'\n\t'

99% of the time I never ever noticed the difference in my shell scripts, and if you really need splitting on space you can always prefix your command with IFS=$' '.

You may also want to save the original IFS value and you can do so by:

ORIGINAL_IFS=$IFS
IFS=$'\n\t'

This way you can still use the old IFS value by prefixing commands with IFS=$ORIGINAL_IFS.

Use traps

….

Use shellcheck

I don’t think I came across something more beneficial to my shell script writing than Shellcheck. Shellcheck is a static analysis tool for shell scripts (BASH is supported too).

The reason why you must run this is that will help you avoid all those errors you may be unconsciously making right now.

Stop formatting your code

…manually!. Use shfmt to do it automatically! shfmt is part of a Golang-based shell parser and interpreter, a really cool project.

Test BASH scripts with `bats`

Unit testing, in BASH. HELL YEAH! bats is a small framework to do unit testing in BASH (you can actually use it for any command that BASH can run, but let’s stay focused here).

Happy coding 😉

Bring out the trash!

Mon, 28 Nov 2022 20:31:39 +0100

I recently stumbled upon the usual issue with all my development machines: “no space left on device”.

As it happens frequently, I had to spin up the trusted disk analyser, Baobab and dig into it.

Something was filling up my 512GB disk, but exactly what?

My home folder

Well, that wasn’t that suprising, but:

My Trash was 31.2GB!
I used asdf to manage programming language interpreter/runtimes, this was taking ~30GB, 24 of them being taken by Golang installations and relative packages
My code folder, where I keep all code I’ve been hacking/working on, was ~24GB ⛰ Massive!

Nix

I started playing with Nix and Home Manager to manage external binaries I want to add to my workstation (that runs elementary OS Hera at the moment).

It replaces Homebrew, that I loved but has it’s drawbacks on Linux: compiling things was a total nightmare, and it happened quite a lot.

I was also extremely intrigued by the declarative approach Nix that promises reproducible configurations, something I’m trying to achieve to reduce surprises (works on my computer means work on remote server, this is a dream for a SysAdmin).

In the end I had 99GB in /nix 😱

To clean them up I discovered I needed to run GC and optimise the nix-store:

$ nix-store --gc
$ nix-store --optimise
$ nix-collect-garbage -d

These commands combined cleaned up around 50GB of disk space 😎

I followed along this and this articles.

Snaps

The next step were snaps, where ~7GB where used.

2 steps here:

reduced the number of retained versions to 2 (default is 3)
cleanup all disabled snaps

I followed instructions on this article, with the addition of using this command to check for active services:

$ snap info "$snapname" | grep -w active

Golang string comparison benchmark

Thu, 10 Nov 2022 22:48:03 +0100

I recently read No Safe Efficient Ways to Do Three-way String Comparisons in Go - Go 101 Blog about performing string comparisons and some performances issue that the strings.Compare function has.

The main statement of the post is

Benchmark code constantly shows strings.Compare uses 2x CPU time of bytes.Compare when comparing unequal same-length byte sequences (we view both strings and byte slices as byte sequences here).

It got me curious so I jumped to the editor to check what was happening. I was not interested in debunking the claim, I was interested in understanding how different string comparison implementation performances differed.

In the linked post you can find the source code for strings.Compare and bytes.Compare.

My initial assumptions:

strings.Compare is 2x slower than bytes.Compare as stated in the article
built-ins are faster than strings.Compare as implied by Golang documentation, that states: “It is usually clearer and always faster to use the built-in string comparison operators ==, <, >, and so on.”
converting string to bytes and using bytes.Compare may be beneficial (especially for longer strings)

To test my assumptions I implemented 4 test cases (and corresponding _long versions):

BuiltIn test; string comparison using built-in operators
Strings test; string comparison using strings.Compare
Bytes test; bytes comparison using bytes.Compare with string to bytes conversion just for the comparison
BytesWithBytes test; bytes comparison using bytes.Compare with byte sequences (as control)

You can find The Code™ in this repository: endorama/go-strcmp-bench.

The setup

Golang 1.19.2
Benchmark run with go test -bench=. -v --benchmem -count=20 and shown with benchstat
2 short strings (6 chars) and 2 long strings (255 chars) and their bytes conversion. There is an interesting worst case scenario in strings.Compare: its performances are at worst when comparing 2 unequal strings of equal length. For this reason the strings are equal length and differ for the last char.

Benchmark results with short strings

name time/op
BuiltIn-12 15.3ns ± 3%
Bytes-12 42.3ns ± 3%
BytesWithBytes-12 9.23ns ± 2%
Strings-12 15.6ns ± 1%

As expected strings.Compare is ~2x slower than bytes.Compare.
Unexpectedly, using built-in is not that much faster?
More unexpectedly, converting to bytes is massively less performing, ~3x less than comparing strings and ~4x less than comparing bytes.

Benchmark results with long strings

name time/op
BuiltIn_long-12 32.1ns ± 5%
Bytes_long-12 366ns ± 6%
BytesWithBytes_long-12 20.7ns ± 3%
Strings_long-12 37.4ns ± 3%

As expected, strings.Compare is still 2x slower than bytes.Compare. ✔
Built-in is slightly better. 🤔
Don’t look at bytes conversion. 🙈

Conclusions / TL;DR

name time/op (short) time/op (long) diff to bytes diff to strings
BuiltIn 15.3ns ± 3% 32.1ns ± 5% ~2x to 1.5x 1x
Bytes 42.3ns ± 3% 366ns ± 6% ~4x to ~18x ~3x to ~11x
BytesWithBytes 9.23ns ± 2% 20.7ns ± 3% 1x ~0.6x
Strings 15.6ns ± 1% 37.4ns ± 3% ~1.5x to ~2x ~1x

built-in is not (that) faster 🤷
use strings.Compare if you think is more readable and you have short strings or you’re not performing a lot of comparisons
don’t convert a string to bytes, as it nukes performances 🪃
- is possible to make this faster using unsafe pointers, but 🐉 here be dragons 🐉

I haven’t delve into a deep comparison for built-ins and strings.Compare for this time.

Stop observing, start making

Thu, 03 Nov 2022 18:21:16 +0100

Observability has become the read-it-everywhere buzzword of software development. Like others before it.

Observing any piece of technology is incredibly easy.
Look at your screen, intensely.
Do it some more.
You are observing technology. If it’s on, you are observing software too.

So what now? You basically know nothing more about your system now that you did before observing it (except for, is it on 🔥?).

The reason why observing a piece of software is just a useless exercise is because observability is not an action. There is no way you can observe a system, unless the system itself is observable.

The term “observability” was coined by engineer Rudolf E. Kálmán in 1960.
In his 1960 paper, Kálmán introduced a characterization he called observability to describe mathematical control systems. In control theory, observability is defined as a measure of how well internal states of a system can be inferred from knowledge of its external outputs.

The Mathematical Definition of Observability from Observability Engineering by Charity Majors, Liz Fong-Jones, George Miranda

Observability is not the act of observing a system, is building its observable properties so you can extract meaningful information when observing it. Gathering the data is the first step in “observability”, but you have to explore them to conclude they are of any value.

So stop observing and start making your systems observable.

If this reasoning did not convince you yet, happy to provide more clues supporting this conclusion.

Charity Majors in Observability is a Many-Splendored Definition

Plenty of tools are terrific at helping you ask the questions you could predict wanting to ask in advance. […] But if you can’t predict all the questions you’ll need to ask in advance, or if you don’t know what you’re looking for, then you’re in o11y territory.

Jean Yang in Observability Isn’t About Logs, Metrics, and Traces

To generalize, observability is about helping people build models of their systems so they can: Understand your system better to move faster. Get visibility into how your system is actually running. Keep an eye on how your system changes.

Dotan Horovits in Observability Is A Data Analytics Problem

However logs, metrics and traces are, by themselves, not Observability. In fact, many organizations collect logs, metrics and traces, and still end up with poor observability. […] But more importantly, we need to remember that these signals are, after all, the raw data. And we’re looking for the insights and the root cause analysis, to understand our system.

Multiple authors @ Google in Site Reliability Workbook

Your Users, Not Your Monitoring, Decide Your Reliability
[…] Your monitoring, logs, and alerting are valuable only insofar as they help you notice problems before your customers do.

It’s a long journey from here, but rewarding.

Recovery metadata from pdf

Thu, 20 Oct 2022 19:14:07 +0200

Recently a friend of mine asked my opinion on Muun, an open source Bitcoin and Lighting client.

I’m not particularly involved with cryptocurrencies due to their environmental impact but I think the technology and the evolution around its space are interesting to study. The idea of distributed consensus may not yet have a worldwide practical application but is nevertheless an interesting achievement.

What made it interesting is the self-custodial aspect. I got interested when I read about the Emergency Kit, which is probably still a burden for most users but it’s a great step forward in usability.

While looking at the Muun source code I discovered they have a Go tool called recovery (that allows to transfer funds out of your account without independently of the platform itself).

I started reading the code after looking at the initial GIF displaying the tool loading the Emergency Kit data from PDF 🧐

In the code I found an interesting function that extracts a JSON file from the PDF attachments and uses it to load emergency kit data.

// ReadMetadata returns the deserialized metadata file embedded in the SrcFile PDF.
func (mr *MetadataReader) ReadMetadata() (*Metadata, error) {

Check out the code

That’s a really cool idea to provide a great human + machine experience for a sensitive operation like account recovery.

Making your shell scripts bulletproof

Mon, 19 Sep 2022 21:13:23 +0200

Writing shell scripts in BASH is something I started doing around 2008 probably, but it’s only after some years I started doing it in a way that was preventing me from shooting myself in the foot.

BASH is great and powerful, allows to harness the power of UNIX tools, but its syntax it’s unforgiving and not easy to master (and at times even after more than 10 years, confusing).

In this post I’ll go through some of my mandatory patterns to add robustness to my shell scripts.

TL;DR:

The full template for these suggestions would be:

#!/usr/bin/env bash
set -euo pipefail

Using the interpreter from the environment

Use env for the shebang:

#!/usr/bin/env bash

There are occasions where this may not be advisable (especially if you use the shebang for other languages), but for BASH generally is better to use /usr/bin/env for the shebang. See this StackOverflow answer for a broader discussion.

Using /usr/bin/env tells the OS to search for the specified executable in the user PATH instead on a fixed location, this makes it more portable. env is usually available at /usr/bin/env, you can double check that with command -v env.

Catch unset variables

Treat unset variables and parameters other than the special parameters “@” and “*” as an error when performing parameter expansion.

From BASH manual, under SHELL BUILTIN COMMANDS > set.

One of the most common issues is running a BASH script with an undefined variable. BASH does not complain at all (and that’s how I nuked 6.5TB of hard drives with a missing variable - but that’s for another story) and execute your script as if an empty string was your value.

Why wouldn’t you want to do that? Let’s look at our example script:

#!/usr/bin/env bash
folder=$1
rm -fr /home/$USER/$folder

This simple script is supposed to delete user provided $folder in user home directory. What do you think will happen if $1 (shell script first argument) is not provided? 😱 It will delete all your home folder. YAY. This is bad because rm is reasonably fast and unlinks the files, which means the OS may decide to write on the just created free space almost immediately; before you realize and hit CTRL+C (don’t do that, read here) a huge portion will have been unlinked already. Recovering them is painful.

To prevent this you can use:

set -u

or its longest form:

set -o nounset

What to do with variables that you want to be empty, for example because you want to check for value? nounset refers to unset variables, which means any other value (even an empty one) satisfies this requirement.

My usual case is checking for environment variable, which can be done easily with:


if [ -z "${ENVVAR:-}" ]; then
 # here ENVVAR has non-zero length
else
 # here ENVVAR has zero length but does not trigger nounset option
fi

(-z is part of test utility and checks if the supplied argument is a zero-length string)

Exit on error

Exit immediately if a pipeline (which may consist of a single simple command), a list, or a compound command […], exits with a non-zero status.

From BASH manual, under SHELL BUILTIN COMMANDS > set.

When an error occurs in a BASH script, the script does not exit immediately but continue running; this can create weird situations (think about changing to a different folder, or running a setup command).

To prevent this use:

set -e

or its longest form:

set -o errexit

Use the rightmost exit value for a pipeline

If set, the return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands in the pipeline exit successfully.

From BASH manual, under SHELL BUILTIN COMMANDS > set.

When running a command pipeline (i.e cat file | grep foo) the exit status of the pipeline depends from the first command run. In this cat/grep case, if the file exist but does not contain foo, cat will exit with 0 but grep will exit with a non-zero exit code. Nevertheless the exit value for the entire expression would be 0.

To prevent this use:

set -o pipefail

Combining -e and -o pipefail makes impossible to have “failing” commands (not part of conditions), like in the case above, because -o pipefail will make the pipeline exit with a non-zero code and -e will immediately halt the script. To prevent this, when you want to continue even in case of error you can end the pipeline with || true; this instruction will run true, a utility that “do nothing, successfully” and change the rightmost exit code to 0.

Es:

set -eo pipefail
cat file | grep foo || true
echo "continue"

Happy coding 😉

2ami

Sat, 17 Sep 2022 00:32:25 +0200

Some years ago I was learning Golang and researching a 2FA tool to use in the command line (that I wanted to integrate with ULauncher, a Linux alternative to Alfred). The only tool I found was saving all secrets in a plain text file and I really wanted to avoid that.

So 2ami was born! 2ami is a small CLI tool that allows to generate 2FA tokens. Is a single binary you can run and uses your OS secure storage to store secrets. It’s simple but complete and being my companion for years now.

I recently released 0.6.0, the first cross platform (Linux, macOS, Windows) build after fighting a lot with the CI (it needs CGO) and thanks to GitHub Actions (which I keep admiring for how powerful and effective they are).

If you want to checkit out ➡ https://lnkd.in/dWY-Y_6e

Golang divisions

Thu, 15 Sep 2022 12:31:05 +0200

At work today I read a piece of code in Golang that got me curious:

maxQuerySize := 500
// metricDataQueries is user provided
sliceLength := len(metricDataQueries)/maxQuerySize + 1
// what's the value of sliceLength?
for i := 0; i < sliceLength; i++ {
 whatever
}

This is a simple integer division, but the result is then used as a parameter to the for loop. Integer division has a drawback, that can create a bug in this case: the remainder, as float in for loop declarations are supported only if no truncation happens when converting to int (i.e. 10.0 works, 10.1 don’t).

How does Go handle this? The answer is as simple as it gets: remainder is ignore and the result is the quotient. The reason why this happens is explained in Go specs, under Constant expressions:

Any other operation on untyped constants results in an untyped constant of the same kind; that is, a boolean, integer, floating-point, complex, or string constant. If the untyped operands of a binary operation (other than a shift) are of different kinds, the result is of the operand’s kind that appears later in this list: integer, rune, floating-point, complex.

Which means that to have “true” division you need to convert at least one argument to float:

a := 10.0 / 2
b := 10 / 2.0
c := 10.0 / 2.0
d := 10 / 2
fmt.Println(reflect.TypeOf(a)) // float
fmt.Println(reflect.TypeOf(b)) // float
fmt.Println(reflect.TypeOf(c)) // float
fmt.Println(reflect.TypeOf(d)) // int

go.dev/play

Doing integer division effectively “floors” the result of the operation. What if we want the ceil of it?

Go math package has Ceil(x float64) float64 for this, which implies doing a “true” division and applying Ceil on the result.

All of this was not really surprising, but it’s cool to see this being implemented in a very ergonomic way.

Just a note: if you want to get a float value you must convert to float all division arguments before the division.

Do this: float64(2) / float64(10) // 0.2 and not this: a := float64(2 / 10) // 0.

asdf, little awesome tool

Wed, 17 Aug 2022 23:46:30 +0200

Some years ago I discovered a little tool with a huge potential. I started using it and since then it proved useful in so many ways it’s now a mandatory tool in my toolset.

This tool is asdf. asdf is a CLI tool version manager. You may already be using multiple ones (rbenv, nvm, goenv to name a few) to manage versions of programming language tooling and accessories.

Having multiple tools for this purposes creates a series of disadvantages:

setting up your workstation is more painful (multiple tools to install, multiple versions to manage); this become more complex when you work in a team and want to provide a great onboarding experience
adopting a new tool force you to first search for a dedicated version manager, then install the tool (you may go without one, but that’s going to bite your future self)
how to share the same version in all environments, in particular dev and CI? Was that file .go-version, .golang-version? Each tool uses it’s own convention, sometimes they align but having a consistent experience is not in their DNA;
what’s the command again? Each tool may have different CLI commands, making switching between them a pain; this may happen more easily if you’re in a team;
remember the Good Old Times™ of “spaces VS tabs”? Well, why start a 🔥 war around which tool to use?

Let’s summarise why asdf is ground breaking:

helps developer onboarding across a wide set of tool;
helps team collaborating;
removes useless discussion drive by preference;
has a consistent UI.

Would that be enough? NO 😜

It delivers more:

creating plugins is so easy you can just add whatever you want (I did it here, here, here and here);
creating plugins is most probably not needed because what you need is already available; asdf-community has 60 repositories (so around 60 plugins), just a asdf plugin-add away;
are we forgetting the CI? Hell no! Note that this only works if you are using GitHub Actions, but if that’s the case there are actions ready to be used.

In case you don’t want to use the install action provided by the asdf community because it does not leverages other optimized setup actions, I developed an action that parses the .tool-versions file extracting each tool version so they can then be reused in other actions. No asdf needed!
This action is available here.

If you reached this line, STOP right here and go install asdf.

Contact me

Wed, 17 Aug 2022 00:37:52 +0200

Before contacting me, please take note of the following:

please be polite, I’ll always be polite to you;
if you’re a recruiter I would prefer if you get in touch through LinkedIn, where you can also see a more detailed profile.

Email: its2023@endorama.dev

Email is the easiest way to get in touch with me. Expect a reply in a day or two.

About me

Mon, 08 Aug 2022 19:37:52 +0200

👋 Nice to meet you, I’m Edoardo!

Disclaimer

This is my personal website. The opinions expressed here are my own and not those of my employer.

In addition, my opinion change in time as I meet new ideas, opinions and facts; I consider this healthy and necessary, the consequence of an open mind.

This blog acts as a semi-permanent, point in time snapshot of my thoughs and opinions. As such, any thoughts and opinions expressed within my previous posts may not be the same, or even similar, to those I may hold today.

Anyway…

I’m currently based in Italy.

I started my professional life around 2010, I’m a self-taught software developer and sysadmin, Open source advocate and contributor, obsessed with creating value for others.

See what I’m up to now.

In my professional life, I’ve worn many hats:

DevOps engineer at Arduino, the Open-Source electronic prototyping platform;
Front-end, Back-end and Cloud developer at Vitsoe, a British furniture manufacturing company, realising furniture designs of Dieter Rams;
Front-end, Back-end and maker at TODO, an Italian interaction design studio;
Front-end, Backend and SysAdmin at weLaika, an Italian web agency.

I started learning programming through online resources and volunteering in an association were we developed websites and did SysAdmin work. I got my first job as beta tester and SysAdmin in 2010 at weLaika. During all the years I learn a bunch of web frameworks, multiple programming languages and an infinite amount of tools, and I keep learning.

I’m a Linux (mainly Ubuntu) user since ~2010 and currently using elementary OS as a daily driver.

I’m married, enjoy hiking and volunteering. I’m passionate about the Open Source, internet culture and knowledge management systems. I absolutely love crossovers, where cultures, styles, whatever mix to create something new - original + beautiful == 👨‍🍳👌.

Hello World

Mon, 08 Aug 2022 19:25:09 +0200

All stories should start from somewhere! This is where the story for my website starts.

Mon, 01 Jan 0001 00:00:00 +0000

👋 Edoardo is a software engineer and maker from Italy. Self taught and curious, with more than 10 years of experience in the field. Come say Hi!

Edoardo Tenani

Improving LUKS performances on SSDs

To make or not to make

Pitfalls

Can we do better?

Rake

Task

Just

Conclusions

Smartphone Ecodesign labelling comes to EU

Per project LSP settings in LunarVim

Bonus trick

Old blog, new blog

Inspecting TLS traffic golang

Better detection of Golang wrapped multiple errors

A philosophy of CI design

The CI problem

Developer toolchain

Reproducibility across environments

Command discovery

Troubleshooting

Principles

Shortening the critical path

If it’s impossible, does it make sense trying?

The bad parts

🟨 Waterfall development

🟨 Layered development

🟨 100% utilisation

🟨 Bus factor 1

The good parts

🟦 Self-organised and self-managed teams

🟦 Own your tools

🟦 Kaizen

🟦 “Stop starting, start finishing”

🟦 Is done really done?

Let it be

SSH config for ec2 machines

Golang 1.20 - multiple errors

Wrapping multiple errors

Caveats

error does not implement Unwrap() []error

errors.Unwrap() will return nil for multiple errors

Examples

Using errors

Formatting errors

Custom error

Remove yourself from the critical path

What’s the critical path?

Am I in the critical path?

Shadowing

Pairing

Proposal documents

Pitfalls

Not all tasks are the right candidate for delegation

Don’t setup people for failure

Be okay with people failing!

Be okay with having hard talks

Stop thinking about the effort, start thinking about the outcome

My note taking in 2022

1 Minute history of note taking

Linear note taking

Non-linear note taking

Welcome to the digital age

2022 ends

What’s next?

What I'm doing Now

Work

Open Source

Life

Making your shell scripts bulletproof pt.2

Don’t split words on space

Use traps

Use shellcheck

Stop formatting your code

Test BASH scripts with bats

Bring out the trash!

My home folder

Nix

Snaps

Golang string comparison benchmark

`error` does not implement `Unwrap() []error`

`errors.Unwrap()` will return `nil` for multiple errors

Using `error`s

Formatting `error`s

Test BASH scripts with `bats`