Luciano Mammino

Finally/First

Luciano Mammino — Thu, 16 Jan 2014 23:04:36 GMT

Yep! Finally here’s the first post on the path of this blog.

Finally/first, indeed a curious binomial with a weird sound, but I think it’s a good title to start this adventure. I’ll try to keep it simple and avoid to bloat things out… just start dancing by answering to a simple question, maybe the proper one now: Why start a blog? Well, I’m a web developer, fascinated by the web since I was a child and started writing my first lines of code in qbasic, and as any other serious web developer it happens to me to read tons of blogs to keep myself updated with the latest trends that pop out in this field. It also constantly happen to me to have the need to experiment something new in my free time. Summing it up I felt the need to build a space of mine in wich report my thoughts about web development and the result of some of these experiments.

I think that’s all for now. Stay tuned ;)

Symfony security: authentication made simple (well, maybe!)

Luciano Mammino — Fri, 14 Feb 2014 00:34:23 GMT

The Symfony2 security component has the fame of being one of the most complex in the framework. I tend to believe that’s partially true, not because the component is really that complex, but because there are (really) a lot of concepts involved and it may be difficult to understand them all at once and have a clear vision as a whole.

I am in the process of writing an OAuth bundle and I faced the need to deeply understand how the authentication process work in Symfony2 to being able to integrate the OAuth protocol with the Symfony authentication component. Notice that I know there are a lot of great and stable OAuth bundles out there (have a look at the great HWIOAuthBundle to which I had chance to submit small contributions in the past). My intention is not to build a better one, but I want to build my own bundle with some peculiar features and use it as a way to experiment and learn new things. But I am getting off topic!

Going back to the Symfony2 security component, the point is that I found out difficult at first glance to get a clear idea of what is going on behind the scenes and what I need to write to create a custom authentication mechanism.

So in this post I will try to collect few interesting resources that helped me understanding it better and a graph I drawn to resume what I learned.

Interesting links

I highly suggest you to read them in the proposed order.

Symfony Security component documentation: must read! The official documentation about the security component. It covers both authentication and authorization. Be warned that it will probably not make things clear at first, but I believe you must start your learning from there!
Diving into security layer: a great blog post from Francesco Tassi of Screenfony. It carefully describes which classes and interfaces are involved in the authentication layer and how they are wired togheter. He does not present any example of code, but in my opinion this is one of the most interesting and clarifying post about this topic.
Love and Loss: A Symfony Security Play: slides by the great Kris Wallsmith from the Symfony Live 2013 in Portland. Also a Video is available. That’s a great resource because Kris tries to explain with visual graphs how the various components are tied together and how the whole flow goes from request to response. Very clarifying!
Symfony2: Introduction to The Security Component Part I, Part II & Part III: a series of articles by Matthias Noback that cover the various aspect of the Symfony security (both about authentication and authorization). They have been merged to the official documentation, so read it fast as a good way to recap ideas.
Security in real life: A video from the Symfony Live Paris 2012 by Johannes Schmitt. Johannes explains some common problems people face with the Symfony security component. He also provides a great introduction describing the most important classes and how they are tied together.
Custom authentication provider: a cookbook article from the official documentation that presents a code example on how to write a custom authentication provider to integrate the WSSE protocol in Symfony. Great code example!
How to Authenticate Users with API Keys: another great cookbook article that will shed some light on the topic by providing a simple example.
How to Create a Custom Form Password Authenticator: yet another interesting cookbook article with a code example on how to modify the authentication logic behind the form (username/password) based authenticator.
Code of the RememberMe Listener: the code of the RememberMeListener class. A great and simple example on how to write a security authentication listener.

Symfony Authentication graph

Notice that the following graph tries to represent the classical schema available in the Symfony Standard Edition. As the framework is extensible at any given point you are free to change the whole logic. The generic idea is that you start with a Request object and you should check if it can be used to produce an AuthenticatedToken according to some authentication logic.

Get the image in full resolution on Google Drive (you can also propose edits with comments if you want) or Download the PDF.

That’s all, I hope this post has been useful. If you have interesting link related to this topic feel free to add them in the comment box and I will add them into the article.

3 invitations to try Atom.io

Luciano Mammino — Wed, 09 Apr 2014 10:58:22 GMT

I have finally got my invitation to try the new text editor Atom.io. Atom is built with web technologies (node.js in it’s core) by the folks at GitHub and it’s somewhat very similar to the famous SublimeText. In my opinion it seems very promising (even if it’s significantly slower than SublimeText), anyway I have to try it a lot more before shaping a serious opinion about it.

The point of this topic is that Atom.io is currently in closed beta and I have received 3 invitations to share with whom is willing to try it.

I would be glad to share the invitations with the first 3 readers who would follow me on GitHub or Twitter and write a comment on this post.

PS: You would probably need a GitHub account!

UPDATE, April 12: Sorry guys, my invitations have gone for now. Anyway it seems that GitHub distributes other invitations sometime. So if you haven’t been in time you can still write a comment and if I ever manage to have other invites i will follow the order of the order to share them with you.

Hope you will enjoy it.

Introducing ORM Cheatsheet

Luciano Mammino — Fri, 11 Apr 2014 08:50:53 GMT

This week I received an interesting email from Martin Stradej, the developer of ORM Designer, a tool to design ORM relationships with a simple but powerful graphic user interface.

Martin wrote the mail because he wanted to acknowledge me of his latest project: ORM Cheatsheet.

ORM Cheatsheet, as the name suggests, is nothing more than a reference website for those who struggles with some of the most common Php ORM libraries (it currently supports Doctrine2 and Doctrine, but it seems that Propel and Cake PHP will be supported too).

In my honest opinion, the great thing about the website is that it is a really useful resource if you always spend a lot of time searching the official Doctrine reference to check how the hell a given annotation is supposed to work or how to create a particular kind of relationship. If you are already confident with Doctrine and just can’t remember how to set up something it is the perfect place to go to freshen up your memory.

It also guides you on how to configure the Doctrine library to work with a standalone PHP project or even with a Symfony (both version 2 and 1.4) or a Zend Framework 2 based one.

The project has its own GitHub repository so everyone can submit a pull request and improve the project.

That’s all

Have a nice weekend

5 lessons learned at the Bank of Ireland Accelerator

Luciano Mammino — Mon, 22 Dec 2014 21:23:58 GMT

What a hell of 3 months!

During the last 3 months I haven’t got a single minute to take care of this blog. I started a new adventure and moved to Ireland to attend the Bank of Ireland Accelerator Programme in Cork with my startup Sbaam. There I spent 3 incredible months with a huge amount of lessons learned about startups and entrepreneurship. I want to share with you 5 of the best quotes I heard from the mentors and explain what I learned from them.

Consider this is my first “non technical” post with a lot of english written text. You’ll probalby find it plenty of mistakes, so feel free to send me an email and help me to improve it (and improve my english too :P)

Being an entrepreneur is hell

“Welcome… to hell!”

– Bill Liao (@liaonet)

This is the very first phrase I remember from a mentor. It was the opening day and Bill was one of the first speakers. He wanted to warn us about the upcoming months, but also about our choice of being entrepreneurs.

Yes, it may sound trivial but it’s true. Many think that being an entrepreneur means freedom, money and 0 responsibilities. The truth is that this is absolutely not the case!

Entrepreneurs work a lot more than the usual workers (and this very often includes nights, weekends and holidays). They have to care about their business in every single moment and every stakeholder ends up to be their boss: customers, investors and partners require a lot of attention, their money (and time) are not supposed to be wasted!

And, speaking about entrepreneurs’ earning, get ready to spend months, sometimes even years, without a single cent in your pocket especially if it’s your first time as an entrepreneur. Every penny you have should be invested in your business because at the end of the day you are the “number one” investor of your company.

Understand who you are, what you want to accomplish but don’t forget about your customers

“Nail it and then scale it! Find your first 100 happy customers and really understand them”

– Sean O’Sullivan (@sosventures)

Sean explained us why most of the businesses fail: they simply are not focused in trying to solve a problem for their customers! It’s crucial to understand what you are doing and what your customers need. You shouldn’t really keep adding 10 thousand features to your product or trying to attack every possible market and regions from day one.

Start small. Find a niche and try to grasp what’s the real pain there. Then see if your solution solves the problem for real and see if people are willing to pay for it. If that’s the case you found your first customers, well done! They are your most valuable asset, so you have to be sure they are really happy with you and that your solution is the best for them before trying to move on to a broader audience.

Passion breaks the point!

“You need to be totally and utterly in love with your business as it keeps you alive and your business itself alive”

– Daniel Ramamoorthy (@mynameisdanram)

Daniel has an incredible attitude of life and he is a surprisingly positive person. This quality is not something he is born with, but something that he developed in time and something that everyone can develop.

Great success can be achieved only if you work hard on staying positive and if you are totally passionate about what you are doing. You will spend so much time and energies on your business that you can really perform only if this motivates you and makes you feel good.

There will be times when everything will go wrong and it will be almost impossible to remain positive. These are the moments in which you should find your way to keep going and recover your positivity. Going out for a walk, practicing some sport, reading a book, watching a movie, listen to your favourite music, talking with a friend are just few of the thousand possible solutions. Everyone is different and the key here is to find out what’s the best way for you to release the stress and be ready to start again.

Daniel told us that he printed his favourite quote on the ceiling over his bed. Everytime he wakes up in the morning this is the first thing he see and this allows him to remember who he is and what he is passionate about, this makes him ready to do his best during the day.

Surround yourself with great people

“You are the average of the five people you spend most time with”

– Seàn Gallagher (@seangallagher1)

Nobody is alone. Unless you are an hermit in the Himalayas your days will be spent with different people from work to family and friends. The fact is that every one can give you something, so the trick is to surround yourself with the best people you can find. People that can teach you how to be better at something, people that can make you feel positive, people that can transmit their experience to you, people that can compensate your weaknesses. If you can do this you will surely became a better person day by day and will live happier.

This obviously applies also to your business: team, team, team! If you are an entrepreneur you will spend most of your days with your team so you have to build the best team in the whole World. Not necessarily the best in terms of individual skills but the best in terms of people and the synergy with them.

Just imagine what can happen to your business if your core team is not the right one…

You should work really hard if you want to look as good as a swan!

“You need to be as smooth as a graceful swan on a lake but you must keep pedalling very hard under the surface to maintain that forward momentum”

– Wayne Murphy, programme manager (@waynemurphy77)

This is really a great metaphor. I simply loved it. I have two different interpretation of it.

The first is that you have to look great to every external observer, almost pretending that everything is possible for you. But being able to do so requires a huge and continuous effort, you can’t fake it, you have to earn it!

My second interpretation is that once you start having some traction and you are floating smoothly on your way you have to start working even harder to keep the momentum and reach the next great milestone. So don’t make the mistake to relax at the first goal reached, but immediately look at the next one and do what’s needed to reach it, immediately!

Conclusion

These three months have been probably the best of my life in terms of learning and experience, so I have to say a huge “thank you” to everyone that made this possible.

Secondly i’d like to have your opinion and interpretation of all this quotes. Feel free to leave a comment and tell me which one of these five lessons you liked most.

New PHP library: PHPoAuthUserData

Luciano Mammino — Mon, 10 Feb 2014 12:13:50 GMT

I recently wrote a new PHP library to simplify the extraction of user data (name, email, id, etc…) from various OAuth providers such as Facebook, Twitter and Linkedin.

Is well know that OAuth 1 and 2 are great standard protocols to authenticate users in our apps. Anyway we often need to go further the authentication process and extract various information about the authenticated users. Unfortunately this is something that is not standardized and obviously each OAuth provider manages user data in very specific manner according to its specific purposes.

So each OAuth provider offer a set of APIs with specific data schemes to allow developers to extract data about the authenticated users.

That’s not a big deal if we build apps that adopts a single OAuth provider, but, if we want to adopt more of them, you need to deal with several different APIs and data schemes! Yes, things can get really cumbersome!

Just to make things clearer suppose you want to allow users in your app to sign up with Facebook, Twitter and Linkedin. Probably, to increase conversion rate and speed up the sign up process, you may want to populate the user profile on your app by copying data from the OAuth provider user profile he used to sign up. Yes, you have to deal with 3 different sets of APIs and data schemes! And suppose you would be able to add GitHub and Google one day, that will count for 5 different APIs and data schemes… not so maintainable, isn’t it?

The library I wrote is called PHPoAuthUserData. It sits on top of the excellent OAuth library Lusitanian/PHPoAuthLib and aims to resolve the user extraction data problem in the most simple and effective way.

It offers a uniform and (really) simple interface to extract the most interesting and common user data such as Name, Username, Id and so on.

Just to give you a quick idea of what is possible with the library have a look at the following snippet:

// $service is an istance of \OAuth\Common\Service\ServiceInterface (eg. the "Facebook" service) with a valid access token

$extractorFactory = new \OAuth\UserData\ExtractorFactory();
$extractor = $extractorFactory->get($service); // get the extractor for the given service
echo $extractor->getUniqueId(); // prints out the unique id of the user
echo $extractor->getUsername(); // prints out the username of the user
echo $extractor->getImageUrl(); // prints out the url of the user profile image

The code is available on Github where you will find detailed information on how to install and use the library.

I Hope you will enjoy it and be willing to contribute to the code base. If you want to know more, read the next post that explains how to write an extractor for the library.

Integrating Twig.js and BazingaJsTranslationBundle

Luciano Mammino — Fri, 28 Feb 2014 11:40:05 GMT

Recently I had the need to run a twig template that uses the trans filter on my frontend using twig.js, a pure JavaScript port of twig written by the good Johannes Schmitt. The JavaScript version does not handle all the functionalities offered by the original PHP version (even if it goes pretty close) and in particular it does not natively handle the trans filter.

So, at first, I got a JavaScript runtime exception on my page when trying to use the template. Luckily enough the JavaScript version of twig is extensible like the PHP one and it is very easy to add new filters and functions.

In my specific case I had a Symfony application where I was already using BazingaJsTranslationBundle to manage dynamic translations on the frontend. As I discovered the twig.js extensibility, it was very easy to start building a twig.js extension by using the Translator JavaScript object offered by the Bazinga bundle.

Note: I will not go into the details about how to install twig.js and BazingaJsTranslationBundle in a Symfony application. You can find all the needed informations on their websites/github pages.

In my first attempt I wrote something like this:

Twig.setFilter('trans', function(id, params, domain, locale) {
  return Translator.trans(id, params, domain, locale)
})

That seemed to work pretty good until I started to use translation strings with parameters. Parameters were not replaced with their respective values! The problem laid in a subtle differece on how the BazingaJsTranslationBundle and the standard twig handle parameters. Let’s see a simple example.

Suppose we have the string hello %name%. With twig we expect to do something like:

{{ 'hello %name%'|trans({ '%name%' : 'Alice' }) }}

Note the % delimiters around the parameter name.

The Translator.trans method expects an hash map without parameter delimiters in it. So we would have to do something like this:

Translator.trans("hello %name%", { 'name' ; 'Alice' });

Note that there’s no % delimiter this time. The Translator.trans method manages the detection of parameters by itself and you can also decide to customize the delimiters by setting the values: Translator.placeHolderPrefix and Translator.placeHolderSuffix. Obviously I suggest you to be consistent and use the same placeholders you use with PHP (especially if you need to share templates and translations from the backend to the frontend).

So my final solution was the following:

Twig.setFilter('trans', function(id, params, domain, locale) {
  params = params || {}

  // normalizes params (removes placeholder prefixes and suffixes)
  for (var key in params) {
    if (
      params.hasOwnProperty(key) &&
      key[0] == Translator.placeHolderPrefix &&
      key[key.length - 1] == Translator.placeHolderSuffix
    ) {
      params[key.substr(1, key.length - 2)] = params[key]
      delete params[key]
    }
  }

  return Translator.trans(id, params, domain, locale)
})

This way it automatically normalizes parameters for the Translator object (by removing any delimiter from parameter keys) and I have a consistent behavior between twig and twig.js. My normalization approach is very rough and you can surely find a better approach (maybe using a regex). Let me know if you do it ;)

Obviously you can also avoid the normalization and keep the responsibility to pass the parameters hash map in the way the Translator object expects it (without delimiters). In this case you can stick to my first implementation.

That’s all. See ya ;)

Transparent pixel response with Symfony, how to track email opening

Luciano Mammino — Fri, 13 Jun 2014 10:33:04 GMT

If you have ever heard about “transparent pixel”, “1x1 blank pixel”, “tracking pixel” or simply “tracking image” you probably know what we are going to talk about and can just skip to the implementation!

Introduction

Transparent pixel image is a technique often used to track some user behavior (often visits or views to a certain online content) in scenarios where you can’t use javascript.

One of the most common scenarios is probably email opening tracking. Can you put Google Analytics into an email? Well, probably it would not work as expected… So there’s something we can do? Of course there is, let’s jump into a concrete example.

Track email opening

The following image shows the typical tracking flow applied to emails:

A user receives our HTML email. Within the email content there’s a “smart” tracking image: <img src="http://example.com/track.gif?id=1234">. Notice that it points to our server example.com and has a parameter id=1234.
When the user opens the email, his email client will start to download all the resources linked into the HTML code (usually images) and it will trigger a request to download the tracking image.
The request is handled by the example.com webserver. It does not handle the request as a static image but it executes some logic. It checks the id parameter and uses it to determine which email has triggered the request. Then it marks that email as opened in its own database for future reports. The mail client is still waiting for an answer and it expects an image. So the webserver generates on the fly the most small image possible: a 1x1 transparent image!
Then the image is sent back to the client that will render it on the screen. Anyway the image is trasparent and so small that the user will barely notice it.

Symfony implementation

Now let’s see how to implement this tracking flow using the Symfony framework.

Notice that you should have your own logic to generate emails, to store and send them. I will not cover these parts, but only the one related to the tracking flow explained before.

First of all, we want to have a dedicated Response class to send back a transparent pixel to the client, let’s call it TransparentPixelResponse:

<?php

namespace LMammino\Http;

use Symfony\Component\HttpFoundation\Response;

/**
 * Class TransparentPixelResponse
 * @package LMammino\Http
 * @author Luciano Mammino <lucianomammino@gmail.com>
 */
class TransparentPixelResponse extends Response
{
    /**
     * Base 64 encoded contents for 1px transparent gif and png
     * @var string
     */
    const IMAGE_CONTENT =
        'R0lGODlhAQABAJAAAP8AAAAAACH5BAUQAAAALAAAAAABAAEAAAICBAEAOw=='
    ;

    /**
     * The response content type
     * @var string
     */
    const CONTENT_TYPE = 'image/gif';

    /**
     * Constructor
     */
    public function __construct()
    {
        $content = base64_decode(self::IMAGE_CONTENT);
        parent::__construct($content);
        $this->headers->set('Content-Type', self::CONTENT_TYPE);
        $this->setPrivate();
        $this->headers->addCacheControlDirective('no-cache', true);
        $this->headers->addCacheControlDirective('must-revalidate', true);
    }
}

The logic is very simple here. We have hardcoded the content of a 1x1 transparent gif image using a base64 encoded string. We use this string to set the content of the response object. We also set some cache headers to mark the response as not cacheable.

Now we can write a controller that will handle the tracking request:

<?php

use Symfony\Bundle\FrameworkBundle\Controller\Controller;
use Symfony\Component\HttpFoundation\Request;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;

use LMammino\Http\TransparentPixelResponse;

class TrackingController extends Controller
{
  /**
     * @Route('/track.gif')
     */
    public function trackEmailAction(Request $request)
    {
      $id = $request->query->get('id');
        if (null !== $id) {
            //... executes some logic to retrieve the email and mark it as opened
        }
        return new TransparentPixelResponse();
    }
}

Here the logic is pretty simple too. We created a controller with a trackEmail action. The action has been assigned to the route /track.gif using the Route annotation (if you prefer you can do it also by using the yaml or the xml convention). Within this action we just read the parameter id from the request and used it to execute the persistence logic to retrive the email record and mark it as opened (skipped in the example). Then we just have to return a new instance of our TransparentPixelResponse class.

That’s it! Pretty simple, isn’t it?

###Improve performance

UPDATE 2014-06-16: A comment from Lumbendil pointed out that it would be possible to do the “heavy logic” within a kernel.terminate event listener. So let’s refactor our code to follow this good suggestion:

<?php

use Symfony\Bundle\FrameworkBundle\Controller\Controller;
use Symfony\Component\HttpKernel\Event\KernelEvent;
use Symfony\Component\HttpKernel\KernelEvents;
use Symfony\Component\HttpFoundation\Request;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;

use LMammino\Http\TransparentPixelResponse;

class TrackingController extends Controller
{
  /**
     * @Route('/track.gif')
     */
    public function trackEmailAction(Request $request)
    {
      $id = $request->query->get('id');
        if (null !== $id) {
          $dispatcher = $this->get('event_dispatcher');
            $dispatcher->addListener(KernelEvents::TERMINATE,
                function(KernelEvent $event) use ($id){
                   //... executes some logic to retrieve
                   // the email and mark it as opened
                }
            );
        }
        return new TransparentPixelResponse();
    }
}

Notice that we have “wrapped” our potentially heavy logic within a callable function that gets executed when the kernel.terminate event is fired. This way the response is returned immediatly (before executing all the heavy logic) and the requesting client will not have to wait for processing.

Obviously, from the point of view of the server, we are not “really” improving performance. The code is not executed faster, but only in a different order. There’s only an apparent performance improvement for the web client who receives the response quicker and doesn’t care about the processing logic that will keep running on the server side.

Keep in mind that the kernel.terminate event is optional, and should only be called if your kernel implements TerminableInterface (it should work if you are using the Symfony Standard Edition).

Lumbendil also pointed out that this solution is not the only one possible. You can also rely on some messaging/queue system such as RabbitMq, Gearman or Beanstalkd. These are great tools but they add new dependencies and a whole new layer of complexity to the web infrastructure, so I will suggest to use one of them only if your logic is very complex or heavy (or if you are designing you whole infrastructure from scratch to leverage a work queue system).

Considerations

You can apply this method to track email opening or views on other online contents such as Html webpages (in cases where you can’t or don’t want to use javascript). Anyway, regarding tracking email opening, you should be aware that some e-mail clients block images loading when you open an e-mail from an unknown sender. Yes, they do it not only to save your bandwidth, but mostly to avoid triggering tracking images! So in this cases you will not able to track the email opening, unless the user authorizes its client to load all the images in that specific email.

UPDATE 2014-06-17: Jelte Steijaert reported that using email authentication systems such as DKIM or SPF will increase chances for images to get autoloaded by email clients. This authentication methods are also very useful to save your emails from ending up into the spam folder, so they are very recommended!

If you have some other consideration you are very welcome to write a comment, as always!

Introducing flickr-set-get a command line app to download photos

Luciano Mammino — Mon, 11 May 2015 23:04:50 GMT

I recently developed a small command line app that allows you to download an entire gallery from Flickr, it’s called flickr-set-get and you can find it on NPM and GitHub.

Why?

To be honest I had myself the need to download a large set of photos (more than 400 photos) from Flickr and I didn’t wanted to do it manually. I also wasn’t able, after a quick search, to find something simple to solve this task. Given that I am currently getting into deep of Node.js this was the perfect chance to develop something practical.

How it’s built?

As I said it’s built using Node.js. As part of my learning process, I tried to not reinvent the wheel and to follow “the Node way” so I used some very good modules as the foundation to build this app:

Commander, Cli and Prompt for the command line facilities (Options/Arguments parsing, help generation, progress bar, receiving input from the user)
Request: to simplify the creation of the Http request to call the Flickr APIs
Async: to manage all the asynchronous tasks in parallel and with a configurable level of concurrency (one of the features that in my humble opinion makes node/javascript stand out from the crowd of the interpreted programming languages).

As development tools I also used:

JSCS: to check the code standard I adopted
Mocha and Chai: test runner and assertion frameworks for the unit testing
Nock: amazing module to “mock” the Http requests and avoid to hit Flickr servers in my tests
Istanbul: for the code coverage
Travis CI: continous integration service
Coveralls: service to keep track of the coverage changes after every new commit
Gemnasium: service that checks your dependencies and alerts you if they get out of date

Going asynchronous and execute requests in parallel

To download a photo from Flickr, starting from a gallery, you need to make several API calls before you have the right URL to download the photo.

First of all you need to call the flickr.photosets.getPhotos API method to find out the IDs of all the photos in the set. Then, to find out the URL of every photo you need to call the flickr.photos.getSizes API method. This method gives you all the URLs to download all the different sizes of a given photo. Once you got the URL you can simply use that to download a photo into your local drive. The first API call is the starting point and then, since you now have all the IDs, you can make all the other requests in parallel. Thanks to Async it was very easy to create a task function that figured out the specific URL of a given photo and then downloads it into a file. Then I just needed to replicate this task for every photo and run all of them in parallel. Thanks to the Async.parallelLimit() function it’s even possible to run the tasks with a configurable concurrency level.

You can find out how I implemented this specific logic on GitHub: https://github.com/lmammino/flickr-set-get/blob/ecaf0d8bea5791cf24c4b71791a7eb1fd2e60bcb/lib/Flickr.js#L212-L253.

How to use it?

If you’re interested on using it I suggest you to read the official documentation on GitHub, but it should be quite easy to understand and use it if you are a developer (and it’s not meant to be used by other people who are comfortable with the command line!)

Current status

The project has been developed in a weekend as a “learning project” and tested just by me on few Flickr galleries. It’s to be considered ALPHA quality and I expect it to be fully of bugs and uncovered edge cases. It’s a small side project anyway so I don’t expect to put so much effort on it again unless it gets adopted by a relevant number of people (which, quite frankly, is really unlikely to happen). Being an open source project I obviously tried to put in place all the basic infrastructure (tests, code standards, repository, versioning, etc.) to make it easy for other people to work on improving it. Again, more precise informations are included in the official documentation.

Opinions

For me it has been a really good learning experience, especially considering the fact that it was a good scenario to use asynchronous code dealing with several parallel Http requests. As I still consider it a learning experience I would really love you (probably more experienced than me with Node.js) to give some opinions, critics, new ideas. I offer the comment box below for this sake… please don’t be too much indulgent :D

Reset your MySql server password

Luciano Mammino — Sun, 30 Mar 2014 09:07:56 GMT

Few days ago I learned an interesting trick that allows you to reset the password on a MySql server. This trick is pretty useful in case you have lost (or you haven’t ever had) the password of your MySql root user.

You just need to throw some commands at the console. Note that I am assuming you are using Ubuntu or some other Debian derivate. Otherwise the commands may change a bit (but not to much I believe).

1. Stop the mysql daemon

sudo /etc/init.d/mysql stopRun

2. Disable the security checks

mysqld_safe --skip-grant-tables &

The mysqld_safe command will essentially restarts your MySql server but with the option --skip-grant-tables it also disables the grant tables used for authentication. You might think that this way the MySql server will deny every access attempt, but it will do the very contrary: it will allow any! That’s in fact where the trick lies: this way we are able to login to MySql as root (even if we don’t know the password) and edit the MySql users table to reset the root user password.

Yeah, I didn’t think this was possible before discovering it. But it is, and, even if it feels “insecure”, it might be useful in cases like this.

mysql -u root -p mysql

It will ask you for a password… Yes, you get it! You can enter whatever you want and it will always grant you access as root! The last argument (mysql) specifies you want to access the MySql internal database (where user credentials are stored).

4. Change the root password

You’re using the MySql shell now. Just run:

update user set password=PASSWORD("NEW-ROOT-PASSWORD") where User="root";
exit;

You need to change the "NEW-ROOT-PASSWORD" with a password of your choice (obviously).

5. Restart MySql

You’re back in your bash shell:

sudo service mysql restart

This will restart the MySql server and will enable the security checks again.

So that’s all! You can now login into your MySql server with the new password! And try to not lose it again, at least for a while ;)

Having fun with Node.js, Slack and Chuck Norris

Luciano Mammino — Mon, 14 Sep 2015 22:05:30 GMT

I recently wrote a Slack bot in Node.js in collaboration with one of my favourite web development websites: the amazing Scotch.io.

It’s a simple and funny experiment I have done in my free time to learn something new. It’s nothing special but it will add a bit of fun inside your Slack organisation… Who doesn’t want to have a bot that tells random jokes about Chuck Norris in his own Company chat room? :)

The bot is available on Github (lmammino/norrisbot) and on NPM (norrisbot) and it can be easily installed configured and installed on every VPS and even on Heroku (using the free plan!).

If you are interested in knowing how it works, in understanding the code and in learning how to deploy it on Heroku, you should definitely read the article we published on Scotch.io:

Building a Slack Bot with Node.js and Chuck Norris Super Powers

Enjoy it and feel free to leave your comments here or on the article.

Cheers!

8 invitations to try Keybase.io

Luciano Mammino — Tue, 26 May 2015 22:25:45 GMT

Hello dear developers, today I finally got my invitation to access the private beta of Keybase.io! 😎 🎉 For those who are not aware of this new service, Keybase.io is a website and a command line application that makes easier to adopt asymmetric cryptography to encrypt and authenticate messages. But it’s not just this, it’s also a “directory of people”, as they like to say. It’s like a social network, where everyone can easily share its public key and attach it to its social profiles (like Twitter, Github or Reddit). You can also pair your bitcoin address with it…

…and furthermore the website is full of wonderful illustrations a-la-Dropbox :D

A quick demo

Just to give you a practical idea of how it works, let’s suppose I want to send a super-secret message to my twitter friend Daniel Li (@d4nyll). Once I have the Keybase command line app installed (and configured) I can just run:

$ keybase encrypt twitter://d4nyll -m "Hey pal, I finally crafted the perfect plan to rule this World. Give me a ring-3"

and it will print out a weird message that only Daniel, with his private key will be able to decrypt:

Now I just need to copy and paste this message to Daniel through Skype, an email, Slack or whatever.

Daniel will be able to easily decrypt the message through the same command line app or though the Keybase website.

The clear advantage is that I don’t need to go around and find a person public key every time I want to send an encrypted message to someone. With Keybase I just have to remember his username from Twitter, Github or whatever.

It’s still a little bit boring that I have to transfer my message manually, but the Keybase team has already developed a set of APIs that, I guess, makes possible to fill the gap in terms of integration with other messaging apps.

The command line application has a lot of other handy features like messages certification and verification (signature) and in case you dislike the command line (which is unlikely if you are reading this blog), you can do pretty much everything directly on the website.

Do you want to give it a try?

As you probably have guessed from the title of this post and the first paragraph Keybase is still in private beta and you need an invite to use it. If you haven’t already received your I have 8 invite code that I am willing to share. They go to the first 8 persons who will leave a comment* here 😉

Update (27 may 2015 12:20 GMT): Invites are over :(
I will try to request new ones. If you want you can keep commenting and I will give new eventual invites in the order of entry.

What are you waiting for?

PS: In case you want to “Track” (verify+follow) me on Keybase, check out my Keybase profile.

Symfony, edit the Response globally using the Kernel Response event

Luciano Mammino — Sat, 21 Feb 2015 01:12:00 GMT

One of the things I like most of the Symfony framework is its Http Kernel component. Not only it does offer a very straightforward abstraction to handle requests and responses in an object-oriented way but it also allows you to interact with the whole response generation process through events.

This approach is very convenient and flexible and in fact the Http Kernel component is the foundation of the Symfony framework but also of several other famous frameworks (Silex, Laravel) and CMSes (Drupal, BackBee CMS).

The Kernel Response event

One event that I’ve been using a lot lately is the Kernel Response event which allows you to edit the response after it has been generated.

Thanks to this event you can easily modify the response object (cookies, headers, content, etc.) before it gets sent out to the user without affecting the specific logic of every controller thus avoiding code cluttering and duplication.

I will present two different real case scenarios to show how useful (and simple) it is.

Example 1. Add custom Http headers to notify remaining api calls

Let’s suppose we developed a wonderful rate limited api and now we want to add some custom headers to notify the user about how much he is using the API. It seems a good idea to copy the approach adopted by the GitHub APIs and add three custome headers: X-RateLimit-Limit (maximum number of requests per period), X-RateLimit-Remaining (remaining requests in the current period) and X-RateLimit-Reset (the timestamp on which the current period ends).

As I don’t want to implement a fully working solution here let’s assume we have already written a rate limit checker service registered as rate_limit_checker that implements the following interface:

interface RateLimitCheckerInterface
{
  public function getRateLimit();

  public function getRateLimitRemaining();

    public function getRateLimitReset();
}

Let’s now write our RateLimitHeadersListener:

use Symfony\Component\HttpKernel\Event\FilterResponseEvent;

final class RateLimitHeadersListener
{
  private $rateLimitChecker;

  public function __construct(
      RateLimitCheckerInterface $rateLimitChecker
    ){
      $this->rateLimitChecker = $rateLimitChecker;
    }

    public function onKernelResponse(FilterResponseEvent $event)
    {
        $headers = $event->getResponse()->headers;

        $headers->set(
          'X-RateLimit-Limit',
            $this->rateLimitChecker->getRateLimit()
        );

        $headers->set(
          'X-RateLimit-Remaining',
            $this->rateLimitChecker->getRateLimitRemaining()
        );

        $headers->set(
          'X-RateLimit-Reset',
            $this->rateLimitChecker->getRateLimitReset()
        );
    }
}

Now we need to register the listener as a tagged service:

rate_limit_listener:
  class: RateLimitHeadersListener
  arguments: ['@rate_limit_checker']
  tags:
    - {
        name: kernel.event_listener,
        event: kernel.response,
        method: onKernelResponse,
      }

That’s it. Really straightforward, isn’t it? Should be clear now that, by using this event based approach, we don’t have to touch the logic of every single controller.

Take a small break and get ready to jump to another example.

Well, now imagine that we have to build an affiliate program based on referral links (I did it lately).

The general idea is that our affiliates are identified by an ID that they can attach to every url of the website as query parameter. This way every URL of our website can be an entry point for our visitors and our affiliates are free to promote the content that is more relevant for them. Doing so we need to verify every possibile request to check for the referrral parameter and keep track of the whole session of the visitor (or even better monitor him for a given amount of days) to see if his visits converts into some kind of action for which we have to reward the affiliate. To write a more formal specification we have to:

Allow any of our affiliates to share links with a special referral code as query parameter: ?_ref=<REF_ID> (where REF_ID is the unique id of the affiliate).
Intercept visitor referred by affiliates through the referral parameter and identify the referral
Create a cookie to track the referred visitor for 30 days

Again we can use the Kernel Response Event and create a dedicated listener for this task. Before doing so suppose we have already developed a mechanism to store our affiliates and that we have coded a affiliate_repository service which implementats of the following interface:

interface AffiliateRepositoryInterface
{
  public function findOneById($id);
}

Now let’s write our listener to intercept clicks on referral links:

final class AffiliateLinkClickListener
{
  private $affiliateRepository;

    public function __construct(
      AffiliateRepositoryInterface $affiliateRepository
    ){
      $this->affiliateRepository = $affiliateRepository;
    }

    public function onKernelResponse(FilterResponseEvent $event)
    {
      $request = $event->getRequest();
        $response = $event->getResponse();

        // 1.
        if ($request->query->has('_ref')) {
          $affiliateId = $request->query->get('_ref');
            // 2.
            if (null !== $this->affiliateRepository->findOneById($affiliateId)) {
              // 3.
                $cookie = new Cookie('_ref', $affiliateId, new \DateTime('+30 days'));
                $response->headers->setCookie($cookie);
            }
        }
    }
}

The code is pretty simple here:

We check if there’s a _ref parameter in the current request
If so we check if we have an affiliate with the id found in the _ref parameter
If that’s the case we create a cookie that will allow us to keep track of the referral for 30 days.

Obviously don’t forget to register the listener as a tagged service:

affiliate_link_click_listener:
  class: AffiliateLinkClickListener
  arguments: ['@affiliate_repository']
  tags:
    - {
        name: kernel.event_listener,
        event: kernel.response,
        method: onKernelResponse,
      }

As I said, this mechanisms allows you to track the reference but you also need to track any conversion and check the cookie to see if it has been generated by some affiliate.

Conclusion

As in every tutorial, bear in mind that the code is just a sample to give you a general idea about an approach you can follow to solve some generic problem. You need to adapt it to your use case and consider relevant factors such as security, testing, etc.

Let me know what you think with a comment (comments) and feel free to suggest other real-life use cases for the Kernel Response Event.

Thanks!

PS: Huge thanks to @AxelLessio and @JavierEguiluz for taking the time to review my very bad english ;)

Writing a new Extractor for PHPoAuthUserData

Luciano Mammino — Mon, 10 Feb 2014 17:23:54 GMT

In my previous post I introduced my latest library PHPoAuthUserData that allows to abstract the process of extracting user profile data from various OAuth providers (Facebook, Twitter, Linkedin, etc).

The library still need a lot of work, especially to write the logic to extract data from all the services available in the parent OAuth library.

At the moment the library supports only the most common OAuth providers. I would be glad to support all the following services:

Amazon, BitBucket, BitLy, Box, Dailymotion, Dropbox, Etsy, FitBit, Flickr, Foursquare, GitHub, Google, Heroku, Mailchimp, Microsoft, PayPal, Reddit, RunKeeper, SoundCloud, Tumblr, Vkontakte, Yammer.

So that’s a lot of work! And yes, of course I would be glad to share it with someone interested in using the library.

This article illustrates how to add support for a new service by writing a dedicate extractor class. It’s really simple so stick with me and you will be able to submit your pull request in minutes!

What is an extractor

Extractors defines the logic to request information to a given service API and to normalize the received data according to a common interface. The most basic way to define an extractor is to write a class that implements the ExtractorInterface (that is pretty self-explanatory).

You could extend the class Extractor that implements most of the needed code to get you started. Anyway, extractors should really extend the class LazyExtractor where possible because this class acts as a boilerplate to define highly optimized extractors. It easily allows you to implement extractors that lazy loads data (perform requests only when needed to) and caches data (does not make the same request more than once and avoids normalizing the same data more than once). Everything is done behind the scenes, so you’ll need to focus only on methods that define how to make requests and how to normalize data.

To understand how to write a new extractor by adopting the LazyExtractor we need to clarify some concepts:

Supported fields: an array of the fields that can be extracted (you should use field constants from the ExtractorInterface).
Loaders: methods whose responsibility is to trigger the proper request to the OAuth provider endpoint to load a specific set of raw data. Generally you need to define a loader for each block of information that could be retrieved from the endpoint. this methods must have the suffix Loader in their name. Most of the service will allow you to retrieve all the user data with a single request, so, in this cases, you would have only a single loader method (eg: profileLoader).
Normalizers: methods that accept raw data (the one previously fetched by some loader method) and uses it to extract the value for a given field. Usually you have a normalizer for each supported field. Normalizers methods must have the suffix Normalizer (eg. uniqueIdNormalizer or descriptionNormalizer).
LoadersMap: an array that associates supported fields (keys) to loaders methods (values). Loaders methods must be referenced without the Loader suffix. Most of the time, if you have only the profileLoader loader you will have an array with all fields mapping to the string profile.
NormalizersMap: an array that associates supported fields (keys) to the related normalizer methods (values). Normalizers methods must be referenced without the Normalizer suffix. It’s highly suggested to use the same name of the field for its related normalizer, so, most of the time, you will end up by having an array that maps field constants to the same field constant (eg. array(self::FIELD_UNIQUE_ID => self::FIELD_UNIQUE_ID)) for every supported field.

Once you defined Supported Fields, Loaders, Normalizers, Loaders Map and Normalizers Map from within your new extractor class you must wire them to the underlying logic by passing them to the parent constructor. So if you defined methods such as getSupportedField, getLoadersMap and getNormalizersMap you will end up with a constructor like this:

public function __construct()
{
    parent::__construct(
        self::getLoadersMap(),
        self::getNormalizersMap(),
        self::getSupportedFields()
    );
}

But let’s see how I built the Instagram extractor to have a better understanding on the whole process.

Writing the Instagram extractor

First of all I had a look on a bit of documentation to find out what kind of data can be extracted from Instagram users.

So I discovered that the request to retrieve information about the user is: /users/self and its response is a json object that looks like the following:

{
  "meta": {
    "code": 200
  },
  "data": {
    "username": "johnnydonny",
    "bio": "A life on the edge",
    "website": "http://blog.johnnydonny.com",
    "profile_picture": "http://images.ak.instagram.com/profiles/profile_weird_numbers.jpg",
    "full_name": "John Doe",
    "counts": {
      "media": 131,
      "followed_by": 80,
      "follows": 64
    },
    "id": "1111222333"
  }
}

So I understood wich fields can be mapped and started writing the Instagram class under the OAuth\UserData\Extractor namespace.

<?php

namespace OAuth\UserData\Extractor;

class Instagram extends LazyExtractor
{
  //...
}

First of all I wrote the method profileLoader and added a class constant that defines the url of the request.

const REQUEST_PROFILE = '/users/self';

protected function profileLoader()
{
  return json_decode($this->service->request(self::REQUEST_PROFILE), true);
}

Each extractor has access to the property $this->service that is an instance of the specific OAuth service from the parent library (OAuth\OAuth2\Service\Instagram) in this case. With this instance you can easily make request to the provider API endpoint.

Then I added the getSupportedFields method:

protected static function getSupportedFields()
{
  return array(
    self::FIELD_UNIQUE_ID,
    self::FIELD_USERNAME,
    self::FIELD_FULL_NAME,
    self::FIELD_FIRST_NAME,
    self::FIELD_LAST_NAME,
    self::FIELD_DESCRIPTION,
    self::FIELD_WEBSITES,
    self::FIELD_IMAGE_URL,
    self::FIELD_PROFILE_URL,
    self::FIELD_EXTRA
  );
}

The fields first_name, last_name and profile_url are not directly available on the json response but are easy to reconstruct by using the full_name and username fields.

Than I started writing all the normalizer methods to map the raw data to the respective supported fields:

protected function uniqueIdNormalizer($data)
{
  return isset($data['data']['id']) ? $data['data']['id'] : null;
}

protected function usernameNormalizer($data)
{
  return isset($data['data']['username']) ? $data['data']['username'] : null;
}

protected function fullNameNormalizer($data)
{
  return isset($data['data']['full_name']) ? $data['data']['full_name'] : null;
}

protected function firstNameNormalizer()
{
  $fullName = $this->getField(self::FIELD_FULL_NAME);
  if ($fullName) {
    $names = explode(' ', $fullName);

    return $names[0];
  }

  return null;
}

protected function lastNameNormalizer()
{
  $fullName = $this->getField(self::FIELD_FULL_NAME);
  if ($fullName) {
    $names = explode(' ', $fullName);

      return $names[sizeof($names) - 1];
  }

  return null;
}

protected function descriptionNormalizer($data)
{
  return isset($data['data']['bio']) ? $data['data']['bio'] : null;
}

protected function websitesNormalizer($data)
{
  $websites = array();
  if (isset($data['data']['website'])) {
    $websites[] = $data['data']['website'];
  }

  return $websites;
}

protected function profileUrlNormalizer()
{
  $username = $this->getField(self::FIELD_USERNAME);

  if (null !== $username) {
    return sprintf('http://instagram.com/%s', $username);
  }

  return null;
}

protected function imageUrlNormalizer($data)
{
  return isset($data['data']['profile_picture']) ? $data['data']['profile_picture'] : null;
}

protected function extraNormalizer($data)
{
  return ArrayUtils::removeKeys($data['data'], array(
            'id',
            'username',
            'full_name',
            'website',
            'profile_picture',
            'bio',
  ));
}

Notice that each normalizer must return null if the field is not defined. That’s a best pratice to follow for safety.

Also notice that the extraNormalizer method has the purpose to keep track of all the fields that could not be mapped to the ExtractorInterface fields. So we use the OAuth\UserData\Utils\ArrayUtils::removeKeys method to simply remove already mapped data.

Finally we need to wire our profileLoader method and all our normalizers methods in the constructor:

public function __construct()
{
  parent::__construct(
    self::getDefaultLoadersMap(),
    self::getDefaultNormalizersMap(),
    self::getSupportedFields()
  );
}

The methods self::getDefaultLoadersMap and self::getDefaultNormalizersMap are convenience methods defined in the LazyExtractor class that defines respectively a loaders map and a normalizers map that are good in most of the cases.

That’s all. To see the complete class have a look here.

Important note: if you are willing to submit a pull request to integrate a new extractor be sure to follow PSR-2 code style and to add a dedicated test case. Have a look at the InstagramTest class to understand how to do it.

6 Rules of thumb to build blazing fast web server applications

Luciano Mammino — Sat, 25 Jul 2015 18:57:00 GMT

In this post I will try to highlight some of the most common principles that you have to take under consideration when you want to achieve a great level of performance while building a web application (specifically on the backend part). I believe the concepts discussed here can be applied to any language and framework even if, due to my specific experience, I will mention some examples, design patterns, conventions and tools that are mostly used in the PHP world.

You can read this article in other languages:

Japanese on POSTD: 高速な Web サーバアプリケーションを構築するための 6 つの経験則
Italian on HTML5 Today: Performance: 6 regole per sviluppare applicazioni web

TLDR; the basic rules are:

Rule 1. Avoid premature optimization
Rule 2. Do the minimum amount of work to solve the problem
Rule 3. Defer the work you don’t need to do immediately
Rule 4. Use cache when you can
Rule 5. Understand and avoid the N+1 query problem with relational databases
Rule 6. Prepare your app for horizontal scalability when possible

Rule 1: Avoid premature optimization

One of the most famous Donald Knuth’s quotes says:

“premature optimization is the root of all evil”

Tweet this

Knuth’s noticed that a lot of software developers generally waste a huge amount of time thinking about performance of non-critical parts of the code they are writing. This very often happens because such developers don’t really know what are the critical parts of their code or the ones that need to be optimized more, so they start to worry about futile things like “are double quoted strings slower than single quoted ones?”

To avoid to fall into the premature optimization trap you should write the first version of your code without worrying much about performance. Then you can use a profiler to instrument your code and see where the bottlenecks are. This way you can focus on improving only the parts that really need your attention.

Note: I want to make it clear that Knuth’s quote doesn’t mean that you don’t have to care about optimization at all and it’s not an excuse to write shitty code and then abandon it. I just intended it as an encouragement to learn how to “optimize smartly” and that’s the way you should read it as well.

If you are working on the PHP land there are a loot of tools that you can easily adopt to profile your code:

xdebug: probably the most famous PHP debugger and profiler, it must be installed as a PHP extension and it’s easily integrable in most of the IDEs.
xhprof: a function-level hierarchical profiler for PHP. It comes with a simple HTML based navigational interface and offers some cool diff capabilities to compare the performance of different versions of your code.
Symfony profiler: The Symfony profiler it’s a one of the best features of the Symfony framework. It allows you to inspect the execution time of every request, showcasing a nice timeline that allows you to easily understand which part of your code is the most time-consuming. It is automatically enabled in “development” mode and does not need any PHP extension to be installed.
The Stopwatch component: It’s the low level library used in the Symfony profiler to measure the execution time of a piece of PHP code. It can be easily integrated in any PHP project and does not require any extension.
Blackfire.io: a profiler optimized for PHP that offers a very nice web interface that allows you to understand visually what your code does and where the CPU spends most of its time.
Tideways: a promising alternative to Blackfire, offers a lot of graphical tools (timeline, call graphs, etc.) to make it really easy to find bottlenecks. It’s meant to be run continuously (also in production).

If you want to know more on this specific topic you can have a read at the following articles and papers:

On optimization in PHP by Anthony Ferrara
The fallacy of premature optimization by Randall Hyde
Premature optimization by Cunningham & Cunningham, Inc

Rule 2: Do just what you need to

Very often your code does more things than it’s required to do to produce the expected output. This is especially true if you are using complex libraries and frameworks in your code. Just to give you some examples you might load classes that you’ll never use, you might open a database connection or read a file for every request even when these resources are not needed to generate the output for a specific request.

There are a number of design patterns and techniques that can help you to avoid these situations and achieve better performances.

Autoloading: it’s a PHP feature that allows you to require the file containing the definition of a class only when you are about to use that class (instantiation, static method call, access to a constant, etc.). This way you should not worry about which files to include in your script, but just to use that classes that you need. Autoloading will do the rest for you. Configuring autoloading has been a little bit complex in the past, especially because every library used its own conventions, but today thanks to the PSR-0 and PSR-4 standards and tools like Composer it is a piece of cake to use autoloading.
Dependency Injection: it’s a very common design pattern in the Java world that in the last years has got a lot of traction even in the PHP world thanks also to the effort of frameworks like Symfony, Zend and Laravel that use and advocate it widely. Basically it allows to inject components through the constructor or a setter method. This has the effect of forcing the developer to think in terms of dependencies and to create small isolated components focused on doing just one thing and do it well.
Lazy Loading: another important design pattern used to defer initialization of an object until the point at which it is needed. It’s mostly used with objects that deals with heavy resources like database connections or file based data sources.

Rule 3: Mama, I’ll do it tomorrow!

How many times do you needed to send an email to a user after he/she triggered a specific event in your web app (e.g. password changed or order completed)? How many times you needed to resize an image after the user uploaded it? Well it’s quite common to do these “heavy” operations before sending a success message to the user. To put it another way, our users expects to see some message in their browsers as soon as possible and we need to ensure that any additional task (not directly related with creating that message) should be deferred. The most common way to do that is to use job queues, which means that you have to store the minimum amount of data needed to perform the deferred task into a queue of some kind (e.g. a database, a message broker) and forget about it. You have to get back immediately to your main task: generating the output for the user! There will be some kind of worker in place with the goal to read from the queue periodically and perform the deferred job (e.g. sending the e-mail or generating the image thumbnails).

A simple queue system can be easily done with any kind of data store (very often Redis or MongoDB are used) or a message broker like RabbitMQ or ActiveMQ. There are also a lot of implementation already done for the PHP world:

Resque: A PHP queue library that uses Redis as data store.
Laravel Queues: the Laravel/Lumen out-of-the-box solution to defer jobs using queues and workers. It can be configured to use different data stores.
Gearman: A generic job server that supports the wide majority of languages (PHP among the others).
Beanstalkd: Another fast work queue with client libraries for the most common languages (Ruby, PHP, etc.)

Rule 4: Gotta cache ‘em all!

Nowadays web apps are really complex pieces of code. In order to generate a response to every request we generally do a lot of things: connect to one or more database, call external APIs, read configuration files, compute and aggregate data, serialize the results into some parseable format (Xml, Json, etc.) or render it with a template engine into a wonderful HTML page. Using a naive approach we can do that for every request that we get, our servers will never get bored to do repetitive tasks.

But there’s a smarter (and faster) way to do repetitive tasks, avoiding to calculate the same results again and again: enter “the cache”!

Cache, which is pronounced “cash” (not “catch” or “cashay”), stores recently used information so that it can be quickly accessed at a later time.

Cache is used widely in computer science and you can find it pretty much everywhere. For example the RAM itself can be considered as a way to cache the code of running programs to avoid the CPU to read the (slow) hard disk sparsely millions and millions of times.

In general with web programming we can focus on several different levels of cache:

Byte Code Cache: it’s a common feature of many interpreted languages (PHP, Python, Ruby, etc.) and it allows to avoid to interpretate source code files again and again if they are not changed since the last time. Some languages have this feature integrated in the core (Python), others like PHP needs to have it as an extension and several extensions exists for this sake: APC, eAccelerator, Xcache. Since PHP 5.5 we can use the Opcache extension which has been integrated in the core.
Application Cache: to not be confused with HTML5 Application cache, is the cache logic that regards your specific application and it’s probably the most important in terms of performance. Are you computing the 1264575th number in the Fibonacci sequence over and over again in your app? Well put the result in a cache and avoid to recompute it every time. Or, to give a more realistic example, are you always making the same expensive queries to the database over and over again to render the front page of you app? Well cache the results of the queries (or even the output of the whole page when possible) and avoid to hit the database for every user request. In these cases it’s a good idea to use cache servers like Memcached, Redis or Gibson.
HTTP Cache: Fetching data over the network is slow: a lot of round trips between the client and server are required and a lot of time is wasted before the browser is able to display the content. Wouldn’t it be useful to have ways to tell the browser to reuse content that he already downloaded? Well, you can do that by using HTTP Cache headers like Etag and Cache-control. This turns to be the cheapest way in terms of server resources to leverage cache (because everything is already in the browser and the server doesn’t get requests at all), you should only be sure to use it properly to avoid returning visitors to see stale content.
Proxy Cache: this technique refers to the usage of a dedicated server that receives all the HTTP traffic and it may have copies of the web pages requested by the users (often called Reverse proxy). In these cases it returns the copy of the page directly without requiring the app server to re-elaborate the request. It generally keeps the copy of the data in memory and avoids many network round trips so it’s generally an out-of-the-box approach to speed up very trafficked web sites where the content doesn’t change too frequently. Famous proxy servers are Varnish, Nginx and Squid. Also Apache can be configured to act as a reverse proxy.

Anyway, once you got the concept of caching, it is really easy to adopt it. The issues arise when you need to understand whether something changed and the cached version of your data might not be relevant anymore. In such cases you need to delete the data on the cache to be sure it gets correctly recomputed the next it’s requested. This process is called “cache invalidation” and it generally makes developers insane to the point that a very famous quote exists:

There are only two hard things in Computer Science: cache invalidation and naming things.

— Phil Karlton

Tweet this

If you have been into software development for a while I’m quite sure that you already had chance to read it!

There’s no silver bullet to make cache invalidation easy, it really depends from the architecture of your code and the requirements of your application. In general the less caching layers you have the better: always avoid to add complexity!

Here follows some articles that might be interesting to know more about caching for web applications:

Web application caching by DocForge
Fine tune your Opcache configuration to avoid caching surprises by Tideways blog
Beginners guide to HTTP cache headers by Mobify
HTTP Caching, optimizing content efficiency by Google
Using Http headers with Symfony by Symfony
What is a reverse proxy server by Nginx
Laravel cache by Laravel

Rule 5: Avoid the damn N+1 Query Problem

The “N+1 Query Problem” is a very common anti-pattern unintentionally used especially when dealing with relational databases. Basically it reads N record from the database by generating N+1 queries (one to read the n IDs and 1 for every record). Take a look at the following piece of code to have a real case (well… almost real) example:

<?php

function getUsers() {
  //... retrieve the users from the database (1 query)
  return $users;
}

function loadLastLoginsForUsers($users) {
  foreach ($users as $user) {
    $lastLogins = ... // load the last logins for the user (1 query, executed n times)
    $user->setLastLogins($lastLogins);
  }

  return $users;
}

$users = getUsers();
loadLastLoginsForUsers($users);

The given piece of code loads a list of users at first and then, for every user, it loads his last login times from the database. This code produces the following N+1 queries:

SELECT id FROM Users; -- ids: 1, 2, 3, 4, 5, 6...
SELECT * FROM Logins WHERE user_id = 1;
SELECT * FROM Logins WHERE user_id = 2;
SELECT * FROM Logins WHERE user_id = 3;
SELECT * FROM Logins WHERE user_id = 4;
SELECT * FROM Logins WHERE user_id = 5;
SELECT * FROM Logins WHERE user_id = 6;
-- ...

That’s obviously inefficient and it happens quite often with “has many” relationships in databases, especially when you are using some kind of magic ORM and you don’t exactly know what it is doing out of the box (and probably you haven’t configured it properly).

In general you can solve this problem by producing a query like the following:

SELECT id FROM Users; -- ids: 1, 2, 3, 4, 5, 6...
SELECT * FROM Logins WHERE user_id IN (1, 2, 3, 4, 5, 6, ...);

or by using the JOIN syntax where possible.

This problem can be only addressed when you are in control of your SQL queries or if you have a clear understanding of the ORM library you are using (if you are using one). Anyway keep it in mind and be sure you don’t fall in the N+1 queries trap, especially when you deal with large datasets. Many PHP profilers allows you to inspect the generated queries for every page request, they can be very useful companion to understand if you are doing things properly in terms of avoiding the N+1 queries problem.

On a side note, speaking of databases, be sure to keep a single connection open to your data source and to not reconnect for every query.

Don’t expect this to be exhaustive. This problem is very broad and it has a lot of other important case to analyze, so if you want to know more about it have a look at the following articles and books:

Performance: N+1 Query Problem by Phabricator
Nested Loops by Use the Index, Luke
Laravel’s Eloquent ORM Eager Loading by Laravel
Book Solving The N+1 Problem In PHP by Paul M. Jones

Rule 6: Scale… horizontally!

“Scalability” is not exactly the same thing of “performance”, but the two things are tightly intertwined. To give you my personal definition, “scalability” is the ability of a system to adapt and remain functional without noticeable performance issues when the number of users (and requests) grows.

It’s a very complex and broad topic and I don’t want to get into the details here. But for the sake of performance it’s worth to understand and keep in mind some simple things that you can do to be sure your app can be easily scaled horizontally. Horizontal scaling is a particular scaling strategy in which you add more machines to the cluster where your app is deployed. This way the load is split among all the machines and thus the whole system can remain performant even when there are a lot of simultaneous requests.

The two major problems to take in consideration when preparing for horizontal scaling are user sessions and user files consistency.

Sessions: quite often (especially in PHP applications) user session data is stored on the local filesystem where the app is deployed. Using this strategy is not only slower but it will not work if you have two machines to handle the requests (the session data stored in one machine will be different than the one stored in the other). A better solution is to use some kind of database to store the user session data. Most of the frameworks easily allows you to do that with just few lines of configuration. At the beginning, when your app is small and it’s not so popular, you can install your favourite session storage platform in the same machine of your web sever. Than when you will need to scale your architecture you can easily move the session storage into a separate machine and connect all your web machines to it.
User files consistency: Same problem of sessions happens when users can store files within your app. In these cases you need to be sure that whatever is the web server they end up with they are able to see their files. So you need to keep the files into a dedicated storage (like Amazon S3 or Rackspace Cloudfiles). Otherwise you can keep the files locally on the machines but you need to find a way to keep them synchronized within all the machines of the cluster. In this case you can use NFS or GlusterFS to create a shared filesystem.

Here’s a list of other interesting resources to know more about scalable web applications:

Horizontally scalable web applications by Inviqa
Horizontally Scaling PHP Applications: A Practical Overview by Digital Ocean
Best Practices For Horizontal Application Scaling by OpenShift
Scalable Web Architecture and Distributed Systems by Kate Matsudaira
Intuitively Showing How To Scale A Web Application Using A Coffee Shop As An Example by HighScalability
Book The art of scalability by Martin Abbot and Michael Fisher
Slides 7 Stages of scaling web applications by Rackspace

Conclusions

I really hope this long post has been useful for you. I just wanted to give you an idea of what are the general concerns to take in mind when you start to write an app that should take performance under consideration. As I said in the first rule, don’t fall into the trap of premature optimization and just focus on writing the right code for the right job. Anyway if you have this points clear in your mind you will almost automatically think about good solutions to achieve a good level of performance and scalability from the very first versions of your app and you can also make some smart architectural decisions from the very beginning.

In case you are an experienced web developer let me know if I forgot to mention any important rule and, if you commonly follow these rules, what you think about them.

I hope some great discussion will start from the comments here.

Thanks for taking your time to read this post.

Until next time!

Write a console application using Symfony and Pimple

Luciano Mammino — Sat, 15 Mar 2014 02:16:20 GMT

In this article I will show you how to set up a stand alone command line application by using the Symfony Console component and Pimple.

Writing console commands for Symfony (full stack framework) is easy and enjoyable. It helps a lot when you need to handle particular collateral tasks that are needed, in a way or another, to make your website running (processing and crunching data, make async requests, create reports, etc.).

Anyway I discovered that writing stand alone command line applications using only the Symfony/Console component is a lot more easy and enjoyable and that a lot of famous command line applications uses it (Composer and Laravel/Artisan just to name a few). Furthermore by using Symfony I became a great fan of the Dependency Injection and Inversion of Control (IoC) design pattern and, as my dependencies started to grow, I wanted to put some sort of container in my command line apps. I decided to go with Pimple: a really simple dependency injection container written by Fabien Potencier, the notorious head behind the Symfony framework and Sensio.

Let’s start

I will demonstrate my approach by creating a simple “hello $name” command line application that will be able to count how many times you greet someone. You can find the whole code in a dedicated GitHub repository.

So we will be able to run

app/console greet Alice

and it will print out

Hello Alice
(First time!)

Yes, intentionally simple! ;)

Let’s start by creating our composer.json file. We will need the Symfony console and the Pimple packages. We also include the Symfony Yaml component as we will store data in a yaml file (obviously we could have been using json but I believe yaml is cooler :P).

{
  "name": "lmammino/symfony-console-pimple",
  "description": "A sample Symfony Console app using Pimple",
  "require": {
    "symfony/console": "dev-master",
    "pimple/pimple": "dev-master",
    "symfony/yaml": "dev-master"
  },
  "license": "MIT",
  "authors": [
    {
      "name": "Luciano Mammino",
      "email": "lmammino@oryzone.com"
    }
  ],
  "autoload": {
    "psr-4": {
      "LMammino\\ConsoleApp\\": "src/"
    }
  }
}

Yes, let’s run composer update to download all the libraries.

Folder structure

Let’s structure our code. We want to separate application and configuration code from the main source code. So we will end up with the following folder structure:

app
src
vendors

The app folder will contain our console executable file, a bootstrap file and a config folder. We will get into the details in a while.

The Greeter service

Let’s just define our core service by Writing the Greeter class. This class defines the business logic of our greeting application.

<?php

namespace LMammino\ConsoleApp;

use Symfony\Component\Yaml\Yaml;

class Greeter
{
    /**
     * @var string $file
     */
    protected $file;

    /**
     * @var array $greetings
     */
    protected $greetings;

    /**
     * Constructor
     *
     * @param string $file
     */
    public function __construct($file)
    {
        $this->file = $file;
        if (file_exists($file)) {
            $this->greetings = Yaml::parse(file_get_contents($file));
        } else {
            $this->greetings = array();
        }
    }

    /**
     * Destructor
     */
    public function __destruct()
    {
        file_put_contents($this->file, Yaml::dump($this->greetings));
    }

    /**
     * Builds the greeting for someone (you can yell on it if you want!)
     *
     * @param  string $name
     * @param  bool   $yell wanna yell?
     * @return string
     */
    public function greet($name, $yell = false)
    {
        $output = sprintf('Hello %s', $name);
        if ($yell) {
            $output = strtoupper($output);
        }

        $name = strtolower($name);
        if (!isset($this->greetings[$name])) {
            $this->greetings[$name] = 1;
        } else {
            $this->greetings[$name]++;
        }

        return $output;
    }

    /**
     * Will tell you how many times you greet someone
     *
     * @param  string $name
     * @return int
     */
    public function countGreetings($name)
    {
        $name = strtolower($name);
        if (!isset($this->greetings[$name])) {
            return 0;
        }

        return $this->greetings[$name];
    }
}

The class is really simple. The main methods are greet and countGreetings that allows you to build the greet string for someone and to count how many times you have greet someone.

Note that this class needs to know on costruction which file to use to read and store the greetings count. This will be something we will configure through Pimple as a container parameter.

The GreetCommand

Now we have a service with the main business logic, let’s just write a Symfony command to run it:

<?php

namespace LMammino\ConsoleApp\Command;

use LMammino\ConsoleApp\Greeter;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;

class GreetCommand extends Command
{
    /**
     * @var \LMammino\ConsoleApp\Greeter $greeter
     */
    protected $greeter;

    /**
     * Constructor
     *
     * @param Greeter $greeter
     */
    public function __construct(Greeter $greeter)
    {
        parent::__construct();
        $this->greeter = $greeter;
    }

    /**
     * {@inheritDoc}
     */
    protected function configure()
    {
        $this->setName('greet')
            ->setDescription('Greet someone')
            ->addArgument('name', InputArgument::OPTIONAL, 'The name of the one you want to greet', 'World')
            ->addOption('yell', 'Y', InputOption::VALUE_NONE, 'If set will scream out the greeting. Use with caution!');
    }

    /**
     * {@inheritDoc}
     */
    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $name = $input->getArgument('name');
        $yell = $input->getOption('yell');

        $output->writeln($this->greeter->greet($name, $yell));
        if (1 === ($count = $this->greeter->countGreetings($name))) {
            $output->writeln('(First time!)');
        } else {
            $output->writeln(sprintf('(%d times)', $count));
        }
    }
}

The command is totally self explainatory! It just defines the greet command offering a name argument and a yell option (both optional). The point here is that our command has a dependency on the Greeter class we wrote before. So we need to pass it on construction (or we need to configure our Pimple container to do so).

Ladies and gents, the Pimple container!

Finally it’s time to write our Pimple container. Before getting to the code let’s recap things a bit. We have a parameter (the name of the greetings count file) and two services (the Greeter service and the GreetCommand). We will create a app/config/container.php file to define our parameters and services with Pimple:

<?php

$c = new Pimple();

$c['parameters'] = array(
    'greetings.file' => 'greetings.yaml'
);

$c['greeter'] = function($c) {
    return new \LMammino\ConsoleApp\Greeter($c['parameters']['greetings.file']);
};

$c['command.greet'] = function($c) {
    return new \LMammino\ConsoleApp\Command\GreetCommand($c['greeter']);
};

$c['commands'] = function($c) {
    return array(
        $c['command.greet']
    );
};

$c['application'] = function($c) {
    $application = new \Symfony\Component\Console\Application();
    $application->addCommands($c['commands']);
    return $application;
};

return $c;

If you were not familiar with Pimple you should have noticed the simple syntax of this container. We just need to create a Pimple instance and it acts like an array. In this “array” we put parameters as simple data (scalar or array values) and service definitions as functions that return instantiated services. Let’s check our definitions one by one:

$c['parameters'] contains the applications parameters (in a more complex application with a lot of parameters you can load the values from an external configuration file)
$c['greeter'] defines the construction of our Greeter service
$c['command.greet'] defines the construction of our greet command
$c['commands'] is an helper definition that returns an array with all the commands that we want to add in our application
$c['application'] defines the creation of our command line application

Ok, we are almost done. We just need to write our bootstrap file and our console executable file.

The bootstrap file

The file app/bootstrap.php is used to load the composer autoloader class and our container:

<?php

set_time_limit(0);

require __DIR__ . '/../vendor/autoload.php';

$container = require(__DIR__ . '/config/container.php');

set_time_limit(0) ensures that our script will not be killed after a certain amount of seconds (if your php.ini wants so). It’s almost useless in this particular case (our command will run in a few milliseconds) but adding it in Php command line applications is a good practice (especially when you have to deal with long running tasks).

The executable console file

The last step needed to make our application executable is to write the app/console file. This is a php file that can be executed from the command line (you need to chmod +x it).

#!/usr/bin/env php
<?php

require __DIR__ . '/bootstrap.php';

$application = $container['application'];
$application->run();

By using a container it just need to load our “application” service and call run() on it.

Note that the first “shebang” line (#!/usr/bin/env php) allows us to run this file by calling app/console (so that you can avoid to call the php interpreter explicitly).

Conclusions

This app is very simple and it will be easy to build even without adopting a container. Anyway I think this approach guarantees a good organization for your code and will became really useful when your command line application start to grow in terms of complexity. I recently had to build a command line app that uses Doctrine and JMS/Serializer (plus several other dependecies). I can say that adopting a container like Pimple helped me a lot to keep things organized and services decoupled.

Just to make a final recap I think this approach guarantees several benefits:

Write “container agnostic” commands (they does not know the container, but have only the required dependencies injected)
Attach new commands from configuration: you just need to add them to the $container['commands'] array
Help you a lot to write thin commands (yes, I think commands acts like controllers and they should be “thin” too), because you have a simple way to declare services and their dependencies and you are able to inject only the needed ones in every command
Allow you to have parameters and configuration (useful when you have to enstablish a connection with a database or use external resources that needs configuration such as an external API)

That should be all. Feel free to comment this post or to contribute to the sample app repository if you feel that this approach can be tweaked.

UPDATE (March 16, 2014):
Javier Egiluz, great Symfony evangelist, pointed out that one of his command line applications, easybook, uses the Symfony Console component in conjunction with Pimple. So if you want to have a look to a more complete and realistic (and complex :P) use case I really suggest you to have a look at the easybook code base.

Have a nice weekend!

Simple echo server written in Go, dockerized!

Luciano Mammino — Sun, 29 Jun 2014 14:43:56 GMT

In this post we will see how to write a (very) simple Go server app and how to put it into a Docker container and run it.

Disclaimer: I’m an absolute beginner with both Go and Docker and this is just an experiment I did to start to explore this two interesting technologies. If you’ll find something weird or wrong you are very encouraged to scream at me in the comments (Comments), I will be extremely thankful ;)

Introduction

We are going to write a simple echo server in Go and then we will pack it into a Docker container for future execution/distribution.

“Luciano, you are a PHP developer, why bothering with Go?”

Nice question! Well, there are at least two reasons for this choice:

Experimenting! Yes Go seems to be the next cool thing in the field and I wanted to have an excuse to try it.
Simplicity! Writing a server app in Go is just a matter of writing few lines of code in a file, so it’s an very simple scenario to test the integration with Docker.

For those who are not familiar with these two technologies I suggest give the official websites a shot (Docker.com and Golang.org) as I will not provide many specific details about them. I assume you have both technologies installed on your machine. If you are using Windows or a Mac have a look at Boot2Docker, it will definitely make your life easier.

Beware that Go on your local machine is needed only if you want to test the application before “dockerizing” it. We will put the Go runtime into the Docker container in the second part of the post and this way we will be able to execute it without having Go installed in our local machine (that’s infact the real advantage of using Docker).

The Go echo server

Ok, first of all let’s write and test our server app. The app should open a socket and listen for TCP requests on a given port. Then when it receives some data it should reply by saying something like that “Hi, I received your message! It was X bytes long and that’s what it said: XXX ! Honestly I have no clue about what to do with your messages, so Bye Bye!”.

So let’s just write some code. The following script has been widely inspired by this one.

package main

import (
    "fmt"
    "net"
    "os"
    "strconv"
    "bytes"
)

const (
    CONN_HOST = ""
    CONN_PORT = "3333"
    CONN_TYPE = "tcp"
)

func main() {
    // Listen for incoming connections.
    l, err := net.Listen(CONN_TYPE, ":"+CONN_PORT)
    if err != nil {
        fmt.Println("Error listening:", err.Error())
        os.Exit(1)
    }
    // Close the listener when the application closes.
    defer l.Close()
    fmt.Println("Listening on " + CONN_HOST + ":" + CONN_PORT)
    for {
        // Listen for an incoming connection.
        conn, err := l.Accept()
        if err != nil {
            fmt.Println("Error accepting: ", err.Error())
            os.Exit(1)
        }

        //logs an incoming message
        fmt.Printf("Received message %s -> %s \n", conn.RemoteAddr(), conn.LocalAddr())

        // Handle connections in a new goroutine.
        go handleRequest(conn)
    }
}

// Handles incoming requests.
func handleRequest(conn net.Conn) {
  // Make a buffer to hold incoming data.
  buf := make([]byte, 1024)
  // Read the incoming connection into the buffer.
  reqLen, err := conn.Read(buf)
  if err != nil {
    fmt.Println("Error reading:", err.Error())
  }
  // Builds the message.
  message := "Hi, I received your message! It was "
  message += strconv.Itoa(reqLen)
  message += " bytes long and that's what it said: \""
  n := bytes.Index(buf, []byte{0})
  message += string(buf[:n-1])
  message += "\" ! Honestly I have no clue about what to do with your messages, so Bye Bye!\n"

  // Write the message in the connection channel.
  conn.Write([]byte(message));
  // Close the connection when you're done with it.
  conn.Close()
}

Ok, the code is commented at it should be understandable enough. Let’s try if it works.

Just launch the server with the command:

go run server.go

We will have our server running and it will print out something like:

Listening on :3333

Let’s open another terminal window to try to speak with him. We will use netcat:

echo "Hello server" | nc localhost 3333

That’s what we should see in the two terminal windows:

The terminal window on the top is running our server and it logs all the received requests (by printing the incoming TCP address and the local one), while the second terminal is our client that sent out a simple “Hello server” message and received a very useful response from the server.

That’s all from the Go side. In the next part of the post we will see how to “dockerize” this simple Go app. For now, you can obviously quit the server with a CTRL+C.

Dockerizing the application

Ok, now we want to “dockerize” this simple application. Wait, what does “dockerize” mean? Well, by “dockerizing” an application we will be able to put the application itself and all its dependencies within a Docker container that can be easily shipped and executed elsewhere.

Images, Containers and Dockerfile(s)

An extract from the Docker documentation should make things a bit clearer:

Docker enables apps to be quickly assembled from components and eliminates the friction between development, QA, and production environments. As a result, IT can ship faster and run the same app, unchanged, on laptops, data center VMs, and any cloud.

There are several ways to dockerize an application and create a dedicated Docker container. The one I prefer at the moment is by adopting a “Dockerfile”.

A “Dockerfile” is a file called exactly Dockerfile that contains several reproducible instructions to create a Docker image from scratch.

What the hell is an image and how it is different from a container?

If you are a noob with Docker (like me) you are probably asking yourself this question. I want to quote a brilliant answer from stackoverflow that should make things a bit clearer:

An image is the set of layers that are built up and can be moved around. Images are read-only.

http://docs.docker.io/en/latest/terms/image/ > http://docs.docker.io/en/latest/terms/layer/ A container is an active (or inactive if exited) stateful instantiation of an image.

http://docs.docker.io/en/latest/terms/container/

Feel free to visit these links if things are not clear yet.

Now let’s get back to our Dockerfile. Every Dockerfile contains some metadata (base image from which to start, the name of the maintainer, etc.) and a set of instruction used to build the image. These instructions are usually used to “install” all the needed dependencies and describe how to execute the contained application.

Let’s write the Dockerfile

Enough talking, let’s jump to the code:

FROM ubuntu:12.04

MAINTAINER Luciano Mammino

RUN apt-get install -y python-software-properties

RUN add-apt-repository ppa:duh/golang
RUN apt-get update
RUN apt-get install -y golang

ADD server.go /var/server/server.go

EXPOSE 3333

CMD ["go", "run", "/var/server/server.go"]

Note: A reader mentioned in a comment that this dockerfile is no longer working (probably the PPA has beem removed or its temporary unavailable). If that’s your case you can use a minimal dockerfile optimized for GoLang provided by in the GoLang blog itself.

Let’s analyze all the instructions one by one:

FROM ubuntu:12.04 defines the base image from which to start. In this case we will use the ubuntu:12.04 image, a very lightweight image based on ubuntu 12.04.
MAINTAINER Luciano Mammino isn’t it self-explanatory enough? ;)
RUN apt-get install -y python-software-properties installs the python-software-properties package to be able to execute add-apt-repository next.
RUN add-apt-repository ppa:duh/golang, RUN apt-get update, RUN apt-get install -y golang and RUN apt-get install -y golang: adds a custom repository for Go and installs it.
ADD server.go /var/server/server.go: copies the file server.go into the image (storing it into the /var/server/ folder)
EXPOSE 3333: exposes the port 3333 to allow containers launched from this image to listen on this port. We need exactly this port because it is the one used by our Go application.
CMD ["go", "run", "/var/server/server.go"]: describes the command to execute our Go application when the container is launched (yes, it will execute go run /var/server/server.go).

Let’s build!

Before building the image from our Dockerfile, if you are using boot2docker, you need to run it with:

boot2docker start

In case it prints out something like this:

2014/06/29 16:09:32 Started.
2014/06/29 16:09:32 To connect the Docker client to the Docker daemon, please set:
2014/06/29 16:09:32     export DOCKER_HOST=tcp://192.168.59.103:2375

Run the suggested export command. Also take note of the given IP address as we would need to use later to connect to our server.

Now just cd into the folder that contains the Dockerfile and the server.go file and run:

docker build -t goecho .

Notice the . that refers to the current folder (yes you got it, it says Docker to look for a Dockerfile whitin it).

The -t goecho option is not mandatory and is used to “give a convenience name” to the resulting image.

You will see a series of commands being executed (yes the ones we wrote within our Dockerfile) and, if everything goes well, you should see something like this at the end:

Successfully built 713c09526bc1

You can also execute

docker images

To list all the available images (you should see our goecho on the top of the list).

Our image is ready to be run and trasformed into a live container. We’ll see how to do it in the next part of the post.

Running the container

Running the image and having a live container is just a matter of running:

docker run -i -t -p 3333:3333 goecho

The options -i and -t allows us to execute the container into the interactive mode that allows us to shut it down with CTRL+C when needed.

The option -p 3333:3333 maps the container port 3333 to the same port on our machine. It will effectively allows us to talk with the echo server using the port 3333.

Now our container is live and running and our server application is active.

Let’s open a new terminal window and try again:

echo "Hello server" | nc <ip> 3333

<ip> is localhost if you are not using boot2docker. If you are using it you need to provide the IP of the intermediade virtual machine provided by boot2docker. It is the one we recorded before, but if you can’t find it you can simply run boot2docker ip to discover it (it’s usually 192.168.59.103).

Conclusions

That’s only the very beginning with Docker, there are a bunch of other interesting features and approaches. I will probably write some other post while I will keep learning and using Docker, so stay tuned and write all your suggestions in the comments.

Thank you

Backup your server with Dropbox

Luciano Mammino — Fri, 17 Jan 2014 03:44:19 GMT

In my early days as CTO at Sbaam I had to setup a web server from the ground up. As it happens in many startups the work had to be done quickly and with an almost-0-budget, so it left no space to sophisticated solutions for recurring tasks such as backup. I always have been a web developer and focused on coding so, I admit I had really a poor knowledge about how to setup a remote unix virtual machine.

So, speaking about backups, I needed a solution that would be cost-effective, easy to install and easy to maintain at the same time. I would have loved it if it can be as simple as sharing a Dropbox folder. As this thought crossed my mind I wondered if there was some way to interact with dropbox from a script to create files and folders and started googling about it. Luckily Dropbox offers a good command line client that allows to bring your synced files and folders also on graphic-less machines.

Ultimately my solution was to install the Dropbox command line on the server using a dedicated Dropbox account and backup files by simply copying/linking them on the Dropbox folder. This way I prepared and scheduled a script that simply had to copy the files I wanted to backup on the dropbox folder. Then I have the files backupped in the cloud and automatically synced on my local machine. Furthermore i had chance to share the backup folder with all my collaborators.

This solution works very well for small projects so I will resume all the steps I followed to install and use dropbox this way. I used an ubuntu machine so I suppose the following steps should work on debian machines.

Prepare the dropbox user

I preferred to have a dedicated user to handle the whole Dropbox daemon and folder so just create it now:

sudo useradd -d /dropbox -m dropbox

It will have the directory /dropbox as home and the name dropbox. You can change these values if you like.

Then you have to set the password for the new user:

sudo passwd dropbox

Choose whatever password you like.

Security concern: If you have ssh access enabled (obviously it is) it’s better to disable the ssh access for the new user. So edit the file /etc/ssh/sshd_config and add the rule DenyUsers dropbox, the restart ssh with sudo service ssh restart.

Install the dropbox client

First of all you need to switch to the user created in the previous step, so the Dropbox installer will create the Dropbox folder under its home. At the end you will have /dropbox/Dropbox as the synced folder:

su dropbox

(enter the password for the user dropbox)

Now you’re the dropbox user. Be sure to switch to your user folder with cd ~ and let’s download and install the daemon.

wget -O dropbox.tar.gz "http://www.dropbox.com/download/?plat=lnx.x86"

for 32bit or

wget -O dropbox.tar.gz "http://www.dropbox.com/download/?plat=lnx.x86_64"

for 64bit.

Extract it:

tar -xvzf dropbox.tar.gz

It will extract to the ~/.dropbox-dist. folder. Now run the client:

~/.dropbox-dist/dropbox

You will see an output like the following:

This client is not linked to any account… Please visit {SOME_URL} to link this machine. […]

Copy and paste the provided URL in the browser bar of your local machine and it will ask you to enter the password of your dropbox account. This way it is able to authenticate your command line client and start syncing your files. At this point it should have been started its work but our shell is still locked by the client. We need to kill and daemonize it to being able to manage it as a service. Press CTRL+C and get back to your user with the exit command.

Dropbox as a service

At this point we need to define dropbox as a service. So let’s create an etc init script. Download my gist

wget -O dropbox_init_script "https://gist.github.com/lmammino/8467336/raw/dropbox"

and move it in the init folder:

sudo mv dropbox_init_script /etc/init.d/dropbox

Make it executable:

sudo chmod +x /etc/init.d/dropbox

And set it to load at startup:

sudo update-rc.d dropbox defaults

Now it’s a service! Run:

sudo /etc/init.d/dropbox start

to start it and feel free to use common service command such as stop, restart and status.

Enjoy

At this point you have all your dropbox data in the /dropbox/Dropbox folder. Feel free to copy all the files you want in there or to schedule jobs that does all the dirty work for you. You can also create symlinks into the dropbox folder to keep files and folders placed in other locations synced.

Security concerns: consider that your dropbox folder acts as a normal dropbox folder so it’s synced both ways. If someone breaks into your dropbox account (or the account of some collaborator who shares the folder with you) he can use dropbox as an hole to inject malicious files into your server or steal sensible data. So, for example, avoid to have scheduled scripts and unencrypted sensible data in that folder.

Bonus

Dropbox released the Dropbox CLI, a python command line application that you can use to perform some useful task such as Selective Sync, disable the LAN sync or retrive public links of your files. I suggest to download it by using the dropbox user and place it under /dropbox/bin. So you can simply switch to the dropbox user (again with su dropbox), download it and make it executable:

mkdir ~/bin
wget -O ~/bin/dropbox.py "https://www.dropbox.com/download?dl=packages/dropbox.py"
chmod +x ~/bin/dropbox.py

At this point you can run the Dropbox CLI. For example if you want to disable the LAN sync (heavily suggested in this case) you can simply do:

/dropbox/bin/dropbox.py lansync n

That’s all. It was a long read but I hope it has been useful ;)

Developing a web application with Lumen and MySql

Luciano Mammino — Sat, 18 Apr 2015 16:10:35 GMT

Lumen is a new Php micro-framework developed by Taylor Otwell, the same author of the famous Laravel framework. I wanted to give it a try and I am here to share my experimentations. I am not an expert of Lumen (yet), but I think one of the best characteristics of this framework is that it makes really really easy to bootstrap a new project. So to prove this, we will now build a fully functional app backed by a MySql database in less than 30 minutes. Are you ready to start?

A motivational quote everyday

Our app should be quite simple but I’d like also to make something useful. I am a big fan of motivational quotes, and if you follow me on Twitter (you should!) you probably already know it!

So, the idea is to build a web app that showcases a new quote everyday. This way, everyday when you wake up you can run your application and be inspired and motivated by a wise and energising quote to do your best! To work best our app should obviously be fancy, with über cool background images, like in the screenshot above. That’s an importante detail!

A new Lumen project

Let’s start. First of all, to create a new Lumen project, we need to have the Lumen command line installer. A very simple tool, which in turn uses Composer, that allow us to bootstrap a new Lumen project in few seconds. To get it ensure to have installed Composer globally and run the following command:

composer global require "laravel/lumen-installer=~1.0"

Note: you might need to use sudo, depending on the way you installed Composer.

Once we have done this we have a new toy in our shell: the lumen command. We can now create a new project by running:

lumen new motivational

motivational is the name of our new app. The command creates a new folder for it and downloads all the dependencies.

To see our application live we need to cd into our motivational folder and run

php artisan serve

Our project will be immediately up and running on http://localhost:8000.

Define the data model

We said we want to showcase quotes and in our case a quote is made up by:

a text (The quoted text itself)
an author (The name of the author of the quote)
a background image (Yes, to make everything fancier)

In order to manage data from the database, we need to enable Eloquent (the Lumen/Laravel ORM library), configure our database connection, create a migration and a model and finally seed our database. Let’s do it step by step.

Enable Eloquent

To enable Eloquent we need to edit the file bootstrap/app.php and remove the comment on the following lines:

$app->withFacades();
$app->withEloquent();

Notice that the first line enables the support for Facades (a very common feature used in Laravel and inherited by Lumen) that simplifies the usage of some of the core classes of the framework.

Configure the database connection

First of all, be sure to have an instance of MySql running on your machine and to have the credentials to connect to it. Now let’s create our .env (Dotenv) configuration file. Copy the .env.example file into a new .env file and open it in your favourite editor.

Here we need to edit the following lines and provide the details needed to connect to our local mysql instance:

DB_CONNECTION=mysql
DB_HOST=localhost
DB_DATABASE=homestead
DB_USERNAME=homestead
DB_PASSWORD=secret

Be sure to create the database (homestead in this case, but you can obviously customise it).

It’s also a good idea in general to change the APP_KEY value into some random string in case you are building a “serious” application.

To make Lumen load this configuration file we need, again, to edit the bootstrap/app.php file and uncomment the following line:

Dotenv::load(__DIR__.'/../');

Create a migration

Migrations allows the framework to keep the database schema under control. They define all the database tables and fields programmatically and keep track of the various changes on them (so that we can easily update and rollback the whole schema when needed). We need to initialise the migration system with the command:

php artisan migrate:install

This command creates a special table in our database called migrations that will be used internally from the framework to keep track of all the available migrations and the current one used.

Every migrations is identified by a file that generally lives in the database/migrations folder. The file describes the changes in our schema (eg. new tables, new fields, new indexes, tables to be deleted, etc…). In our case we need to create a new table, the “quotes” table to be precise. Let’s run this:

php artisan make:migration --create=quotes create_quotes_table

It creates a new file under the database/migrations folder that we can easily edit to add the fields we want to have in our table. We just need to tweak the up() function a bit:

public function up()
{
    Schema::create('quotes', function(Blueprint $table)
    {
         $table->increments('id');
               $table->timestamps();

         // our new fields
               $table->string('text');
               $table->string('author');
               $table->string('background');
    });
}

To execute the migration (and effectively create the table on the database) we have to run:

php artisan migrate

Create the Quote model

In general, a model is a class used to abstract our data and represent it as an object. In ORMs a model class also offers static methods to query the data storage to retrieve the data from the data source and create the corresponding objects.

In our case we need to define the Quote model in app/Models/Quote.php:

<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Model;

final class Quote extends Model
{

}

Yes, that’s it… we don’t really need to write anything else! We are extending the Eloquent Model class that does all the hard work for us, proving a standard model configuration that is good enough most of the times. In this case it automatically maps the class Quote to the quotes table that we created before.

Seed the database

The seeding process allows us to populate our database with initial data. In our case we can use it to provide some quotes. It’s quite useful because we will not build a full fledged admin to do the data entry at this stage (and we also will not have to touch the database manually).

Seeds files are stored in database/seeds and we need to create a new file there. We will call it QuoteTableSeeder.php:

<?php

use App\Models\Quote;
use Illuminate\Database\Seeder;

class QuoteTableSeeder extends Seeder
{
    public function run()
    {
        Quote::create([
            'text' => 'Success is going from failure to failure without losing your enthusiasm',
            'author' => 'Winston Churchill',
            'background' => '1.jpg'
        ]);

        Quote::create([
            'text' => 'Dream big and dare to fail',
            'author' => 'Norman Vaughan',
            'background' => '2.jpg'
        ]);

        Quote::create([
            'text' => 'It does not matter how slowly you go as long as you do not stop',
            'author' => 'Confucius',
            'background' => '3.jpg'
        ]);

        //... add more quotes if you want!
    }
}

The code is quite self explanatory: every call to Quote::create inserts a new record into the quotes table with the data provided into the array passed as argument. The only thing worth noticing is that we are passing a relative reference to a file in the background field. You need to have these files into your public/img/ folder, as this allows us to serve these files to the browser. If you need some good royalty-free photos have a look at the Unsplash project.

To enable this seed script we need to link it to the main DatabaseSeeder script by adding the following line within the run() method:

$this->call('QuoteTableSeeder');

Now we just need to launch the following command to execute the seed script and populate the database:

php artisan db:seed

Damn, we got a Class QuoteTableSeeder does not exist! That’s why by default Lumen Composer file maps the database path with the classmap strategy. That means that every time Composer dumps the autoloader, it creates a static map of all the classes available inside that folder. So every time we add a new class there we need to manually re-dump the autoloader script:

composer dump-autoload

Now let’s run again php artisan db:seed and this time everything should be fine. If you explore your database you will see some records within the quotes table.

The routing

Until now we just defined the data model of our application and populated our database. Now we will add some business logic and we will map it to some routes.

We will have two routes:

GET / - the main route, providing a new quote everyday
GET /{id} - the route of a specific quote, mapped by id

To define the business logic associate to a route we have to edit the app/Http/routes.php file:

<?php

use App\Models\Quote;

/**
 * Display the today quote
 */
$app->get('/', function() use ($app) {

    /*
     * Picks a different quote every day
     * (for a maximum of 366 quotes)
     *
     *   - $count: the total number of available quotes
     *   - $day: the current day of the year (from 0 to 365)
     *   - $page: the page to look for to retrieve the
     *            correct record
     */
    $count = Quote::query()->get()->count();
    $day = (int) date('z');
    $page = $day % $count + 1;

    $quotes = Quote::query()->get()->forPage($page, 1)->all();

    if (empty($quotes)) {
        throw new \Illuminate\Database\Eloquent\ModelNotFoundException();
    }

    return view('quote', ['quote' => $quotes[0]]);
});

/**
 * Display a specific quote
 */
$app->get('/{id}', function($id) use ($app) {
    $quote = Quote::query()->findOrFail($id);
    return view('quote', ['quote' => $quote]);
});

The two $app->get defines the two routes we need for our app. For every route we define the business logic within a closure function. The code is very straightforward and, thanks to the comments, it should be quite easy to understand.

The view() function allow to render a template. In this case we are rendering the quote template passing the model as quote variable. In the next paragraph we will see how to define our template.

Note: if you don’t want to use closures in your routing and you want to have a greater control on the structure of your code and your business logic you can leverage Laravel’s controllers. I will not go into the detail of this, but you can check the Documentation that shows how to do it. It’s quite simple anyway.

The template

Lumen uses the Laravel default template language, Blade. Blade allows us to render complex HTML code in a easy way. All the templates live in the resources/views folder. Let’s create the quote.blade.php file:

<html>
<head>
    <title>Motivaitonal — Your daily source of motivation!</title>
    <link href="/css/style.css" rel="stylesheet" type="text/css"/>
    <link href='http://fonts.googleapis.com/css?family=Alegreya:400,700|Roboto+Condensed' rel='stylesheet' type='text/css'>
</head>
<body style="background-image: url('/img/{{$quote->background}}')">
<div class="container">
    <div class="quote-container">
        <p class="text">{{$quote->text}}</p>
        <p class="author">— {{$quote->author}}</p>
    </div>
</div>
</body>
</html>

As you can see we can use the double curly braces syntax to reference variable values.

To finish we just need to create our public/css/style.css stylesheet file:

html,
body {
  height: 100%;
  padding: 0;
  margin: 0;
}

body {
  background-size: cover;
}

.container {
  height: 100%;
  background: rgba(0, 0, 0, 0.3);
}

.quote-container {
  position: relative;
  top: 50%;
  transform: translateY(-50%);
  padding: 2em 4em;
}

.quote-container p {
  text-align: center;
  color: #fff;
  text-shadow: 1px 1px 1px rgba(150, 150, 150, 0.8);
}

.quote-container p.text {
  font-family: 'Alegreya', serif;
  font-size: 4em;
}

.quote-container p.author {
  font-family: 'Roboto Condensed', sans-serif;
  font-size: 1.2em;
}

That’s it, now your app is up and running. Isn’t it beautiful? Hey, let me know if you decide to publish it and motivate the whole world! ;)

Conclusions

Lumen seems to be a very promising framework for the fast prototyping of small web apps. I look forward to use it again for slightly more complex use cases where I can adopt other interesting features like the cache layer and the job queue library. Have you already used Lumen? Do you think it will become a mainstream framework along Laravel or it will just be another of the hundreds of Php framework available out there? I’m really curious to know what you think about it, let me know it in the comments!

Best regards and have a very motivated day ;)

Versioning and deploying a static website with Git, Flightplan and Nginx

Luciano Mammino — Tue, 08 Sep 2015 22:16:27 GMT

Do you ever wondered how to manage the versioning and deployment process of a website? It seems to be a very interesting yet complex topic for which there are already thousands of different solutions. In a recent collaboration with Usersnap I had the pleasure to write a very detailed article for their blog that proposes a solution based on Flightplan.js, Git and Nginx.

My solution is very simple, it requires very few dependencies on your system (Git and Node.js) and it has been thought to give you the basics of how to define a minimal yet complete setup that it’s easy to customize and extend for more complex requirements.

Without further ado I really advice you to go and read the article on the Usersnap blog:

A beginner’s guide to deploying static sites with versioning and rollbacks using Flightplan

If you like the article or you have any question or comment, please use the comment box on the Usersnap blog post.

Cheers :)

Introducing mongo-uri-builder, a Node.js module to easily create mongodb connection strings using objects

Luciano Mammino — Tue, 29 Sep 2015 23:07:02 GMT

A couple of days ago I had the need to store the MongoDB connection string for a Node.js application I am currently building. Of course it was not a big deal and at first I stored it in a file. Anyway at some point I realised that I would needed to override parts of this string to change some settings in production (e.g. adding replicas and authentication settings). For this sake it would have been nice to have a way to store this configuration as a “well organised” object and then override just the properties I wanted to change.

I often use the config module to store my configuration and so I wanted to be able to do something like this in my configuration file:

{
  "mongo": {
    "host": "localhost",
    "port": 27017,
    "database": "mydb",
    "username": "loige",
    "password": "whyDoYouWantToKnowMyPassword"
  }
}

And then to be able to retrieve this data and use it with a new MongoClient instance:

var MongoClient = require('mongodb').MongoClient
var config = require('config')

var mongoConfig = config.get('mongo')
MongoClient.connect(
  createConnectionString(mongoConfig),
  function(err, db) {
    //...
  }
)

The missing bit here was the function createConnectionString. How to do that? I made a quick search on NPM and I wasn’t able to find something ready to be used… So, given that it was a quite easy task and that I enjoy to create new packages, I decided to build it by myself: welcome mongo-uri-builder! It’s alive!

(Yes, this was sort of the feeling I had after launching npm publish, call me crazy…)

The mongo-uri-builder package

mongo-uri-builder is a Node.js package to easily create mongodb connection strings using configuration objects.

The configuration object that the module expects looks like this:

var mongoConnectionConfig = {
  username: 'user', // the username
  password: 'pass', // the password
  host: 'host1', // the main host (default: "localhost")
  port: 1111, // the main port
  replicas: [
    // an array of replica databases
    // every replica must define an host, the port is optional
    { host: 'host2', port: 2222 },
    { host: 'host3', port: 3333 },
  ],
  database: 'db', // the name of the database
  options: {
    // an arbitrary object of connection options
    w: 0,
    readPreference: 'secondary',
  },
}

All the properties are optional and you can even use an empty object. The classic mongodb://localhost will be generated as default in this case.

As we are used with NPM, installing the module is as easy as running:

npm install --save mongo-uri-builder

Then to use it you can do something like this:

var mongoUriBuilder = require('mongo-uri-builder')

var connectionString = mongoUriBuilder({
  username: 'user',
  password: 'pass',
  host: 'host1',
  port: 1111,
  replicas: [{ host: 'host2', port: 2222 }, { host: 'host3', port: 3333 }],
  database: 'db',
  options: {
    w: 0,
    readPreference: 'secondary',
  },
})

console.log(connectionString)

// outputs "mongodb://user:pass@host1:1111,host2:2222,host3:3333/db?w=0&readPreference=secondary"

How easy and “well-readable” it is now? :)

Contributing & Issues

As I often do I put the code of the module on GitHub, you can find the repository at lmammino/mongo-uri-builder. Everyone is more than welcome to contribute to the project. You can contribute just by submitting bugs and pull requests or suggesting improvements by opening an issue.

Wrap up

This module is really something naive but it is a nice thing to have for me, especially in conjunction with config, that allows me to have different configuration files for every environment (e.g. development and production) and to override properties from a default configuration file. This way I can just override the parts of the connection string that are effectively different from the default configuration. I can even use environment variables if I don’t want for example to store the username and password of my database as clear text in a file.

I really look forward to knowing what you think about it and if you found it useful. Of course I also hope that you will be willing to give it a spin in your next Node.js project.

Ah, yeah, if you liked it don’t forget to share the love and “star” it on Github and Npm! ;)

Cheers!

Gulp and FTP: update a website "on the fly"

Luciano Mammino — Sun, 25 Oct 2015 18:47:00 GMT

In this post I will explain you how to use Gulp to interact with the files on your server through FTP. You will learn how to watch the files you are editing on your local machine and automatically synchronize them with your server as they change and how to setup a task to launch an FTP deploy on demand.

Premise

Few days ago I was assigned to take care of updating few sections on a legacy website which development environment is currently accessible only through FTP. So no local virtualized development environment, no Git, no automated deploy, only old-school “FTP live edit and test”!

I would have used an FTP desktop app to create a virtual drive to allow me to edit files easily and open the full project in Sublime, but I couldn’t find any good one for the last version of OS X (El Capitan), so I decided to work out a simple solution by myself using Gulp.

If you ever had a similar situation or if you just want to have a simple and fast command line solution to edit a website you can access to only through FTP, keep reading, this post is for you!

NOTE: I just want to stress on the fact that this solution must not be considered a best practice for development and deploy, but only a useful trick in case you have to deal with a very basic setup, especially if you are not in control of the development and deployment environments.

Installing Gulp

In case you don’t know it, Gulp is a very popular build system written in Node.js. It uses streams, so it’s quite efficient in dealing with data transformations and in managing streams of data from different sources.

To be able to run the script we are going to write you need to have Node.js and Gulp installed in your machine. If you don’t have Gulp already you can easily install it with NPM:

npm install -g gulp

Preparing the project

You will need to have a copy of the full project on your local machine, so if you don’t have it already you need to login to your server and download a copy of the source files. You can use any FTP software to do so, but if you are on a Mac there’s a quick trick that allows you to avoid to install any additional software: you can simply open finder and press ⌘ + k to start the connect to server window and specify the FTP url of your project with the following syntax: ftp://username:password@host:port/path. This will allow you to see and copy all the files from your project and, in case you are wondering, this is a read-only access!

Once you copied all the files you have to initialize NPM on the copied project with:

npm init

Follow the on screen instructions and provide all the missing informations.

Then you need to install Gulp also as a local dependency:

npm install --save-dev gulp

Also we are going to use a couple of external libraries, so let’s install them as well:

npm install --save-dev gulp-util vinyl-ftp

gulp-util is a module that offers a series of utilities functions for gulp (we will mainly use the log related functions) and vinyl-ftp is an FTP library that allows you to access FTP files and folders as streams and it’s a perfect match for Gulp.

Ok, now the last step before moving to some code is to initialize a new gulp project. To do so just run:

gulp init

It will create a new gulpfile.js file, let’s finally jump to some code.

The gulpfile

We are going to implement two different Gulp commands: ftp-deploy and ftp-deploy-watch. The first one allows us to deploy all the local changes to the server, the second one instead activates a live watch that constantly monitors our files and uploads immediately any change to the files, very useful when you want to live-code against a development server.

To enable these two commands you need to copy paste the following code inside your gulpfile.js file.

'use strict'

var gulp = require('gulp')
var gutil = require('gulp-util')
var ftp = require('vinyl-ftp')

/** Configuration **/
var user = process.env.FTP_USER
var password = process.env.FTP_PWD
var host = 'your hostname or ip address'
var port = 21
var localFilesGlob = ['./**/*']
var remoteFolder = '/myApp'

// helper function to build an FTP connection based on our configuration
function getFtpConnection() {
  return ftp.create({
    host: host,
    port: port,
    user: user,
    password: password,
    parallel: 5,
    log: gutil.log,
  })
}

/**
 * Deploy task.
 * Copies the new files to the server
 *
 * Usage: `FTP_USER=someuser FTP_PWD=somepwd gulp ftp-deploy`
 */
gulp.task('ftp-deploy', function() {
  var conn = getFtpConnection()

  return gulp
    .src(localFilesGlob, { base: '.', buffer: false })
    .pipe(conn.newer(remoteFolder)) // only upload newer files
    .pipe(conn.dest(remoteFolder))
})

/**
 * Watch deploy task.
 * Watches the local copy for changes and copies the new files to the server whenever an update is detected
 *
 * Usage: `FTP_USER=someuser FTP_PWD=somepwd gulp ftp-deploy-watch`
 */
gulp.task('ftp-deploy-watch', function() {
  var conn = getFtpConnection()

  gulp.watch(localFilesGlob).on('change', function(event) {
    console.log(
      'Changes detected! Uploading file "' + event.path + '", ' + event.type
    )

    return gulp
      .src([event.path], { base: '.', buffer: false })
      .pipe(conn.newer(remoteFolder)) // only upload newer files
      .pipe(conn.dest(remoteFolder))
  })
})

The Gulp script is very simple and pretty self-descriptive, but let’s try to have a look at it in greater detail.

After importing our modules, we have a series of configuration variables. The ones you have to set are host, port, localFilesGlob and remoteFolder.

host is the hostname or the IP address of your FTP server (e.g. myserver.com or 123.124.125.126)
port is the port where your FTP server is listening (generally it’s the default: 21)
localFilesGlob is an array containing one or more glob expressions. These expressions are used to determine which files from your local copy should be watched and deployed into the server. The default option is very “open”, it will copy all the files in your project folder. Most of the time it’s better to be more specific so, be sure to build your own custom array of glob to avoid copying unnecessary or sensible data.
remoteFolder is the folder in your remote server that contains the whole project (where the new files will be copied).

As you probably noticed we also have the username and password variables mapped to an environment variable. Is generally a good idea to not write passwords in configuration files, especially if you are going to publish this file somewhere. With this approach we can also share the same gulpfile with other people in the team and allow everyone to use their personal credentials.

After the configuration variables block we have the getFtpConnection function. This function is used as an helper to build an FTP connection using the previously specified options. You can also add some additional options, for instance parallel allows you to define the maximum number of files to upload in parallel.

At the end we have our two real Gulp commands: ftp-deploy and ftp-deploy-watch.

They simply defines a stream starting from the glob expressions and pipe it to the destination server through the FTP connection. The only difference between the two commands is that in the ftp-deploy we read all the files from the glob expressions and send them through the connection, in the second case we watch the files and send only the one that changes while command is running.

It’s also worth noticing that we are piping the files through the function conn.newer. This function ensures that only the files that are different from the server gets overwritten.

Running the commands

To run the commands you must first specify the FTP_USER and FTP_PWD environment variables. For example you can do it this way:

FTP_USER=someuser FTP_PWD=somepwd gulp ftp-deploy

FTP_USER=someuser FTP_PWD=somepwd gulp ftp-deploy-watch

Or if you prefer you can export the variables in your .bashrc or .bash_profile files (in this case I suggest you to rename the variables into something more specific to avoid collisions).

Conclusions

This approach allowed me to solve the problem easily and quickly having a mechanism that is quite fast to use and easy to maintain.

I am also more and more impressed about how easy is to have simple but effective solutions up and running with Node.js and the tools in its ecosystem. I am also impressed by the versatility of Gulp and how its stream-based approach makes things very concise and uniform.

I hope this article was useful and interesting for you. I’d really love to receive your opinions in the comments box about it and, in case you are a veteran Gulp user, I’m curious to know what was the most interesting command you have ever written using Gulp.

Until next time!

Regards

Introducing Gulp cozy - Manage your gulp tasks in a cozier way

Luciano Mammino — Mon, 18 Jan 2016 23:27:40 GMT

Gulp-cozy is an experimental NPM package that allows you to organize your Gulp tasks in a more modular way with simplicity and “cozyness” in mind.

Rationale

Ever found yourself digging into a gigantic monstrous Gulpfile with hundreds of functions and tasks scattered all around? If it happened to you I am sure you can tell that it’s not a great feeling…

That’s the reason why I created this small module as an attempt to help with keeping yourself cozier (and happier!) when working with Gulp. In a way it tries to bring a bit of the Node philosophy (also known as “The Node way”) into your Gulpfile.

Gulp-cozy in fact offers a very easy way (read: very-very-very-easy!) to separate all your Gulp tasks into small modules organized inside a dedicated folder. Gulp-cozy will take care to load all the modules and to register them as Gulp tasks. With this approach you will end up with several small modules that serve one specific purpose (a task), which in turn result easier to maintain and to reason about.

Installation and usage

Enough talking, let’s jump to the practice! As most of the NPM packages to install gulp-cozy you just need to run the following command inside your project folder:

npm install --save-dev gulp-cozy

Of course I am assuming you have already installed Gulp globally and as local dev dependency for your project.

Once NPM finished downloading (it should be fast, the module is pretty small!) you can turn you Gulpfile.js into this:

var gulp = require('gulp');
var cozy = require('gulp-cozy');
cozy(gulp);

YES! This will be the only content of your Gulpfile… So you might ask “where do all the tasks logic go?”

The idea is to keep all the tasks inside a gulp folder in your main project folder (doesn’t it make you think to the node_modules folder approach?!). Every task is a file which name will represent the name of the gulp task.

Don’t worry too much for now, let’s just create this new gulp folder and everything will be cleaner in a moment with a more practical example.

Let’s assume we want to create a new task to build the css files for our new project. For this sake we can add the build-css task by creating the build-css.js file as follows:

var concat = require('gulp-concat');
var minifyCss = require('gulp-minify-css');

module.exports = function(gulp) {
  return function() {
    return gulp.src([
      './node_modules/bootstrap/dist/css/bootstrap.css'
    ])
      .pipe(concat('all.css'))
      .pipe(minifyCss({compatibility: 'ie8'}))
      .pipe(gulp.dest('./assets/'))
    ;
  }
}

Notice that the module exports a function that receives the current instance of Gulp as argument. This function is a factory for the real Gulp task logic so it should return a function which, will be executed when calling the build-css task with:

gulp build-css

You can also create a module to call a series tasks as in the following example.

module.exports = ['clean', 'build-css', 'build-js', 'compress', 'upload'];

In this case, rather than using a factory function, the module needs to export just a plain array containing the names of the tasks to be invoked. You can launch this new task with:

gulp build

What do you think? Doesn’t it make things more separated, organised and most importantly “cozy”?

Long way to perfection

As I told in the very first line this approach has to be considered experimental for now and there are still many concerns that I have about it.

First of all the module tries to expose a very small surface and to do one simple thing. That’s generally good and it sticks to the node philosophy. But at the same time it’s probably not very flexible.

Would be great to have a way to define generic task templates, like for example processing CSS files (compile from sass, concatenate and minify) and than have a configurable way to use this template with different sets of files and producing different assets.

At the moment you could do this by creating a generic configurable module outside your gulp folder and than require it and use it inside one or more tasks in the new folder. This approach should work without big efforts but doesn’t feel very right to me at the moment.

Probably the module should expose more functions to deal with more complex case, but it boils down to the eternal struggle between “keeping things simple” and “making the new jack-of-all-trades module”…

Contributing

If you want to contribute to this project you can find the sources on GitHub (of course!). If this project is interesting to you you are very welcome to contribute by submitting a pull request or simply by opening an issue.

If you decide to go with the pull request, please take care to maintain the existing coding style and add unit tests for any new or changed functionality. The project is currently using XO for styleguide and style checks run with the regular test suite.

Conclusion

I am really curious to see if this topic and this library can be of any interest for other people. I’d like to know if you ever questioned yourself to find better solutions to organize your Gulp tasks and if you came out with some kind of personal solution. I would be very happy to see some comments down this article, maybe a nice discussion might born from this.

Before leaving you I have to thank my dearest friend Andrea for discussing these topics and give me the inspiration to spend some time on this subject.

Until next time! :)

Get an invitation for GitKraken

Luciano Mammino — Tue, 19 Jan 2016 21:10:23 GMT

Axosoft is currently working on GitKraken an innovative multi-platform graphical client for git which is already available for Windows, Linux and Mac OSX in private beta version.

The private beta is accessible through an invite code. I was lucky to get an invite for me and ~~I have~~ some extra invites that I am willing to share with the first people who is going to leave a comment on this post.

UPDATE 2016-01-19 22:00 GMT: I finished all the invites in less than 1 hour! That’s incredible :O Sorry for whoever remained out, I’ll let you know if I manage to get more invites!
UPDATE 2016-01-19 23:00 GMT: Seems that I was very very lucky and I got some NEW INVITES, keep commenting to get them! ;)
UPDATE 2016-01-23 09:00 GMT: Invites are over!
UPDATE 2016-02-03 18:50 GMT: Yesterday GitKraken 0.6 was released as Public Beta and it’s free to be downloaded on the official website.

What is GitKraken

As I said, GitKraken is a new shiny graphical interface for git. I am a big fan of git from command line and even if I used a number of different GUIs in the past I always switched back to the console. By the way, GitKraken looks promising thanks to a set of interesting features. Let’s have a look at the ones that impressed me most after a first try.

Network view

This is probably the killer feature at the moment, a nice-looking graphical view of your git network that allows you to understand what is the current status of you repository in terms of commits and branches. Every dot in the graph represents a commit and it is interactive. When you select one of them you can see all the changes applied by that commit, who pushed it, the commit description and, most importantly, if you right click on it you can immediately perform a number of actions which are notoriously not so trivial from the command line:

cherry-pick
create a branch from the specific commit
create a new tag pointing that commit
reset master to that commit
edit commit message

Integration with GitHub

GitKraken allows you to connect with your GitHub social account and easily access and clone from your remote repositories or fork an existing repository.

This is a lovely feature that will save you some amount of time if you are an active open source contributor or if your organisation is using GitHub for business.

The integration is currently limited to cloning, but I hope it will be improved to support also pull requests and issues (which would be definitely amazing!).

Also it’s worth mentioning that the support for BitBucket seems to be coming soon!

Branching and Stashing

There are a couple of other interesting operations that are made trivial by GitKraken… I am talking about branching and stashing. By just clicking a button you will be able to create a new branch starting from the current head or from a specific commit and you can also stash your current uncommitted changes. When you stash something you will see a dot in your network graph that represents the stashed data as if it was a commit and you will be able to unstash that changes at any time later in the future. Personally I believe these features can simplify the adoption of a Git branching model as GitFlow within your team.

Built with Electron and Node.js

While this is not a big differentiator at the moment (apart from making it immediately cross platform), I am quite sure that this feature will allow GitKraken to offer a nice extensibility layer offering people a way to write their own plugins/integrations. Just look at recent softwares like nylan/N1 and Atom to understand what is possible with this technology and why developers are loving these products.

Conclusion

I think it’s still to early to form a strong opinion about this product, but the premises are excellent and I would say we can expect great things for the first public release. I hope Axosoft will decide to keep it free as Atlassian is doing with SourceTree and that this choice might be another incentive to allow people to adopt or get even more familiar with Git.

I am curious to know your opinion about it and if you prefer to use a GUI rather than the command line, so please leave a comment.

Also remember to ~~leave a comment down below this post if you need an invitation to try it~~ (invites over)!

Until next time :)

6 Tips to Build Fast Web Applications (Php Dublin March 2016 Talk)

Luciano Mammino — Wed, 23 Mar 2016 09:16:00 GMT

Following one of my previous posts about performance in web applications, yesterday night I had the chance to give a related talk at the Php Dublin user group.

My talk was preceded by a great talk (and demo) by Ricardo Melo (@rjsmelo) called Docker and Php, a practical use case. If you are interested in the topic, be sure to check it out!

It was a great night, I learned a lot and I had a lot of fun, so thank you to everyone involved.

If you are interested in having a look at the slides, here they are:

http://slides.com/lucianomammino/6tips-web-perf

Feel free to leave your opinion in the comments.

Cheers!

Extracting data from Wikipedia using curl, grep, cut and other shell commands

Luciano Mammino — Mon, 15 Aug 2016 17:17:00 GMT

In this article I am going to show you how I was able to extract and process some information from Wikipedia only using a combination of common bash utilities like curl and grep.

The need

If you are a sport lover like me I guess your heart is currently being warmed by the Rio 2016 Olympic games. My favourite sport in the games is Judo and now that the competitions are over I was wondering who were the best olympic “judokas” of all the times by number of medals collected during the games (no matter the kind of medal). I tried to google the answer for a while but it wasn’t easy to find an up to date result, so I decided to do some quick research and trying to get to a conclusion by myself. I have to say it was I bit tougher than I expected, but it was definitively fun…

The dataset

The first thing I needed was a reliable and up to date data source listing all the Judo Olympic medal winners in history. This was easy to find on Wikipedia: List of Olympic medalists in Judo. Anyway the data on this Wikipedia page is structured to be easy to read by humans and not to be processed by a machine. Also I didn’t found any way to have the same data in the page in a csv or json format, so the only viable option was to extract the data by myself from the web page. At first I thought about creating a quick and dirty JavaScript command and use some library like cheerio to extract the data directly from the HTML code of the page, but it sounded like to much of work for the simple goal I had in mind. So I had another quick look at Wikipedia to find out if there was any better format to extract the information. Going to the edit option of the page I realized that parsing the wikitext (source) of the page would have been much easier and I could even use a regular expression to extract the relevant information from there. At this stage I wondered if there was a way to get only the wikitext of a specific Wikipedia page. It turns out that it’s possible and it’s very easy: you just need to append the query parameter ?action=raw in the url!

So hitting https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_judo?action=raw will give us our starting dataset which will look like this:

==Men==
===Extra Lightweight===
*60&nbsp;kg
{| {{MedalistTable|type=Games}}
|-
|rowspan=2|[[Judo at the 1980 Summer Olympics|1980 Moscow]]<br>{{DetailsLink|Judo at the 1980 Summer Olympics – Men's 60 kg}}
|rowspan=2|{{flagIOCmedalist|[[Thierry Rey]]|FRA|1980 Summer}}
|rowspan=2|{{flagIOCmedalist|[[José Rodríguez (judoka)|José Rodríguez]]|CUB|1980 Summer}}
|{{flagIOCmedalist|[[Tibor Kincses (judoka)|Tibor Kincses]]|HUN|1980 Summer}}
|-
|{{flagIOCmedalist|[[Aramby Emizh]]|URS|1980 Summer}}
|-
|rowspan=2|[[Judo at the 1984 Summer Olympics|1984 Los Angeles]]<br>{{DetailsLink|Judo at the 1984 Summer Olympics – Men's 60 kg}}
|rowspan=2|{{flagIOCmedalist|[[Shinji Hosokawa]]|JPN|1984 Summer}}
|rowspan=2|{{flagIOCmedalist|[[Kim Jae-Yup]]|KOR|1984 Summer}}
|{{flagIOCmedalist|[[Neil Eckersley]]|GBR|1984 Summer}}
|-
|{{flagIOCmedalist|[[Edward Liddie]]|USA|1984 Summer}}
|-
...

Note that, for the sake of brevity, from now on I will use ... to indicates that there’s a lot of data that was stripped from the example.

We can easily get this data into our bash shell with curl:

curl -sS "https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_judo?action=raw"

In case you don’t know the options -sS allow to strip out the download progress output and just print the downloaded data (or any possible error) in the console.

Now that we have our dataset, to compute the result we need to:

Extract all the athletes names
Count the occurrences of each one of them
Reverse order them by the resulting count.

We will do so with a combination of bash commands, bound together using the pipe operator (|). Pipes let you use the output of a program as the input of another one, effectively creating a stream of data that is transformed step by step by small and easy to understand operations.

Extracting data with `grep`

The dataset shown before is just the wikitext code needed to render the tables of athletes that won medals by category and year. So we can easily assume that all the athletes listed in the page are interesting for us.

As you can see every athlete is referenced in the code using the template flagIOCmedalist and every entry looks like:

{{flagIOCmedalist|[[NAME]]|COUNTRY|OLYMPIC GAME}}

So we can easily extract all the athletes names with a regex like the following:

flagIOCmedalist\|\[\[(.+)\]\]

This regular expression behaves as described in the following picture:

If you want to understand all the details of this regex you can play with it on regex101.

In the shell we can apply a regex to an input data using grep. We need to concatenate the output of curl to grep using the pipe operator (|):

curl ... |  grep -Eoi "flagIOCmedalist\|\[\[(.+)\]\]"

With the grep we are using the options -Eoi which allow us to:

-E: use a POSIX extended regular expression (ERE)
-o: print only the matched (non-empty) parts of a matching line
-i: case insensitive matching (this is not really needed for our problem, but could allow us to match any case variation of the template reference like flagiocmedalist and FLAGIOCMEDALIST).

The previous command is going to output something like this:

flagIOCmedalist|[[Thierry Rey]]
flagIOCmedalist|[[José Rodríguez (judoka)|José Rodríguez]]
flagIOCmedalist|[[Tibor Kincses (judoka)|Tibor Kincses]]
flagIOCmedalist|[[Aramby Emizh]]
...

Which, of course, looks kind of ugly. We ideally want to have only the name of every athlete and not all the template syntax wrapping it.

Let’s try to clean up the data a little bit more.

Cleaning up the data with `cut`

cut is another interesting command line utility that can be used in a whole lot of different ways to extract substrings (or columns) from text.

Using cut we can remove all the unneeded text in 3 steps:

remove the prefix flagIOCmedalist|[[
remove the suffix ]]
clean up the names with the “(judo)” disambiguation note. For example from José Rodríguez (judoka)|José Rodríguez we want to keep only José Rodríguez.

Substring with `cut`

The complete the first step (removing the prefix flagIOCmedalist|[[) we can use the following command:

... | cut -c"19-"

The option -c"19-" means: “take the substring that starts from the character number 19 to the end of the string”. Character 19 is the 20th character considering that strings are “0-indexed”. You can use the -c option to extract any generic substring. For example -c"4-8" will extract the substring from the 5th to the 9th character. If you leave the range open on the right it will take all the remaining string to the end.

This filter will be applied to every line and it will modify our stream of data as follows:

Thierry Rey]]
José Rodríguez (judoka)|José Rodríguez]]
Tibor Kincses (judoka)|Tibor Kincses]]
Aramby Emizh]]
...

Splitting strings with `cut`

Now we would like to remove the suffix ]]. In order to achieve this goal we can use:

... cut -d \] -f 1

This time we are using the options -d and -f. When we use d we can specify a character to be used as delimiter (] in this case), this way cut will not split the string by character index but in chunks (or columns), generating a new chunk every time the delimiter is encountered along the line. The option -f can be combined with -d to select one or more chunks, in this case we want to select the first chunk. As we can expect, this will be the status of our data after this command is executed in the pipeline:

Thierry Rey
José Rodríguez (judoka)|José Rodríguez
Tibor Kincses (judoka)|Tibor Kincses
Aramby Emizh
...

The data looks almost clean, we just need to get rid of the occasional disambiguation notes. These cases can be distinguished by the character | (pipe) that separates the disambiguation definition from the text that must be rendered by the wikitext engine into HTML code. In these cases we want to keep only the part of the string after the pipe character. Again we can use cut with the options -d and -f:

... | cut -d \| -f 2

This time we are using the pipe character as delimiter and we are taking only the second chunk of the string. This might seem obvious at this stage, but what happens in all the rows where there’s no disambiguation note (and no pipe character)? In this cases we will have just one chunk containing the full line, so what do we get by using the option -f 2? Luckily in these cases cut is smart enough to assume the string is not matching our pattern and returns the entire line, so our data is not destroyed.

The final result after this command is:

Thierry Rey
José Rodríguez
Tibor Kincses
Aramby Emizh
...

Our data is finally clean! 🎉

Counting and sorting with `uniq` and `sort`

Now that we have our clean list of names we need to count the occurrences of every athlete and sort. In order to remove duplicates (and count the occurrences) we can use the command uniq. This command, when fed with some text, outputs the text itself with adjacent identical lines collapsed to one. This means that before we can use uniq we need to have all the names sorted alphabetically so that all the athletes with multiple occurrences have their name repeated on multiple lines one after another. This can be achieved with the command sort, which, as you might easily guess, just sorts all the lines received as input in alphabetical order.

Just to give you a practical example, after using the command sort our data will look like:

...
Paweł Nastula
Peter Seisenbacher
Peter Seisenbacher
Priscilla Gneto
Qin Dongya
Radomir Kovačević
Rafael Silva
Rafael Silva
Rafaela Silva
Ramaz Kharshiladze
...

Now we can pipe the uniq -c command to our data processing flow in order to remove duplicates. The option -c prints in front of every line the number of consecutive occurrences originally found for that line. After this command we will have something like this:

...
1 Paweł Nastula
2 Peter Seisenbacher
1 Priscilla Gneto
1 Qin Dongya
1 Radomir Kovačević
2 Rafael Silva
1 Rafaela Silva
1 Ramaz Kharshiladze
...

Note that Rafael Silva and Rafaela Silva are two different athletes, it’s not a typo! 😂

We are almost done. Now we just need to sort our data once again, but this time in reverse order. To do so we can use the sort command once again, this time as follows:

sort -nr

Where the option -n specifies that the first part of every line is a number (so the command can effectively distinguish and sort lines starting for example with 1 and 10). The option -r instead indicates that we want to sort in revers order (bigger first).

This concludes our pipeline of commands!

Combining all together

Combining all the commands together, our final pipeline will be the following:

curl -sS "https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_judo?action=raw" |\
 grep -Eoi "flagIOCmedalist\|\[\[(.+)\]\]" |\
 cut -c"19-" |\
 cut -d \] -f 1 |\
 cut -d \| -f 2 |\
 sort |\
 uniq -c |\
 sort -nr

Now that you understood every single part it shouldn’t look very cryptic.

If we execute it, we can finally discover who are the best Judo Olympic athletes up to Rio 2016:

… drum rolls …

With 4 medals:

With 3 medals:

Lot’s of French people there, uh! It’s a shame no Italian athlete is there yet! 😉

Conclusion

I hope this article showed you that the unix shell is a very powerful tool. If you learn its basic commands and the most common options you will be able to complete a hell of a lot of task directly from the command line by just wisely combining them. You will not need to open an editor and write a long script and then to have an interpreter like Node.js to run it, sometimes the command line is just enough you need!

If you want to experiment a bit more with this topic, I can propose you a nice variation of this tutorial as exercise:

Can you compute the ranking of the nations with the highest number of medals?

If so, please write your solution in the comments 🤓

Until next time!

Cheers

Alternative approaches

EXTRA (Updated on Wed 29th August 2016)

Some people are suggesting interesting alternative solutions. I think it’s nice to list them here.

Improved `grep` with the reset match operator

In the newest versions of Gnu grep (not currently available on Mac!) is possible to use the \K operator (reset match).

\K resets the starting point of the reported match. Any previously consumed characters are no longer included in the final match. (See the complete regex on regex101)

With this approach we can rewrite our command as follows, effectively getting rid of all the cut commands in the pipeline:

curl -sS "https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_judo?action=raw" |\
 grep -Po 'flagIOCmedalist\|\[\[(.* \(judoka\)\|)?\K[^\]]*' |\
 sort |\
 uniq -c |\
 sort -nr

Solution suggested by pakistanprogrammerclub in the comments

quget

quget is a command line utility authored in Node.js that brings together the power of famous node packages such as request, cheerio, and jQuery-like CSS selectors to the command-line.

To run the following command you need to have Node.js installed and to install quget with npm i -g quget:

quget https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_judo "#mw-content-text table.wikitable tr:nth-child(n+2) td:nth-child(n+2) a:first-of-type, #mw-content-text table.wikitable tr:nth-child(n+2) td:only-child a:first-of-type" | sort | uniq -c | sort -nr

Solution by ɹɐqooɟ in the comments.

shapeshed on reddit mentioned pup an alternative command written in Go.

awk

awk is a programming language designed for text processing and typically used as a data extraction and reporting tool. It is a standard feature of most Unix-like operating systems (cit. Wikipedia).

curl -sS "https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_judo?action=raw" | awk -F'\|\\\[\\\[|\\\]' '/flagIOCmedalist/{a[$2]++} END {for (i in a) print a[i], i | "sort -r"}'

This didn’t work for me on a Mac, but it did work on an Ubuntu machine.

Solution by Boris P. on Facebook.

sed

sed (stream editor) is a Unix utility that parses and transforms text, using a simple, compact programming language. (See more on Wikipedia).

We can use sed as an alternative to grep to extract the names of the judokas in one shot. Our final command will look like:

curl -sS "https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_judo?action=raw" | sed -n '/flagIOCmedalist/{s/^.*\[\[//; s/\]\].*//; s/^.*|//;p}' | sort | uniq -c | sort -nr

This approach doesn’t seem to be working on Mac tough.

Solution suggested by pakistanprogrammerclub in the comments

In-Browser developer console

You can extract data from the current web page using the developer console that most browsers offer (Chrome, Opera, Firefox…). After opening our Wikipedia page of interest, in the developer console you can run the following JavaScript one-liner:

;[].slice
  .call(
    document.querySelectorAll(
      'table tr td:nth-child(n+2) > a:nth-child(1), table tr:nth-child(3) td > a:nth-child(1)',
    ),
  )
  .map(function (e) {
    return e.innerText
  })
  .reduce(function (res, el) {
    res[el] = res[el] ? res[el] + 1 : 1
    return res
  }, {})

The result is an object with the medalists as keys, and the count as values. JavaScript objects are unordered so sorting is left as an exercise for the reader.

Solution by lacksconfidence on HackerNews

XPath and Google Spreadsheet or Python

XPath (XML Path Language) is a query language for selecting nodes from an XML document (cit. Wikipedia).

By downloading the HTML of the Wikipedia page and applying an XPath selector like //table//tr/td[2]/a[1]/text() we should be able to extract all the gold medalist in the tables. In the same fashion we can build a slightly more complex solution by combining more selectors:

//table//tr/td[2]/a[1]/text()|//table//tr/td[3]/a[1]/text()|//table//tr/td[4]/a[1]/text()

With this approach we can easily import data into a spreadsheet software like Google Spreadsheet. In Google Spreadsheet we can use the function IMPORTXML to run an XPath expression against an XML or HTML document available on a given URL.

I created an example document that you can check out if you are curious to see this feature in action.

Solution based on comments by turtlebits and san_dimitri on HackerNews.

I’m glad to know I inspired Nikolai Hampton to write an amazing article that illustrates how to solve this problem using XPath expressions in Python. If you like Python the article is absolutely a must, don’t miss it out!

React on the Server for Beginners: Build a Universal React and Node App

Luciano Mammino — Mon, 29 Aug 2016 21:49:47 GMT

I recently wrote a new article in collaboration with one of my favourite web development websites: the amazing Scotch.io.

The article talk about using React both on client and on the server (with Node.js) in an attempt to build a simple “Universal JavaScript” (a.k.a “Isomorphic”) application.

The application is called “Judo Heroes” (did you even noticed recently that I’m kind of fond of Judo?) and it showcases some of the best Judo athletes for their number of medals won during the Olympic Games and in other important international tournaments.

If you are curious you can check out the live demo or have a look at the source code on GitHub.

Then, if you want to follow the tutorial, you should definitely read the article on scotch:

React on the Server for Beginners: Build a Universal React and Node App

In this article we are going to learn how to build a simple “Universal JavaScript” application (a.k.a. “Isomorphic”) using React, React Router and Express. We will also use some Webpack and Babel to leverage the latest ES2015 Syntax.

Enjoy it and feel free to leave your comments here or on the original article.

Cheers!

To promise or to callback? That is the question...

Luciano Mammino — Sun, 14 Feb 2016 13:32:00 GMT

You are building the next cool JavaScript or Node.js module with a lot of asynchronous functions and you are very happy about it. At some point a terrible doubt assaults you:

Should my API offer support for callbacks or should it be promise based?

In this article we are going to show a very simple way to add support for both promises and callbacks in our asynchronous modules, this way we can make everyone happy and our libraries much more flexible.

The problem

Promises can be used as a nice replacement for callbacks, that’s a well know fact in the JavaScript world today. Promises turn out to be very useful in making our code more readable and easy to reason about. But while promises bring many advantages, they also require the developer to understand many non-trivial concepts in order to use them correctly and proficiently. For this and other reasons, in some cases it might be more practical to prefer callbacks over promises.

Now let’s imagine for a moment that we want to build a public library that performs asynchronous operations. What do we do? Do we create a callback oriented API or a promise oriented one? Do we need to be opinionated on one side or another or there are ways to support both and make everyone happy?

There are at least 2 approaches to face this question, let’s see how they work!

##1. The “I don’t fu**in’ care” approach

The first approach, used for instance by libraries like request, redis and mysql as well as all the node native async functions, consists in offering a simple API based only on callbacks and leave the developer the option to promisify the exposed functions if needed. Some of these libraries are a bit more elaborated and they provide helpers to be able to promisify all the asynchronous functions they offer at once, but the developer still needs to someway “convert” the exposed API to be able to use promises. That’s why I believe this approach feels a bit rude like:

Do you want to use Promise? I don’t care, it’s your problem… just promisify what you want and leave me alone!

There are a number of modules out there that can help to promisify a callback based function. The first two that come in my mind are Bluebird with its Promise.promisify() method and es6-promisify which adopts ES2015 promises.

Let’s see a quick example of this approach with es6-promisify. In the following snippet of code we are going to promisify the native fs.readFile() function:

'use strict'

const fs = require('fs')
const promisify = require('es6-promisify')

let readFile = promisify(fs.readFile)

readFile('last_action_hero.txt', 'utf-8')
  .then(content => console.log(content))
  .catch(err => console.error(err))

The example is very straightforward, we just have to call promisify(fs.readFile) to obtain a promisified version of the readFile function. As we might expect we can invoke this function without passing the callback argument and we get a Promise object as output, so we can immediately use it and call the handy then and catch methods.

Unfortunately the new Promise implementation of ES2015 does not offer a built-in promisify mechanism (yet…).

2. The “No strong feelings” way

This approach is more transparent and I would say more… “polite”! It is also based on the concept of offering a simple callback oriented API, but it makes the callback argument optional. Whenever the callback is passed as an argument the function will behave normally executing the callback on completion or on failure. Instead when the callback is not passed to the function, it will immediately return a Promise object.

This approach effectively combines callbacks and promises in a way that allows the developer to choose at call time what interface to adopt, without any need to promisify the function in advance. Many libraries like mongoose and sequelize are supporting this approach.

Let’s see a simple implementation of this approach with an example. Let’s assume we want to implement a dummy module that executes divisions asynchronously:

module.exports = function asyncDivision(dividend, divisor, cb) {
  return new Promise((resolve, reject) => {
    // [1]

    process.nextTick(() => {
      if (
        typeof dividend !== 'number' ||
        typeof divisor !== 'number' ||
        divisor === 0
      ) {
        let error = new Error('Invalid operands')
        if (cb) {
          return cb(error)
        } // [2]
        return reject(error)
      }

      var result = dividend / divisor
      if (cb) {
        return cb(null, result)
      } // [3]
      return resolve(result)
    })
  })
}

The code of the module is very simple, but there are some details (marked with a number in square brackets) that are worth to be underlined:

First, we return a new promise created using the ES2015 Promise constructor. We define the whole logic inside the function passed as argument to the constructor.
In case of error, we reject the promise, but if the callback was passed at call time we also execute the callback to propagate the error.
After we calculate the result we resolve the promise, but again, if there’s a callback, we propagate the result to the callback as well.

Now to complete the example, let’s see now how we can use this module with both callbacks and promises:

// callback oriented usage
asyncDivision(10, 2, (error, result) => {
  if (error) {
    return console.error(error)
  }
  console.log(result)
})

// promise oriented usage
asyncDivision(22, 11)
  .then(result => console.log(result))
  .catch(error => console.error(error))

It should be clear that with very little effort, the developers who are going to use our new module will be able to easily choose the style that best suits their needs without having to introduce an external “promisification” function whenever they want to leverage promises.

A little caveat and an alternative implementation

As Amar Zavery pointed out in a comment here, this approach is not perfect.

If we use the function with the callback approach, we still have created and returned a promise that behaves in an inconsistent way (never resolved, nor rejected). Let’s make this clear with an example:

asyncDivision(2, 0, console.log)
  .then(() => console.log(`Promise resolved`))
  .catch(() => console.error(`Promise rejected`))

In this example, we are passing a callback (console.log) to our function but also using the returned promise to print some extra information. With our implementation of asyncDivision either the then block and the catch block are never executed because the function is interrupted as soon as the callback is used.

This is most of the time negligible because is very unlikely that somebody will be using the function with this mixed async approach (we could even argue that doing so would be a bad practice).

Anyway, we can get rid of this issue by re-implementing our async function as follows:

module.exports = function asyncDivision(dividend, divisor, cb) {
  // internal implementation, callback based
  function _asyncDivision(dividend, divisor, cb) {
    process.nextTick(() => {
      if (
        typeof dividend !== 'number' ||
        typeof divisor !== 'number' ||
        divisor === 0
      ) {
        return cb(new Error('Invalid operands'))
      }

      return cb(null, dividend / divisor)
    })
  }

  if (cb) {
    return _asyncDivision(dividend, divisor, cb)
  }

  // optional promisification, only if no callback
  return new Promise((resolve, reject) =>
    _asyncDivision(
      dividend,
      divisor,
      (err, result) => (err ? reject(err) : resolve(result))
    )
  )
}

This code generates a promise only if no callback is used. So if we provide a callback as the last argument, and then we try to use catch or then on the returning value, we will get an explicit runtime error:

TypeError: Cannot read property 'then' of undefined or
TypeError: Cannot read property 'catch' of undefined

While this approach is probably more correct, it is also more verbose and, if it becomes a recurrent practice in your development process, you are better off using some promisification library.

Conclusion

As usual I hope this article has been useful and that it will ignite some interesting conversation.

I think from the tone of the article it’s quite clear that I prefer to opt for the “polite” approach, but I am very curious to know what are your opinions about this topic and if you are an “I don’t care” or a “whatever” person when you create your asynchronous functions in your modules.

Let me know your thoughts in the comments.

Until next time! :)

Announcing the book "Node.js design patterns - second edition"

Luciano Mammino — Sun, 31 Jul 2016 18:25:00 GMT

In this post I am going to introduce my last side-project: “Node.js design patterns — second edition”, a technical book about Node.js and design patterns published by Packt.

This blog has been a very quite place in the last 8-9 months, this is because almost all my free time has been absorbed by one of my most ambitious side projects: co-writing a book about Node.js and design patterns!

I am finally happy to announce that “Node.js design patterns — second edition” has been published.

You can keep reading this post to find out more or have a look at the official website. If all of this is convincing enough you can even decide to buy the paperback or the e-book on Packt, Amazon or O’Reilly :P

The making of

As you might have noticed, this book is the second edition of an existing book on the topic — I dare to say one of the best books on the topic — that Mario Casciaro put together a couple of years ago.

I was contacted by Mario in November last year because he was looking for somebody willing to help him in creating a new edition of the book. I don’t know yet why he thought I might have been the right person (and I am still flattered about this) but, being his book one of my favourites, I almost immediately decided to accept the challenge and work with him and Packt on this new edition.

It took almost 9 months from the first idea to having the printed book on my desk. I have to admit it was an incredibly interesting experience for me. I learned a lot and I definitely feel I strengthened my knowledge of Node.js and design patterns, I worked with super skilled technical reviewers like Joel Purra and Tane Piper and with the amazing team at Packt. I also had chance to experiment with some new cool technologies like React and new ways of building websites like Universal JavaScript.

Content of the book

The book has been heavily updated: more than the 50% of the original content from the first edition was changed. These changes were mostly about updating all the code examples to Node v6 and EcmaScript 2015, upgrading all the obsolete content, introducing some new interesting examples and design patterns and a new entire chapter dedicated to Universal Javascript.

The final result counts 11 chapters, more than 500 pages and more than 100 code examples.

The first two chapters provide basic informations about the Node.js platform and the new features introduced with EcmaScript 2015.

Chapter 3 and 4 go deep into the topic of asynchronous programming exploring different approaches (callbacks, promises, events, async/await) and design patterns to exploit all the most common scenarios.

Chapter 5 discusses one of the most important patterns in Node.js: streams. It shows you how to process data with transform streams and how to combine them into different layouts.

Chapter 6 is probably the core of this book diving deep into the most popular conventional design patterns and showing how unconventional they might look in Node.js. It also introduces the reader to some emerging design patterns that are used only with JavaScript and Node.js.

Chapter 7 analyses the different solutions for linking the modules of an application together investigating design patterns such as Dependency Injection and the service locator pattern.

Chapter 8 is an entire new chapter that explores one of the most interesting capabilities of modern JavaScript web applications: being able to share application code between the frontend and the backend. Across this chapter we learn the basic principles of Universal (a.k.a. Isomorphic) JavaScript by building a simple web application with React, Webpack, and Babel.

Chapters 9, 10 and 11 go into more advanced and enterprise topics like “Advanced Asynchronous Recipes”, “Scalability and Architectural Patterns” and finally “Messaging and Integration Patterns”.

You can read a sample of the book containing the full table of contents and the first chapter for FREE on the official website, don’t miss it!

Acknowledgements

Apart from Mario, Joel and Tane, which I already mentioned previously, I have to spend a special word of thank you for my dear friends Anton Whalley, Alessandro Cinelli, Andrea Giuliano and Andrea Mangano. They are amazing guys and they have been precious supporters during the creation of this new book with words of encouragement and incredibly meaningful technical advices.

Thank you once again guys :)

What’s next

Well, I don’t plan to start writing a new book anytime soon in case you are wondering… :D But I surely will keep improving my knowledge and experience with Node.js and maybe I will also start to explore some new field in the amazing world of programming. I guess I will also be able to keep posting articles here more often :)

So stay tuned for the next post :)

I will go now to take a pint to celebrate the release… Cheers 🍻

Two Interviews about Node.js, JavaScript and being a book author

Luciano Mammino — Sun, 25 Sep 2016 13:24:08 GMT

In the last month I had the pleasure to seeing published two interviews regarding my life as developer with Node.js and JavaScript and about being a book author with Node.Js design patterns second edition.

The first interview was made with the Packt Publishing team and it was published in their own website:

With Node.js, it’s easy to get things done

The second interview was made by Thomas Peham, marketing manager at UserSnap:

Meet the community: Luciano Mammino, software developer from Sicily

I think both interviews turned out to be pretty cool, especially because I had a chance to express my vision about the future of JavaScript and Node.js, technologies on which I am investing most of my time.

Feel more than welcome to let me know if you share my vision or not here in the comments.

Until next time!

How to crack a JWT token: two articles about distributed computing, ZeroMQ & Node.js

Luciano Mammino — Mon, 24 Oct 2016 22:44:51 GMT

In the last 2 weeks I add the pleasure to release an article (in two parts) in collaboration with RisingStack, one of the most famous companies in the Node.js ecosystem.

The article explains how to build a distributed application using Node.js and ZeroMQ and provides an example that I believe it’s very actual and interesting: a JWT token cracker.

If you are into Node.js, ZeroMQ, security or distributed application you can read the two articles in the community section of RisingStack.

ZeroMQ & Node.js Tutorial - Cracking JWT Tokens (Part 1.), is focused mostly on theory and describes what a JWT token is and what will be our approach to try to crack one of them.

ZeroMQ & Node.js Tutorial - Cracking JWT Tokens (Part 2.), instead, focuses on coding and it’s a step by step guide on how to build the working application.

The application is also freely available on NPM and GitHub in case you want to simply download it or contribute.

My Universal JavaScript Web Applications talk at Codemotion Milan 2016

Luciano Mammino — Sat, 26 Nov 2016 09:53:55 GMT

Yesterday (25/11/2016) I had the pleasure of delivering a talk about Universal JavaScript at Codemotion Milan 2016. It was a great fun and I was very happy to meet such a big pool of technology enthusiasts and professionals all in one places. I’m already looking forward for the next edition!

Video

Update (12-12-2016): a video recording of the talk has been published by the folks at Codemotion:

Slides deck

If you are interested in my talk you can find the slides on Slides.com:

Commentary

Here follows also a quite detailed kind of commentary for every slide (from my notes), so you can easily understand all the concepts discussed during the presentation.

1.

Hello everybody, I am Luciano (also known as “lmammino” or “loige” on GitHub and Twitter). I am a software engineer at Planet9 Energy, a new electricity provider that is focused on using new technologies to give industrial and consumer customers more visibility over their bills and at the same time enabling new ways to perform energy trading very efficiently.

Me and my team are currently building most of the infrastructure using a serverless approach and we make heavy use of AWS lambda and the serverless framework. If you are interested in these topics please catch up with me after this talk.

I am also the co-author of the book “Node.js Design Patterns Second Edition” together with Mario Casciaro. If you are interested in re-enforcing your knowledge of design patterns, and most importantly discover how you can get the best out of them in the Node.js and JavaScript land, this book is probably for you. For what concerns this talk, the book also features an entire chapter about Universal JavaScript.

2.

So today we are going to be talking about Universal JavaScript for Web applications.

First of all we will see what “universal” really means, then we will discuss who is currently using it and why. Then we will move into a quick overview of the main challenges to face when dealing with universal JavaScript applications. Finally, in the second part of this talk, we will see a practical example about how to build a Universal JavaScript Web Application. At first we will build the front-end only version and then by making it “universal”.

3.

So, for some time, what we call today Universal JavaScript was recognised as “Isomorphic” JavaScript. After some discussions it seems that the JavaScript community agreed on using the term “Universal” instead. If you are interested in the full story you can read most of it at this link here.

Anyway, what do we really mean with the term “Universal”? I am going to phrase my own definition here… So please don’t take it as an academic one…

Universal Javascript means writing JavaScript code that can run on different JavaScript runtimes over different contexts. For example on the Browser as well as on the Server.

4.

But that’s not limited only to the web, for example we can talk about Universal JavaScript even when building desktop or mobile applications and some people are even attempting experiments to control hardware devices directly with JavaScript. During the course of this talk we are going to focus only on Web development though.

5.

What are the main advantages of Universal JavaScript? First of all it allows you to deal with a JavaScript only development environment. Having JavaScript both on the Browser and on the Server removes the cost of language switching. This of course happens with any Node.js application, even if not Universal. When it gets to universal you are going to have most of the same code shared between frontend and backend, so in that case your code base would probably contain less duplication and thus will result to be more maintainable. Also with universal JavaScript you will achieve better Serach Engines Optimization, because the server will be able to render the full content of you pages and not just that but it will also be able to return proper http codes in any situation, so cases like “page not found” or “redirects” can be resolved correctly by Google and the other crawlers. Finally we will also achieve a better “perceived” load time. This is a side effect of server rendering, because when the browser finish to download the HTML of the page it will have already most of the information needed to render the page, so the classical flickering effect of classic single page applications would be less likely to happen.

6.

Just to mention a few, the very first early adopter of Universal JavaScript were Netflix, Airbnb, Wordpress (that used this approach to build the new administration panel for wordpress.com) and Dropbox.

7.

So yes, everything looks great, but if you already heard about it, you probably also heard that it’s very complicated… Let’s see why many people think so…

8.

Well, there are some specific pain point to address when it comes to build a Universal application. First of all you will need to have a nice way to write your code (generally in the form of modules) once and make it available between the backend and the frontend. The two environments, even if similar, are not the same. For example they load files and resources in different ways so we need standards and tools in order to “transform” the code in a way that is loadable and functional in both environments. The current open source solutions are UMD, which stands for Universal Module Definition, SystemJS that allows to load modules asynchronously and Browserify and Webpack as module builder for bringing CommonJs/Node.js modules to the browser.

9.

Another issue is universal rendering. When it comes to JavaScript heavy applications we build most of the HTML needed from JavaScript itself (generally using the concept of components). In a regular Single Page Application these components are rendered only by the browser, after the page is fully loaded. With Unviersal JavaScript we want to be able to perform the generation of the full HTML from the server before the page is served to the browser. All the major new framworks like React and Angular 2 supports way to achieve Universal Rendering and there’s a new interesting player in this field called Next.js.

10.

A similar problem is universal routing. We want both the browser and the server to be able to render specific view given specific urls. Ideally we don’t want any code duplication, so the same mechanism to define routes and actions should be defined once and should function in a seamless way in both environments. There are several interesting libraries trying to address this problem and one of the most famous, at least in the React land, is React Router.

11.

Again a similar problems comes in relation to loading the data (often in the form of an API) needed by the application. In these cases one of the most common solutions is to have an api that is accessible both from the server and the client and load it with an universal code base using a library like Axios or Universal Fetch.

12.

Finally one of the last big pain points is Universal State management. There is a tendency to define the full state of an application in a single object generally using libraries like Redux or Cerebral. We want to keep doing this in a way that is fully compatible and functional both on the server and the client.

13.

And well yes, plain old JavaScript doesn’t seem to be enough these days. We often want to use some high level abstraction or language extension like Flow or JSX or even other new languages able to compile directly to JavaScript like TypeScript or Elm. This is not strictly necessary but it might make the developer experience way better.

14.

Yeah I know the current JavaScript panorama sounds a bit too broad and scary…

15.

But let’s stop complaining and let’s build something together… It doesn’t need to be perfect but I will show you that it’s totally possible to build a non trivial Universal application in about 30 minutes.

16.

The tools that we are going to use for this experiment are Webpack, React, ReactRouter and express.

17.

I was for years a Judo practitioner and I am still a big fan of this sport, so we are going to build an application called Judo Heroes. I already published a version of this application, you can see the URL here (judo-heroes.herokuapp.com) and I also wrote an entire tutorial based on it on Scotch.io (bit.ly/judo-heroes-tutorial).

18.

As you can see the application is very simple ad allows us to display some of the best Judo Olympic athletes and read some details about them: their nationality, the year of birth, some pictures and the list of all the medals they won.

19.

Let’s see how Universal comes into play with this app. If we load a specific athlete page, we can see, using the chrome developer tools all the loaded resources.

20.

Then, if from the interface we switch to another athlete, the application loads only the new resources. In this case the images for the new athlete.

21.

And also, if we refresh the page, the server will render the full html for the athlete page straight away. We can see this here with a curl command.

22.

Ok, let’s start looking at some code. Fist of all let’s see out data set. For the sake of this tutorial we will keep things simple and rely on a JavaScript module that exports an array of athletes. Every athletes is an object which contains all the information we need in order to display an athlete: name, country, birth year, images, list of medals, etc. We can easily require this module everywhere we want to access the data.

23.

Now let’s build all the React components that define our interface.

24.

The first component that we are going to see is the Layout component, which has the goal of defining the markup of the global layout we want to use in our application. It contains a static header and a static footer and a variable content.

25.

This is more or less how a component looks like in React. We basically needs to define a render function that receives some parameters (called props) and returns the markup needed to render the component view. This markup looks like a mix of HTML and JavaScript and it’s called JSX. In this case we expect to see:

a. An header section b. A variable content (notice the variable props.children) c. And the footer section

26.

Now let’s move to the IndexPage component page. This component has the goal to show a card based navigation bar that allows us to select an athlete.

27.

The first thing to notice is that we are importing our data module to get the list of all the athletes. Then inside our render method we iterate over this list and, for every athlete in the list, we reference another component called AthletePreview passing to it all the data representing the current athlete.

28.

So the AthletePreview component represents a card in the navigation bar of the IndexPage. Let’s jump to the code.

29.

Let’s use this component to understand better how React and its JSX syntax work.

a. As I told you we have a function that has the goal of providing the JSX markup needed by React to render the component. b. The main tag that we use is Link which represents a React Router component. This will make all the component clickable and will link it to a specific React route. We will see later in greater detail how the routing functionality works. The only important thing you need to notice for now is that the attribute to uses an interpolated variable to build the link reference. We reference here the props object which will contain all the attributes passed to this component by the parent component. c. Then we display the headshot of the athlete, again we access the object props to fetch the picture for the current athlete. d. We do the same to display the name of the athlete. e. Finally we also define the markup to render a little medal icon and the count of the medals won by the athlete.

30.

Now let’s move to the AthletePage which represents the detail page of an athlete.

31.

A lot of code here. But don’t worry this is the biggest component we will see today and at the end it’s quite simple. So let’s zoom into the important details.

32.

This component as well has access to all our data. It receives the id of the current athlete to display in props.params.id, data that will be populate by the routing layer given the current url.

a. We use this information to verify if the current id really exists in our little static database object. If not we simply render a NotFoundPage component that we will see later. If the athlete exists we will have all its data in the athlete variable. b. If we have found the athlete we first render a custom component called AthletesMenu. c. Then we have the markup that define the header of the page (the one with the big background image and the picture of the current athlete).

33.

d. Then we have a the markup needed to render the athlete description. e. Notice that inside this markup we reference another custom component called Flag which is the one responsible for rendering the flag icon inside the description. f. In the next section we iterate over all the athlete medals and display them… g. Using a custom component called Medal h. The last bit is a link that redirects to the index page.

34.

This is the AthletesMenu component that we referenced in the previous component.

35.

In its code we access again to our data module and we iterate over it.

a. For every athlete we reference a Link component pointing to the athlete page. It’s worth noticing that we use the attribute activeClassName. This is a very handy utility of the ReactRouter module that allows us to apply to the link a specific class to the link tag when the linked route matches the current URL. With this utility get very easy to create some CSS code to style active links differently from the other ones.

36.

This is the Flag component we mentioned some slides before.

37.

The code is very simple. We just need to use the props.icon to render the image tag properly. We also have an optional prop called showName that, if set to a truthy value, tells the component to also display the name of the country along the flag icon.

38.

Finally let’s see our Medal component.

39.

In this component, before the render function we have a little dictionary object that we use to map medal types ids to it’s literal description (‘G’ for ‘Gold’, ‘S’ for ‘Silver’, etc.). Then we reference all the properties we receive from the parent component to display the medal year, the city, the name of the competition and the category.

40.

Finally let’s have a look at out super simple NotFoundPage component. This is probably the simplest one as we don’t reference any dynamic property but we just render some markup to display a very simple error page with a link to the index.

41.

Ok, enough React components! I hope you are not bored already… Let’s move to the routing functionality.

42.

In our application we will have two different routes:

a. The index page, mapped to the main route b. and the athlete page which uses the athlete id as parameter.

43.

in our Routes components we define these two routes. Using ReactRouter we can define the routes hierarchy using JSX.

We have a main route that links the Layout component. This will make sure that every component will use our default layout.
Then the common index route is associated to our IndexPage component.
We than map the AthletePage component to the path athlete/:id
And finally we provide a fallback path that matches all the other routers and renders our NotFoundPage component.

44.

Let’s create the real router component now.

a. This component imports our previously defined hierarchy of routes. b. It uses the hashHistory mechanism to detect the current router. We will see later what this really means. c. We reference our routing hierarchy through the routes attribute. d. Finally we use a little function to scroll the page back up every time we move dynamically to a new route. This will make sure that the transition between one page an another looks natural.

45.

So now we have React components and routing. Let’s combine them together and finalize our client app.

46.

The app-client represent our browser application.

a. It loads our router b. and renders it inside the tag with id main.

47.

Let’s see how the main html of our application looks like

48.

We basically have some regular meta headers and a link to a stylesheet. Then in the body:

a. We have our main tag which will contain all our react app b. The bundled script that contains all the code needed to run our app in the frontend.

49.

Let’s how we can build this bundle using Babel and Webpack.

50.

Our Babel config looks extremely simple:

a. We just need to specify that we want to transpile react code and EcmaScript 2015 to regular ES5 code.

Our webpack config is a bit more elaborate:

b. First thing we need to specify is the entry point of our app. Webpack will start to crawl through all the used dependencies starting from this file. c. Then we specify where we want to output the resulting bundled file. d. And finally we specify that we want to process every file in our source directory using babel.

51.

To create our bundle we just need to call webpack from the command line.

52.

The last bit we need before seeing our application working is to write a small server app using Express that allows us to serve the static files that compose our app at this stage. We can do this using the Express.static middleware.

53.

Let’s finally test our app!

(Show frontend only app, show routing with hashes, code only rendered on the client and 404 not understood)

54.

So What we have done so far?

We defined our views combining a bunch of different React components
We added routing using React Router
We compiled our application for the browser using Babel and Webpack
Then we executed the application served through a static Express server

55.

Ok nothing so exiting so far, let’s now add Server Side Rendering and Routing!

56.

When we executed our front-end only application you probably noticed that our URLS were looking a bit weird. The only changing part was prefixed by an hashtag symbol. That’s because we were using the hashHistory mode in our router. This route works very well for frontend-only applications, because it makes shareable link pointing to the right dynamic section, even when in the server you only have an index.html page.

We need now to switch to full routes (without hashtags) and to do so we can simply change our hashHistory to browserHistory.

57.

Now we also want to convert our old index.html to a generic ejs template. Our new file looks almost identical, with the only exception that inside our main block we have now an ejs template variable called markup that allows us to inject arbitrary html code into it. We are going to use this to inject the React generated markup when rendering from the server.

58.

Now the last step is to update our server application to support universal routing and rendering. To do so we need to import some libraries from React, ReactDom e ReactRouter. We also import our routes from our frontend-only app and the NotFoundPage component.

Finally we also enable the support for ejs templates.

59.

Now let’s get to the most interesting part, let’s see how the universal routing and rendering works.

The important part of the code here is the Express route defined with app.get('*', (req, res) => {...}).

This is an Express catch-all route that will intercept all the GET requests to every URL in the server (not previously matched to a static file). Inside this route, we take care of delegating the routing logic to the React Router match function.

match accepts two parameters: the first one is a configuration object and the second is a callback function. The configuration object must have two keys:

routes: used to pass the React Router routes configuration. Here, we are passing the exact same configuration that we used for the client-side rendering.
location: This is used to specify the currently requested URL.

The callback function is called at the end of the matching. It will receive three arguments: error, redirectLocation and renderProps. We can use them to determine what exactly the result of the match operation was.

We can have four different cases that we need to handle:

The first case is when we have an error during the routing resolution. To handle this case, we simply return a 500 internal server error response to the browser.
The second case is when we match a route that is a redirect route. In this case, we need to create a server redirect message (302 redirect) to tell the browser to go to the new destination (this is not really happening in our application because we are not using redirect routes in our React Router configuration, but it’s good to have it ready in case we decide to keep evolving our application).
The third case is when we match a route and we have to render the associated component. In this case, the argument renderProps is an object that contains the data we need to use to render the component. The component we are rendering is RouterContext (contained in the React Router module), which is responsible for rendering the full component tree using the values in renderProps.
The last case is when the route is not matched, and here we can simply return a 404 not found error to the browser.

This is the core of our server- side routing mechanism and we use the ReactDOM renderToString function to be able to render the HTML code that represents the component associated to the currently matched route.

Finally, we inject the resulting HTML into the index.ejs template we defined before to obtain the full HTML page that we send to the browser.

60.

Let’s test the app again.

(Show universal routing and rendering working when changing and reloading pages).

61.

Quick recap about what was done in this presentation:

Create a Single Page Application with React and React Router
Add server side routing and rendering using React and React Router libraries in out Express app

62.

What can we do next to have a fully-fledged Universal app.

Add Universal Data Retrieval
Add Universal State Management

63.

Thanks everybody for being here!

Thanks

Big thanks to the organizers of the event for giving me this chance to be there on the stage. Also great thanks to my great friends Alessandro Cinelli (@cirpo), Andrea Mangano (@andreaman87), Aleksandar Čambas and Peter Caulfield (@quasi_modal) for reviewing my slide deck and for giving me tons of advices!

Some photos from Twitter

Gigantic thanks also to whoever took pictures and shared them on Twitter, some here:

@loige on stage #codemotion Milan! Rock on! pic.twitter.com/9Y97YENhIG
— cirpo (@cirpo) 25 novembre 2016

It's time of #universaljs #reactjs with @loige pic.twitter.com/2XrAZFOBRe
— Andrea Mangano (@andreaman87) 25 novembre 2016

Ottimo talk, su #reactjs. Forse le più belle slide mai viste. Chiare ed essenziali #Codemotion @loige pic.twitter.com/DhCRV8NPMX
— Daniele Montagni (@dmontagni) 25 novembre 2016

I hope you liked this article and that you are now somehow confident that building universal JavaScript application is not that hard… you just need to start somewhere! Feel free to leave any question or comment below on the comments box.

Cheers, see you next time! 🍻

My Serverless & AWS Lambda talk at Node.js Dublin January 2017

Luciano Mammino — Tue, 14 Feb 2017 19:03:00 GMT

Last January I had the pleasure to share the stage with the great Padraig O’Brien to talk about how we use Node.js, AWS Lambda and the Serverless framework at Planet9 Energy.

The talk was hosted by the monthly Node.js Dublin Meetup at Intercom Dublin office, as usual, a very cool location for this kind of tech talks.

Apart from having a great chance to show off my über cool Super Mario T-shirt for about 45 minutes, Podge and I had some good time discussing the following topics:

What does “Serverless” mean and the Serverless framework?
Security concerns like how we achieved authentication and authorization in a serverless and stateless environment.
Code quality and Developer Experience discussing how we do unit and functional tests, Continuous Integration and deployments.
The costs of a serverless infrastructure and the various tools that can allow you to forecast your costs.
Lessons learned and some interesting tips and tricks.

Considering the quite big audience, the number of questions and the stars received on the meetup page I’d say I am jubilant with the outcome of the talk, but of course, I am always eager to deep dive on these topics and discuss my findings with other enthusiasts.

If you are curious you can check out the slides of the talk on SpeakerDeck or watch the recording here below:

And yeah, there are some beautiful pictures too! 😎

Thanks to the organisers and the sponsors for inviting us to speak and supporting this fantastic event. Also thanks to Planet9 Energy for giving us the chance to work on these cool technologies and for encouraging us to deliver these kinds of talks.

If you want to make me even happier, or if you have any question about the talk, feel more than welcome to leave a comment here!

Until next time!

2016 - A year in review

Luciano Mammino — Sat, 24 Dec 2016 18:13:18 GMT

2016 is almost over and I just realised that it has been one of my most productive and exciting years ever. With that in mind I would love to write a recap of all the good and bad things (mostly from a career perspective) that happened in my life during this year. I want to keep track and benchmark my productivity over the coming years. This might be boring for you to read but I hope it’s going to serve myself as a good motivation to keep doing better over the years, both in terms of career and quality of life.

Co-authoring a book about Node.js and Design patterns

One of the biggest achievements this year (and possibly during the course of my entire life) was to co-author a technical book! From November last year to July this year I helped the great Mario Casciaro and Packt Publishing joining them as co-author to revise “Node.js Design Patterns” and come up with a shiny second edition. Being part of this initiative served me a double value. First of all, it was an amazing learning experience, I had to get my hands dirty with so many Node.js details and patterns and nail most of them in a relatively short timespan. Now I feel like I’m fully aware of what’s going on under the hood of Node.js and I am confident I can write pretty decent Node.js code using the right patterns for specific needs. In the second place, being a book author gave me much more visibility as a Node.js enthusiast and now I have more chances to interact with other people interested in this trending topic.

If you are curious about Node.js and Design Patterns you can check out the official website of the book at: https://www.nodejsdesignpatterns.com.

Career

In terms of career, this October I also moved from my position of Senior PHP Developer at Smartbox to a new role as Senior Software Developer at Planet9 Energy. I am still pretty new in the company, but so far this change has been very exciting for a bunch of reasons:

I get to work with a team of insanely skilled professionals like @hughsheehy, @tech_fort, @Podgeypoos79, @quasi_modal, Aleksandar Čambas (who doesn’t like Twitter!) and @katavic_d. It’s a pleasure to learn from them and with them everyday!
Most of my work is in Node.js so I can stick to a language (JavaScript) I am loving and keep improving my knowledge and experience with it.
I can be part of many architectural discussions and decisions about how we evolve a very sophisticated product that is managing an insane amount of data.

I really look forward to keep doing my best next year and see some of the things we are building going live!

Conferences and meet-ups talks

This year I did my first official conference talk at Codemotion Milan in November. If you are interested in Universal JavaScript and React you can find all the details (slides, video and notes) here: My Universal JavaScript Web Applications talk at Codemotion Milan 2016.

I was quite satisfied with the result of the talk, even if I think I still have a lot to improve.

This was the official feedback from the organizers and the audience:

220 people attended your session, among them 68% rated it as “Good”, 30% as “Neutral” and 2% as “Bad”.

In march I also made a talk at the PHP Dublin meet-up group about web application performance. Check it out here: 6 Tips to Build Fast Web Applications (Php Dublin March 2016 Talk).

I really enjoyed both experiences and I hope I will have more chances next year to deliver tech talks in meet-ups and conferences around Europe.

Blog posts

I have to admit this year I wasn’t a very prolific author, I guess the book drained all my energies during my free time. Anyway, I still managed to write some posts. Here are the most interesting ones (based on actual reading stats):

Extracting data from Wikipedia using curl, grep, cut and other shell commands (~18.000 views)
Gulp and FTP: update a website “on the fly” (~7.000 views)
To promise or to callback? That is the question… (~3.000 views)
Introducing Gulp cozy - Manage your gulp tasks in a cozier way (~2.000 views)

Also it’s worth noticing that an article from 2015 is still driving a significant portion of traffic: Developing a web application with Lumen and MySql (~20.000 views).

Even considering such small amount of posts, the blog reached an impressive total of about 90.000 page views.

External posts and Interviews

I also had the time (and the pleasure) to keep going with some interesting collaboration with websites I love like Scotch.io and RisingStack. These are the posts that I enjoyed writing for external websites:

React on the Server for Beginners: Build a Universal React and Node App (~56.000 views)
ZeroMQ & Node.js Tutorial - Cracking JWT Tokens Part 1 & Part 2 (~7.000 views in total)

I also had the pleasure to have two interviews about being a software developer and a book author:

Usersnap: Meet the community: Luciano Mammino, software developer from Sicily (~1.000 views)
Packt blog: With Node.js, it’s easy to get things done (undisclosed stats)

Finally, I’d love to mention that one of my best posts from 2015 has been translated to Italian and published on HTML5Today.it, “Performance: 6 regole per sviluppare applicazioni web”.

Open Sourcing

This year I also spent some time working out some small open source contributions, mostly creating little Node.js libraries or command line utilities for different purposes. Here’s the complete list of the open source repository I have been working on sorted by the number of stars received:

Atinux/schema-inspector (contribution): Schema-Inspector is an JSON API sanitisation and validation module.
silas/node-consul (contribution): Node.js Consul client
lmammino/judo-heroes: A React application to showcase rendering with Universal JavaScript
lmammino/flickr-set-get: A simple command line app to download photos from a flickr set
lmammino/distributed-jwt-cracker: An experimental distributed JWT token cracker built using Node.js and ZeroMQ
lmammino/godaddy-dns: A Node.js script to programmatically update GoDaddy DNS records
lmammino/indexed-string-variation: Experimental JavaScript module to generate all possible variations of strings over an alphabet using an n-ary virtual tree
lmammino/gulp-cozy: Manage your gulp tasks in a cozier way
lmammino/jwt-cracker: Simple HS256 JWT token brute force cracker
lmammino/ghost2hexo: A command line tool to convert Ghost json export to a set of source posts for Hexo
lmammino/x2j-cli: Node.js command line script to convert xml input into json output (can be piped easily)
lmammino/vtt-creator: Very basic Node.js/JavaScript library to generate VTT open subtitles files
lmammino/package-strip-deps: A little command line utility that allows to strip dependencies from a NPM package.json file

Learning

During the last 5 months I slowly started to have a look into the (quite new) Elixir programming language.

For those who never heard before, let me quote the official website:

Elixir is a dynamic, functional language designed for building scalable and maintainable applications.

Elixir leverages the Erlang VM, known for running low-latency, distributed and fault-tolerant systems, while also being successfully used in web development and the embedded software domain.

Why, you might wonder? In the last 2 or 3 year I wanted to spend some time with a pure functional language but Haskell, Clojure, Lisp and other more famous language never really convinced me different reasons. Elixir, for some weird reasons, immediately catch my attention. Plus, it seems to be competing quite well with other “new-shiny” languages:

Finally I also have to admit that my friend @bit_shark had me brainwashed with “how cool Elixir is”! Thanks for that!

Also some other honourable mentions in my learning path this year have been:

AWS (mostly API Gateway, Lambda, Redshift, DynamoDB, S3, EC2, RDS, Route53 and Cloudfront)
Serverless framework
React
ZeroMQ
A bit of Python and Scrapy
A bit of Docker

Failures

Well, so far so good, but… I have to admit it wasn’t all peaches and dandelions. There were some failures here and there and it’s good to take them into account.

The first and more important failure was not being able to focus on a side project. There is a kind-of-secret-project™ I have been working on with my great friend @andreaman87. I think I prioritised and planned really badly the time I wanted to spend on this project and I kept delaying my tasks. That’s a bit of a shame and I need to apologise with Andrea for that!

Also another failure has been in my non-tech life, I haven’t been able to train Brazilian Jiu Jitsu as much as I should have done and I feel like my game and my knowledge hasn’t improved a lot! I guess I am going to keep my white belt still for a while :D

Expectations for next year

Ok, it’s finally time to move to the good intentions for the next year! I have been very lengthy already so I’ll just sort this part out with a simple unordered checklist, I am already looking forward to see, in one year time, how many items will be crossed:

Keep improving my knowledge of Node.js and everything related
Publish the first version of my side project with Andrea
Getting better with AWS and the Serverless framework
Move my blog to a static publishing engine and (possibly) use a serverless hosting approach
Keep investing time on Elixir and build something with it
Presentations and talks in conferences and meet-ups
Get more proficient with VIM (Cheers to @quasi_modal for giving me an amazing book as Christmas present!)
Keep open sourcing stuff
Travel more!
More serious and constant BJJ training! 👊

Well, That’s all… I hope I didn’t bore you to death! Anyway, I’d really love to know what was your biggest achievement in 2016. If you feel like you want to share this information with me feel free to write me on Twitter or to post a comment here! I’ll make sure to have a pint in your honour! :D

“Merry X-Mas and Happy New Year” to you all! 🎅🏻

My Universal JavaScript Web Applications talk at Codemotion Rome 2017

Luciano Mammino — Tue, 11 Apr 2017 22:41:40 GMT

Last month (March 25th) I had the pleasure of delivering again my Universal JavaScript talk in a Codemotion Event, this time in Rome. As expected it was a super interesting event and it gave me the chance to meet fantastic people, attend inspiring talks and refine my presentation.

Judo Heroes Version 2

For this occasion, I updated my common Universal JavaScript Judo Heroes to “Version 2”.

I believe this new version looks a bit more polished and easy to understand, plus it features a number of other relevant changes and improvements:

Updated to Webpack 2, React 15.4, React Router 4 and Express 5
Using pure components where possible
Better structure of components
Top-down data propagation

Video

If you missed the event (or if you are simply curious to hear my “sicilian-english” accent 👌) you can check out the video recording of the talk, just published by the lovely folks at Codemotion:

Slides deck

If you are interested in checking out the slides (and explore all the links in it!) you can find them on Slides.com:

Audience opinion

I was very happy to know that the feedback from the audience was pretty awesome, let me quote the email I received from the organisers:

At this edition of Codemotion Rome we’ve tried to get the most out of attendees feedback by gathering as candid opinions as possible.

Right after your talk “Universal JavaScript Web Applications with React” we asked the attendees to express their evaluation rating it as “Good”, “Neutral” or “Bad”

122 people attended your session, among them, 80% rated it as “Good”, 20% as “Neutral” and 1% as “Bad”.

Such a shame I couldn’t make that 1 or 2 people happy, but that only means that there is still room for improvement, I’ll try harder next time :)

Thanks

As usual, the biggest thank you goes to the organisers of the event for giving me another precious chance to be there on the stage. Then, huge thanks to my great friends Alessandro Cinelli (@cirpo), Andrea Mangano (@andreaman87), Aleksandar Čambas and Peter Caulfield (@quasi_modal) for reviewing my slide deck and for giving me invaluable advice!

Next events

The Universal JavaScript fun is not over for me this year. I will be attending the following conferences soon:

Frontend United Athens (26-27 May, talk)
Web Summer Camp Rovinj (30 Aug – 2 Sept, workshop)

Either if you were among the audience or if you just found out about my talk or my slides, I hope you enjoied the content and the examples and that you are now somehow confident that building universal JavaScript application is not that hard… you just need to start somewhere!

Feel free to leave any question or comment below in the comments section.

See you soon! 🤘

Unshorten (expand) short URLs with Node.js

Luciano Mammino — Fri, 31 Mar 2017 23:23:09 GMT

Short URLs have been an invaluable tool for social media marketing for so many years and we are now used to seeing them everywhere. Most of the credit probably goes to URL shorteners services like Bit.ly, Goo.gl, YOURLS and Rebrandly that popularised the concept and made easy for everyone to start creating short URLs.

When working with URLs in some automation scenarios like analytics, information crawling, data retrieval, etc. it can be important to resolve (or “unshorten” or “expand”) short URLs, which means retrieving the original long URL.

In this article, we are going to see how short URLs work and how we can “expand” them into their original URL.

How short URLs work

A short URLs is a regular URL that most of the time results very short by following a very simple format:

http://<domain>/<id>

The shorter the domain and the id (often called also slashtag), the shorter will, of course, be the URL (e.g. http://loige.link/b).

What happens behind the scene of an URL shortener service is that there is a big database table that contains a map of all the existing short URLs and the related full-length URLs.

A URL shortener service lookup table might look like the following:

Domain	Id	Url
loige.link	b	https://www.nodejsdesignpatterns.com/
loige.link	gh	https://github.com/lmammino
rebrand.ly	luciano-tw	https://twitter.com/loige
…	…	…

Note that in this example the table supports multiple domains and that the pair domain-id must be unique in the table.

URL shorteners have to expose a web server which will be reached through all the domains mapped (in fact, most of the URL shortener services — like Rebrandly — supports different domains or even the possibility to associate your own branded domain).

For every request that satisfies the short URL format, domain and id are extracted and a query is executed against the database to search for the corresponding long URL. Finally, if a match is found, the web server responds with a redirect message, sending the browser of the user to the corresponding long URL.

Resolving short URLs

URL shorteners need to talk with browsers so they have to work as any other web server adopting the HTTP standard. The following diagram illustrates how the HTTP protocol is used when a user clicks a short URL:

A user clicks on a short URL and the browser issues a GET request to that URL.
The request hits the short URL service server which executes a look-up on its database to find the associated long URL. The long URL is returned as a redirect response to the user browser which means an HTTP 301 Moved Permanently response using the header Location to specify the new (long) URL.
The browser automatically follows the redirect response issuing a new GET request to the long URL displaying the content of that URL to the user.

This is what happens when a user through a browser deals with a short URL, everything happens behind the scenes and the user won’t even notice that there was an exchange with a short URL server.

Simple enough, right?

Resolving short URLs in Node.js

Now that we know how short URL services work, it shouldn’t be hard to create a Node.js script that is capable of resolving any type of short URL.

var request = require('request')

var uri = 'http://bit.ly/2fR0xVj'
request(
  {
    uri: uri,
    followRedirect: false,
  },
  function (err, httpResponse) {
    if (err) {
      return console.error(err)
    }
    console.log(httpResponse.headers.location || uri)
  },
)

For simplicity, in this script, we are using the request module to perform the HTTP request. By default, this library follows redirects so we need to explicitly disable this behaviour by specifying the option followRedirect: false. By doing so we have an opportunity to catch a redirect response in our callback and so we can read its Location header which will represent the expanded URL.

In case we are not dealing with a short URL, this script will print back the original URL passed to the function.

A Promise-based, dependency free, Node.js library for expanding short URLs

You can easily adapt the code above to work with your specific use case, but if you want a ready-made and easy to use URL-unshortener NPM library you can trust a library I recently published called tall.

tall is a dependency-free, promise-based library that allows you to easily resolve a short URL and get its corresponding long URL.

To understand how easy to use tall is, just check out this ES2015 sample usage:

import { tall } from 'tall'

tall('http://www.loige.link/codemotion-rome-2017')
  .then((unshortenedUrl) => console.log('Tall url', unshortenedUrl))
  .catch((err) => console.error('AAAW 👻', err))

More examples are available on the official repository.

Bonus: Resolving short URLs with CURL

Of course, you can also use the evergreen CURL to issue the HTTP request for a short URL and then read the Location header to retrieve the long URL:

curl --head "http://bit.ly/2fR0xVj"

This will output something like this:

As you can see the Location header reports our long URL.

If you are a bash and Unix lover you are probably already thinking to pipe this output to grep and other unix command line tools to extract only the long URL out of the full output. If you are curious about this kind of approach you can read a similar article I previously wrote titled “Extracting data from Wikipedia using curl, grep, cut and other shell commands”.

Recap

In this simple article, we learned how short URL services work and thus how we can use them in a programmatic way if we need to know the associated long URL.

I hope this was interesting for you and that you had fun reading the article and the code examples.

As usual, I’d love to have your opinion and know if you plan to use this knowledge or the tall library in one of your next projects. Feel more than welcome to let me know this in the comments!

See you next time!

My Serverless talk at Shift conference in Split

Luciano Mammino — Thu, 15 Jun 2017 13:49:59 GMT

This June, I had an amazing experience talking again about Serverless and how we use it at Planet 9 Energy at the recent Shift Conference in Split, Croatia.

In my talk titled “Serverless: The pros and cons of building a company without infrastructure”, I tried to address what Serverless is and why is becoming such a big thing, also exploring what I believe are the pros and cons of this new paradigm.

The experience was great and I was super happy to have a chance to share the stage with extraordinary people in the industry such as: Chris Heilmann, Ike DeLorenzo, Ryan Christiani, Laura Carvajal, Aki Colovic, Demir Selmanovic, Christoph Reinartz, Dotan Simha and many others!

I really recommend you to watch their talks and follow them on Twitter.

Also, I have to give a big “THANK YOU” to Ivan Burazin for organising this event and for inviting me to talk.

Video

If you missed my talk (or if you really want to hear my curious “sicilian-english” accent again! 😅) you can check out the video recording of the talk, just published by the lovely folks at Shift. I have to say they made an amazing editing, I can almost feel like a professional:

Slides deck

As usual, my slides are available on Slides.com:

Twitter moment

I also created a Twitter moment to collect all my tweets related to this amazing conference.

I hope you will enjoy it:

My experience at @shiftsplit 2017

Closing notes

If you are interested in Serverless and you want to learn how to use it, you might be interested in booking a slot at Skills Matter FullStack conference that is going to be hosted in London from the 12th to the 14th of July.

There, I will host a workshop together with the amazing Christos Matskas from Microsoft about adopting Serverless on AWS (Lambda) and Azure (Functions). It’s going to be fun!

Until next time! 🤙

2017 - A year in review

Luciano Mammino — Wed, 03 Jan 2018 18:59:58 GMT

2017 is over and it’s time to sit down and see what happened during this year. As I already did last year, I would love to write a (potentially boring) recap of all the good and bad things (mostly from a career perspective) that happened in my life during this year.

Last year I promised myself I would keep doing this as a way to keep track and benchmark my productivity over the coming years, so here I am to see how I performed!

The year of conference talks!

Let’s start with a big win! This was the year of conference talks (and workshops too!). I had to travel a fair bit around Europe and I created a dedicated speaking page to keep track of all the events I have been involved with and the related available material (slides, videos, repositories).

Here’s the full list of the talks and workshop I had the pleasure to deliver in 2017:

“AWS Lambda & Serverless framework” at Node.js Meetup, Dublin (January)
“Universal JS Web Applications with React” at Codemotion, Rome (March)
“Building a Serverless company with Node.js, React and the Serverless framework” at JSDay, Verona (May)
“Universal Javascript” at Frontend United, Athens (May)
“Serverless, the pros and cons of building a company without infrastructure” at Shift Conference, Split (June)
“Deep Dive on Serverless Application Development (with Planet 9 Energy)” with Danilo Poccia from AWS and Padraig O’Brien from Planet 9 Energy at AWS Summit, London (June)
“Serverless: getting started with AWS Lambda Functions” (workshop) with Padraig O’Brien at Dublin.js Meetup, Dublin (July)
“Serverless: the battle of the giants” (workshop) with Christos Matskas from Microsoft at FullStack 2017, London (July)
“Universal JS web applications with React” (workshop) at Web Summer Camp, Rovinj (September)
“Serverless from the trenches” with Padraig O’Brien at AWS User Group, Dublin (October)
“Getting started with Serverless and AWS Lambda Functions” (workshop) at Codemotion, Milan (November)
“Cracking JWT tokens: a tale of magic, Node.JS and parallel computing” at Codemotion, Milan (November)
“Cracking JWT tokens: a tale of magic, Node.JS and parallel computing” at Node.js meetup, Dublin (November)
“The future will be Serverless” at The Front Conf, Munich (December)
“Introducing Middy” (Lightning talk) at The Front Conf, Munich (December)
“Building a serverless company on AWS lambda and Serverless framework” with Padraig O’Brien at CodeEurope, Wroclaw (December)
“Cracking JWT tokens: a tale of magic, Node.JS and parallel computing” at CodeEurope, Wroclaw (December)

👍 Compared to 2016, I moved from 2 conferences talks to 17 (+750%), this is a pretty good WIN (and an awesome vanity metrics too 😇)!

I am really curious to see what 2018 will bring on this regard!

Career

In December I finished my contract with Planet 9 Energy and I am very excited to start a new adventure as Solution Architect with Vectra in Dublin. If you never heard about Vectra, this is what Vectra about page has to say about it:

Vectra offers the fastest way to detect and stop hidden cyberattackers – from cloud and data center workloads to user and IoT devices.

The range of products offered by Vectra sounds extremely interesting and I am sure I will be exposed to super interesting topics like IT Security and Machine Learning, so I am absolutely thrilled to get started.

The year of side projects

2017, wasn’t just the year of the conference talks, but also the year of side-projects!

I didn’t write a new book, but I had the real pleasure to be involved in four on-going side projects: Fullstack bulletin, Serverless Lab, Middy and Fastify.

Fullstack bulletin

Fullstack bulletin is a weekly newsletter Andrea Mangano and myself built to support ambitious fullstack developers to stay up to date with the latest trends and news. The idea is to collect 7 interesting links every week regarding topics such as Frontend/Backend web development, APIs, microservices, databases, scalability, but also web design, UI and UX. In every issue, aside from the links, you also get a famous tech-related quote and a suggested tech book.

The links are extracted automatically from the ones tweeted by Andrea and myself during the previous week, using a simple ranking algorithm.

The project is totally open sourced and built on top of serverless technologies (mostly AWS Lambda, Serverless framework and Cloudwatch).

Serverless lab

Serverless Lab is a side-project started by Padraig “Podge” O’Brien and myself to help companies to get started quickly with serverless technologies on AWS.

The support is given with a day-long in-house training course that introduces all the components needed to build a wide range of applications using serverless on AWS. At the moment we defined an introductory course and an advanced one.

Some of our learning materials have been open sourced and you can check them out on the LucPod organisation on GitHub.

Yes, LucPod stands for LUCiano and PODge (Padraig’s nickname)… I know, we have to figure out a better name 😂

Middy

Middy is a Node.js middleware engine for AWS Lambda. It basically allows you to take care of tedious and repetitive tasks such as validation, input deserialization and output serialization by encapsulating this external logic into reusable and testable middlewares. With Middy, your lambda functions will look like this:

const middy = require('middy')
const { middleware1, middleware2, middleware3 } = require('middy/middlewares')

const originalHandler = (event, context, callback) => {
  /* your pure business logic */
}

const handler = middy(originalHandler)

handler
  .use(middleware1())
  .use(middleware2())
  .use(middleware3())

module.exports = { handler }

You can see from the example that the business logic is clearly defined in the originalHandler and that the handler function doesn’t get polluted with additional code that is attached as a decorator with the use function.

Middy is still a pretty young and immature project, but, it’s currently the only solution in this space (as far as I know), so it’s getting a bit of attention. I had the idea of starting this project, but since then it has been a community effort and the current team is composed by some amazing people in the Open Source space:

Also, a special mention goes to Jose Santacruz for bringing Typescript support into Middy.

If you like this idea (and open source in general), remember to give it a star on GitHub.

I hope the project will grow in 2018 and get much more stable and complete.

Fastify

Fastify is a fast and low overhead web framework for Node.js. If you like Express, you are definitely gonna love Fastify.

Fastify was started by two amazing developers: Tomas Della Vedova and Matteo Collina in an attempt to create a performant and scalable, yet very friendly web framework for Node.js.

Now the team has grown up a lot and version 1.0 is very close to being released. You can surely expect big things from this framework in 2018.

For what concerns me, I have been involved mostly in taking care of the Fastify website and its Continuous Integration pipeline. I made sure that every time there’s a new release, the documentation get’s pulled from the main repository and the website is rebuilt, versioned and published. This way, contributors don’t have to worry much about the website look and feel and they just have to maintain a bunch of handy markdown files to document all the aspects of the framework. This part is built by using a bunch of tailored Node.js scripts adopting git and the GitHub APIs, Metalsmith as a static website engine, Circle CI for the CI pipeline, GitHub Pages and Cloudflare for the web hosting.

Blog posts

I have to admit this year I wasn’t a very prolific author (again 😓)… I guess all the stuff described above took almost all my free time. Anyway, I still managed to write some posts. Here are the most interesting ones (based on actual reading stats):

Using Let’s Encrypt and Certbot to automate the creation of certificates for OpenVPN (~3.500 views)
Unshorten (expand) short URLs with Node.js (~2.500 views)
From bare metal to Serverless (~1.500 views)
My Universal JavaScript Web Applications talk at Codemotion Rome 2017 (~1.200 views)

It’s funny to see that most of the traffic is still driven by old (and probably outdated) articles from the previous years like Developing a web application with Lumen and MySql (~12.000 views) and Gulp and FTP: update a website “on the fly” (~5.700 views).

👎 The total number of page views amounts to about 57.000 views, which compared to the previous year total of 90.000 is a dramatic decrease of 36.67%!

I guess I have to write more and find more interesting topics if I want to keep this blog relevant in 2018!

External posts

Aside from the articles published on my blog I had the pleasure to collaborate again with Scotch.io and to establish a new collaboration with Twilio to publish 2 posts as a guest author:

It’s always fun and rewarding to write guest posts and it’s a nice way to get my name out from my common channels.

I guess in 2018 I should keep doing that and maybe build some new relationship with other publishers :)

Open Sourcing

As usual, I kept spending some time working out some small open source contributions, mostly creating little Node.js libraries or command line utilities for different purposes. Here’s the complete list of the open source contributions I made:

Organisations

Repositories

jekyll/jekyll (contribution): Jekyll is a blog-aware, static site generator in Ruby.
mattallty/Caporal.js (contribution): A full-featured framework for building command line applications (cli) with node.js
anaibol/awesome-serverless (contribution): A curated list of awesome services, solutions and resources for serverless / nobackend applications
karlhorky/awesome-speakers (contribution): Awesome speakers in the programming and design communities
mdvorscak/metalsmith-browser-sync (contribution): A Metalsmith plugin to make your workflow easier
1000ch/node-github-markdown (contribution): Parse GitHub flavored markdown to static html
gianarb/wikidiff (contribution): A wikipedia “snapshotter”
lmammino/judo-heroes-2: Universal Javascript sample application with React Router 4 and Express 5
lmammino/webhook-tunnel: A little HTTP proxy suitable to create tunnels for webhook endpoints protected behind a firewall or a VPN
lmammino/slack-shorturl-integration: A slack slash command server to shorten URLs using Rebrandly API
lmammino/terraform-openvpn: A sample terraform setup for OpenVPN using Let’s Encrypt and Certbot to generate certificates
lmammino/lambda-currency-exchange-sms: An AWS lambda function that sends you an SMS (using Twilio) with today exchange rate for a give currency pair
lmammino/sample-jwt-webapp: A sample JWT web app that can be use to demonstrate how to escalate permissions by cracking and forging JWT tokens
lmammino/tall: Promise-based, No-dependency URL unshortner (expander) module for Node.js
lmammino/swagger-jsdoc-generator: Command line script that generates a swagger file based on jsdoc comments
lmammino/json-dynamo-putrequest: Converts an arbitrary JSON into a DynamoDB PutRequest JSON to simplify the import of the raw data into DynamoDB.
lmammino/simple-react-universal-demo: A simple demo to demonstrate Universal JavaScript with React
lmammino/heroku-buildpack-npm-build: Runs npm build on heroku
lmammino/yaml-prune: A simple command line script that allows to prune parts of a yaml file

👍 Comparing last year (13) to this year (19) I had a +46.15% repo contributions increase. Another pretty good vanity metric!

PS: I finally have more than 100 repositories on my GitHub account 🎉

Learning

This year I kept familiarising know technologies such as AWS, serverless, Node.js, React, Docker and python, but I also gained knowledge of some interesting technologies/tools such as Terraform and Ansible.

Also, I am getting more and more interested in the following technologies:

👎 I admit I dropped the ball on Elixir, and even though I spent a bit of time reading about Phoenix (the most famous Elixir web framework), I never really built something barely interesting with it.

Previous years goals

Now it’s time to check whether I met the goals I set last year:

✅ Keep improving my knowledge of Node.js and everything related
😐 Publish the first version of my side project with Andrea
(that mysterious side project was totally dropped, but we managed to publish Fullstack Bulletin instead, so this was not a complete failure)
✅ Getting better with AWS and the Serverless framework
❌ Move my blog to a static publishing engine and (possibly) use a serverless hosting approach
(this is still a work in progress 😟)
❌ Keep investing time on Elixir and build something with it
✅ Presentations and talks at conferences and meet-ups
😐 Get more proficient with VIM
(I think I got slightly better, but VIM it’s not my primary editor still)
✅ Keep open sourcing stuff
✅ Travel more!
😐 More serious and constant BJJ training!
(I finally got my blue belt, but still I feel like I am training way less then I should!)

Expectations for next year

Ok, it’s finally time to move to the good intentions for the next year! I have been very long already so I’ll just sort this part out with a simple unordered checklist, I am already looking forward to seeing, in one year time, how many items will be crossed:

MOAR conference talks/workshops (at least 8)
Write at least 8 quality blog posts (excluding this one!)
Learn a lot about Security, Machine Learning and networking (😇 Vectra, please…)
Keep contributing to Fastify
Keep working on Middy
Move my blog to a static publishing engine and (possibly) use a serverless hosting approach
Learn enough about Rust and build something with it
Learn enough about Elastic Search and build something with it
Learn enough about Kubernetes and build something with it
Do stuff with CSS Grids and Flexbox
Keep learning AWS and serverless
More serious and constant BJJ training!

Well, That’s all… I hope I didn’t bore you to death! Anyway, I’d really love to know what was your biggest achievement in 2017. If you feel like you want to share this information with me feel free to write me on Twitter or to post a comment here! I’ll make sure to have (at least) a pint in your honour! 🍻

Finally, I wish a fantastic 2018 to you all!

AWS Command line: S3 content from stdin or to stdout

Luciano Mammino — Sat, 05 May 2018 12:00:14 GMT

This article presents a quick tip that will help you deal with the content of files in S3 through the AWS command line in a much faster and simpler way.

Did you ever want to simply print the content of a file in S3 from your command line and maybe pipe the output to another command? Or maybe, did you ever needed to pipe the standard output of a sequence of commands directly into a file in S3? I had this need multiple times and, before my amazing colleague Paul made me discover the tip I am about to describe here, I was always using intermediary files to keep track of the input and output of S3 files.

Some examples

Let’s make few practical examples to make this use case easier to grasp.

Imagine you have a PostgreSQL database containing GeoIP data and you want to dump all the data to a CSV, gzip it and store it an S3 bucket.

This is how I used to solve this problem:

# dump the data from PostgreSQL to a compressed csv
psql -U user -d db_name -c "Copy (Select * From geoip_v4) To STDOUT With CSV HEADER DELIMITER ',';" | gzip > geoip_v4_data.csv.gz
# upload the resulting file to S3
aws s3 cp geoip_v4_data.csv.gz s3://my-amazing-bucket/geoip_v4_data.csv.gz

At some point in the future, you probably want to read the file from S3 and search for a given CIDR in the content of the file. Again, this is how I would have solved this problem:

# download the file from S3
aws s3 cp s3://my-amazing-bucket/geoip_v4_data.csv.gz .
# decompress the file and search inside it
gunzip -c geoip_v4_data.csv.gz | grep "1.0.8.0/21"

In both cases, I am creating intermediary files and, as you probably already know, this is not ideal for many reasons. Just to name few, this is a slower operation (not fully stream-able), it takes extra space on disk (imagine you have to deal with very big files), finally, it also needs an extra command. Wouldn’t it be great if we could solve both problems by writing a single pipeline of commands?

The “magic” `-` option in `aws cp`

Buried at the very bottom of the aws s3 cp command help you might (by accident) find this:

Uploading a local file stream to S3

  WARNING:: PowerShell may alter the encoding of or add a CRLF  to  piped
  input.

  The  following  cp  command  uploads  a local file stream from standard
  input to a specified bucket and key:

    aws s3 cp - s3://mybucket/stream.txt

Downloading an S3 object as a local file stream

  WARNING:: PowerShell may alter the encoding of or add a CRLF  to  piped
  or redirected output.

  The  following cp command downloads an S3 object locally as a stream to
  standard output. Downloading as a stream is  not  currently  compatible
  with the --recursive parameter:

    aws s3 cp s3://mybucket/stream.txt -

To make it simple, when running aws s3 cp you can use the special argument - to indicate the content of the standard input or the content of the standard output (depending on where you put the special argument).

Writing to S3 from the standard output

Using this newly acquired piece of knowledge, we now know we can do something like this to write content from the standard output directly to a file in S3:

echo "hello world" | aws s3 cp - s3://some-bucket/hello.txt

This way we can rewrite the solution to our first problem as follows:

psql -U user -d db_name -c "Copy (Select * From geoip_v4) To STDOUT With CSV HEADER DELIMITER ',';" | gzip | aws s3 cp - s3://my-amazing-bucket/geoip_v4_data.csv.gz

This time no intermediary file is created and the data from the gzipped file is immediately streamed to S3 as soon as the first bytes start to be available.

Using data from S3 as input for other commands

The magic - argument can be used also to read the content of files in s3 and pass it in the standard output, for instance, you could do the following:

aws s3 cp s3://some-bucket/hello.txt -

This will output:

hello world

Let’s use this option to rewrite the solution to our second problem as a one-liner:

aws s3 cp s3://my-amazing-bucket/geoip_v4_data.csv.gz - | gunzip -c geoip_v4_data.csv.gz | grep "1.0.8.0/21"

This approach looks much similar to what you would do with a local file and makes integrating other commands seamless with the content of files available in your S3 storage.

Pipeline processing of S3 files

We can combine the learnings from the previous two sections to build processing pipelines for S3 files.

Just to give you a practical example, imagine you have to optimize a png image available in an S3 bucket and save the resulting image in a new bucket.

To optimize an image we can use imagemin which accepts an image in the standard input and returns the optimized image content through the standard output.

Assuming we have our source image in s3://my-images/image.png and we want to save the optimized version in s3://my-images-optimized/image.png, we can write the pipeline as follows:

aws s3 cp s3://my-images/image.png - | imagemin | aws s3 cp - s3://my-images-optimized/image.png

What will happen behind the scene with this pipeline of commands is the following:

S3 will start to stream the binary content of s3://my-images/image.png to the standard output
The standard output is then piped to imagemin and used as input stream
imagemin will start immediately to process the stream and produce an output stream representing the optimized image
This output stream is then piped to the AWS CLI again and the s3 cp command will start to write it to the destination bucket.

No intermediary file is created in the executing machine and the content is just kept in memory in a streaming fashion during the different phases of the pipeline.

The 5GB caveat

If you are writing to S3 files that are bigger than 5GB, you have to use the --expected-size option so that AWS CLI can calculate the proper number of parts in the multi-part upload. If you don’t do this you’ll exceed the number of parts allowed in a multi-part upload and your request will fail.

From the AWS CLI Documentation:

--expected-size (string): This argument specifies the expected size of a stream in terms of bytes. Note that this argument is needed only when a stream is being uploaded to s3 and the size is larger than 5GB. Failure to include this argument under these conditions may result in a failed upload due to too many parts in the upload.

--expected-size should be equal or greater than the size of the upload and it doesn’t have to be perfect. Just close enough.

(Thanks to mahinka for this suggestion)

That’s all folks

I hope this little trick is going be useful to you and that it will allow you to use S3 in a much similar way to how you would use a local file system.

I am really curious to know what kind of use cases you might come up with, so please, let me know in the comments here if you’ll ever use the nifty - option in the aws s3 cp command line utility.

I really look forward to hearing from you!

Special thanks to Paul for making me discover this trick and to mahinka and paul345 (on Reddit) for corrections and suggestions.

Until next time, ciao! 👋

Using Let’s Encrypt and Certbot to automate the creation of certificates for OpenVPN

Luciano Mammino — Mon, 19 Jun 2017 22:00:20 GMT

Recently at Planet 9 Energy, I had to setup a VPN access to secure some of our internal services. One of the requirements was to make the provisioning easy to reproduce over multiple environments, so we ended up playing a bit with Terraform, while obviously adopting OpenVPN for the VPN server.

Another important requirement was to expose the OpenVPN web interface (also called Access Server) using an SSL, which I was expecting to be one of the most challenging things to automate, instead, it turned out to be very easy using Let’s Encrypt and Certbot.

In this article, I will illustrate you how to use Certbot to automate the creation of SSL certificates (for OpenVPN as a practical example) and how to integrate this process in AWS-land using Terraform.

Installing Certbot

Installing Certbot on a Ubuntu (Xenial) machine is as easy as:

sudo apt-get -y install software-properties-common
sudo add-apt-repository -y ppa:certbot/certbot
sudo apt-get -y update
sudo apt-get -y install certbot

This code uses the certbot PPA to install the executable.

A Little tip (in case you don’t know it yet): -y allows the install to be non-interactive and to proceed without the need to confirm every operation from the keyboard.

From this moment on you can use the certbot executable.

If you want to have an idea of its capabilities you can simply run:

certbot --help

The outcome will be something like this:

Generating a certificate with Certbot

Certbot uses Let’s Encrypt to generate a certificate. Let’s encrypt issues a certificate for your domain only if able to verify that you really own that domain and that it is associated with the public IP of the machine from which you are running certbot.

So, in order to pass the verification process, you need to have a web server running on your machine and accessible from the outside world. While Certbot supports the main web servers such as Nginx and Apache, it also features a standalone server that you can use exclusively for the verification process. This web server will run on the standard web ports (80 and 443), so if you have other services using these ports you need to stop them first.

In the case of OpenVPN you can stop its web server with:

sudo service openvpnas stop

Then you can run certbot command line with the following options:

sudo certbot certonly \
  --standalone \
  --non-interactive \
  --agree-tos \
  --email YOUR_CERTIFICATE_EMAIL \
  --domains YOUR_DOMAIN \
  --pre-hook 'sudo service openvpnas stop' \
  --post-hook 'sudo service openvpnas start'

Let’s see briefly what every option is doing:

--standalone: runs the standalone web server for the verification process.
--non-interactive: runs in totally automated mode, never asks for input.
--agree-tos: needed to make the above work, with this parameter you are confirming you agree to the terms of service.
--email: the certificate email.
--domains: the certificate domain.
--pre-hook: this is needed for the auto-renewal of the certificate and describes what is the command to run before the renewal process can be executed. We are using it to stop the OpenVPN web interface.
--post-hook: similar to the previous one, allows us to specify a command to be executed after a certificate is renewed, we use it to restart the OpenVPN web interface.

This command generates numerous files including server.crt and server.key, the two files we need to use in OpenVPN. So let’s link them into the proper OpenVPN folder:

sudo ln -s -f /etc/letsencrypt/live/YOUR_DOMAIN/cert.pem /usr/local/openvpn_as/etc/web-ssl/server.crt
sudo ln -s -f /etc/letsencrypt/live/YOUR_DOMAIN/privkey.pem /usr/local/openvpn_as/etc/web-ssl/server.key

We link them because they will be changed in the future by the renewal process, so we are sure that OpenVPN will be always using the most updated certificate files.

And, finally, we can restart OpenVPN:

sudo service openvpnas start

After few seconds your OpenVPN website should be up and running, and with a shiny green icon indicating that the website is properly encrypted through a signed certificate, woohoo, well done!

A complete example infrastructure with Terraform

I am happy to share with you a simplified version of my Terraform OpenVPN project, to give you an example of how you can use the aforementioned details in a Terraform context.

All the code available in the following section is also available as a repository on GitHub.

Let’s start with a list of the AWS resources we need in order to provision a fully functional OpenVPN:

A VPC (Virtual Private Cloud)
One or more subnets in the VPC
A EC2 instance to host OpenVPN
A key pair to be able to SSH into the OpenVPN machine (for maintenance or troubleshooting)
A security group (to enable traffic inside and outside the EC2 instance)
A DNS record in Route53 to expose our VPN in a subdomain

Quite a few things to be honest, but the declarative nature of Terraform will make things easy.

Let’s create a file called openvpn.tf and let’s start to add things there:

VPC & Subnet

Let’s create the VPC and the subnet:

variable "vpc_cidr_block" {
  default = "10.0.0.0/16"
}

variable "subnet_cidr_block" {
  default = "10.0.0.0/16"
}

resource "aws_vpc" "main" {
  cidr_block = "${var.vpc_cidr_block}"
}

resource "aws_subnet" "vpn_subnet" {
  vpc_id     = "${aws_vpc.main.id}"
  cidr_block = "${var.subnet_cidr_block}"
}

Here we are using two variables vpc_cidr_block and subnet_cidr_block that can be easily reassigned from the outside to change the configuration if needed. By default, we are creating a VPC on the 10.0.0.0/16 IP range and a subnet spawning over the full VPN (same IP range).

The rest of the code describing the VPC and the Subnet resources should be pretty self-explanatory.

The key pair

Let’s create now the SSH key that we can use later in case we want to SSH into the OpenVPN machine.

variable "public_key" {}

resource "aws_key_pair" "openvpn" {
  key_name   = "openvpn-key"
  public_key = "${var.public_key}"
}

Basically we are only defining here a aws_key_pair resource and specifying a name and the public key.

Public key is a string that comes from the variable public_key.

Most of the time you will be creating a dedicated key with the following command:

ssh-keygen -f openvpn.key

this will create openvpn.key (private key) and openvpn.key.pub (public key). The first one is the one we would reference here.

If we want to reference the content of a file we can use "${file("openvpn.key.pub")}" to assign a terraform variable.

Security group

Let’s create our security group:

variable "ssh_port" {
  default = 22
}

variable "ssh_cidr" {
  default = "0.0.0.0/0"
}

variable "https_port" {
  default = 443
}

variable "https_cidr" {
  default = "0.0.0.0/0"
}

variable "tcp_port" {
  default = 943
}

variable "tcp_cidr" {
  default = "0.0.0.0/0"
}

variable "udp_port" {
  default = 1194
}

variable "udp_cidr" {
  default = "0.0.0.0/0"
}

resource "aws_security_group" "openvpn" {
  name        = "openvpn_sg"
  description = "Allow traffic needed by openvpn"
  vpc_id      = "${aws_vpc.main.id}"

  // ssh
  ingress {
    from_port   = "${var.ssh_port}"
    to_port     = "${var.ssh_port}"
    protocol    = "tcp"
    cidr_blocks = ["${var.ssh_cidr}"]
  }

  // https
  ingress {
    from_port   = "${var.https_port}"
    to_port     = "${var.https_port}"
    protocol    = "tcp"
    cidr_blocks = ["${var.https_cidr}"]
  }

  // open vpn tcp
  ingress {
    from_port   = "${var.tcp_port}"
    to_port     = "${var.tcp_port}"
    protocol    = "tcp"
    cidr_blocks = ["${var.tcp_cidr}"]
  }

  // open vpn udp
  ingress {
    from_port   = "${var.udp_port}"
    to_port     = "${var.udp_port}"
    protocol    = "udp"
    cidr_blocks = ["${var.udp_cidr}"]
  }

  // all outbound traffic
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

By default, this Security group allows access from every IP address using the ports 22 (SSH), 433 (HTTPS) and all outside traffic. It also enables the traffic needed for the OpenVPN protocol both on TCP (port 943) and UDP (port 1194). In case you want a custom setup, you can redefine all the relevant variables here (but in that case you might need to customise also the OpenVPN settings, a step which is not covered in this article).

Subdomain

Let’s now create the subdomain from where our VPN will be accessible.

variable "route53_zone_name" {}
variable "subdomain_name" {}

variable "subdomain_ttl" {
  default = "60"
}

data "aws_route53_zone" "main" {
  name = "${var.route53_zone_name}"
}

resource "aws_route53_record" "vpn" {
  zone_id = "${data.aws_route53_zone.main.zone_id}"
  name    = "${var.subdomain_name}"
  type    = "A"
  ttl     = "${var.subdomain_ttl}"
  records = ["${aws_instance.openvpn.public_ip}"]
}

In this part of the code, we are defining 3 variables:

route53_zone_name: the name of your main domain zone (e.g. “example.com.”, notice the dot at the end).
subdomain_name: the name of the subdomain to use for the VPN (e.g. “vpn.example.com”).
subdomain_ttl: the TTL for the domain (60 seconds by default).

Then, we create a reference to the Route53 zone using the data syntax.

Finally, we create the record in the zone using an aws_route53_record resource.

Notice that we are referencing here the EC2 instance public IP, which we haven’t defined yet.

EC2 instance

Let’s see now how we can define the EC2 instance that will host the OpenVPN software for our virtual private cloud:

variable "ami" {
  default = "ami-f53d7386" // ubuntu xenial openvpn ami in eu-west-1
}

variable "instance_type" {
  default = "t2.medium"
}

variable "admin_user" {
  default = "openvpn"
}

variable "admin_password" {
  default = "openvpn"
}

resource "aws_instance" "openvpn" {
  tags {
    Name = "openvpn"
  }

  ami                         = "${var.ami}"
  instance_type               = "${var.instance_type}"
  key_name                    = "${aws_key_pair.openvpn.key_name}"
  subnet_id                   = "${aws_subnet.vpn_subnet.id}"
  vpc_security_group_ids      = ["${aws_security_group.openvpn.id}"]
  associate_public_ip_address = true

  # `admin_user` and `admin_pw` need to be passed in to the appliance through `user_data`, see docs -->
  # https://docs.openvpn.net/how-to-tutorialsguides/virtual-platforms/amazon-ec2-appliance-ami-quick-start-guide/
  user_data = <<USERDATA
admin_user=${var.admin_user}
admin_pw=${var.admin_password}
USERDATA
}

In this piece of code we are defining some variables:

ami: the id of the AMI (Amazon Machine Image) to use to run the EC2 instance. The default one is a Ubuntu Xenial OpenVPN machine that runs on the eu-west-1 region (Ireland). Feel free to select another image that might suit your needs best.
instance_type: the type of instance. You can check the list of all the type of EC2 instances to understand which one might be the best for you. The default here is ok for most of the use cases.
admin_user, admin_password: The username and password of the admin user for your VPN.

Whit all those variable in place, we can describe a aws_instance resource. Notice that we are defining it so that it will be bootstrapped in our VPC and in the subnet we specified before. Also notice that we are setting associate_public_ip_address to true, this way the machine will get a public IP (the one we will associate to the subdomain defined earlier).

Finally we define a user_data property. EC2 User Data allows to provide extra configuration code that is executed as soon as the machine is bootstrapped. It is generally used to install extra software, start some service or edit configuration values. In our case, since our image already contains OpenVPN, we use it only to provide the credentials for the default OpenVPN admin user.

Trigger-based provisioning with `null_resource`

At this point we created all the resources we need and the OpenVPN admin can be already accessed by pointing the browser to your custom subdomain (using the HTTPS protocol).

If you try to do that though, you will get a warning page saying that the certificate for this domain is not valid. This is because OpenVPN is using an auto-generated self-signed certificate which is not bound to your domain, neither signed by a trusted authority (such as Let’s Encrypt).

So let’s do the interesting bit here, adding a Let’s Encrypt certificate for our subdomain!

If you got the basics of Terraform and AWS here you might think that we can do this by simply adding few lines to our EC2 User Data script. While this, in theory, is not wrong, is not going to work in this case because of the way our resources here depend on each other. Let’s clarify exactly why…

So in our current situation, we have to respect the following order of creating resources:

Create a VPC
Create a Subnet in the VPC
Create a key pair and a security group
Create an EC2 instance
Create a subdomain record pointing to the public address of the EC2 instance
(TODO) Create the certificate and install into the current OpenVPN instance

So, the subdomain record can be created only after the EC2 instance is running and it has got a public IP address, only then we have a valid domain that we can use for the certificate. This means we cannot use the EC2 provisioning step, but we need to have a “trigger” that tells us when the subdomain is available and do some extra stuff.

Terraform is smart enough to reconstruct the dependency graph of all our resources and to create the resources in the right order, so we don’t need to worry about that and we can keep loving the declarative style of its syntax.

This means that the only thing left to do is to specify this trigger, which we can do by using a null_resource.

Quoting the official documentation:

The null_resource is a resource that allows you to configure provisioners that are not directly associated with a single existing resource.

Let’s jump straight into the code, which I believe will make everything clear:

variable "certificate_email" {}

resource "null_resource" "provision_openvpn" {
  triggers {
    subdomain_id = "${aws_route53_record.vpn.id}"
  }

  connection {
    type        = "ssh"
    host        = "${aws_instance.openvpn.public_ip}"
    user        = "${var.ssh_user}"
    private_key = "${var.private_key}"
    agent       = false
  }

  provisioner "remote-exec" {
    inline = [
      "sudo apt-get install -y curl vim libltdl7 python3 python3-pip python software-properties-common unattended-upgrades",
      "sudo add-apt-repository -y ppa:certbot/certbot",
      "sudo apt-get -y update",
      "sudo apt-get -y install python-certbot certbot",
      "sudo service openvpnas stop",
      "sudo certbot certonly --standalone --non-interactive --agree-tos --email ${var.certificate_email} --domains ${var.subdomain_name} --pre-hook 'service openvpnas stop' --post-hook 'service openvpnas start'",
      "sudo ln -s -f /etc/letsencrypt/live/${var.subdomain_name}/cert.pem /usr/local/openvpn_as/etc/web-ssl/server.crt",
      "sudo ln -s -f /etc/letsencrypt/live/${var.subdomain_name}/privkey.pem /usr/local/openvpn_as/etc/web-ssl/server.key",
      "sudo service openvpnas start",
    ]
  }
}

As you can see at the very top of this snippet of code, we are declaring a null_resource that is triggered when the value aws_route53_record.vpn.id changes, which happens when the subdomain is created. When this happens, Terraform will execute the provisioning logic defined in the rest of the code here, which essentially describes to Terraform how to connect to the OpenVPN machine and what code needs to be run and which we already discussed in the first part of this article.

The provider configuration and the variable file

Ok, our setup is almost ready… The only two things we have left to do are:

define the needed configuration to connect to our AWS account;
define some values for our configuration before we can run Terraform.

To define the AWS config we need to specify the following terraform code (I generally do that into a dedicated file called provider.tf, but you can write this in any terraform file in your project):

variable "aws_profile_name" {}
variable "aws_region" {}

provider "aws" {
  profile = "${var.aws_profile_name}"
  region  = "${var.aws_region}"
}

Finally we can define the values for our variables. To do so we need to create a file called terraform.tfvars which contains a simple list of key-value pairs:

aws_profile_name = "default"
aws_region = "eu-west-1"
public_key = "${file("key.pub")}"
private_key = "${file("key")}"
certificate_email = "tech@example.com"
route53_zone_name = "example.com."
subdomain_name = "vpn.example.com"

# vpc_cidr_block = "10.0.0.0/16"
# subnet_cidr_block = "10.0.0.0/16"
# ssh_port = 22
# ssh_cidr = "0.0.0.0/0"
# https_port = 443
# https_cidr = "0.0.0.0/0"
# tcp_port = 943
# tcp_cidr = "0.0.0.0/0"
# udp_port = 1194
# udp_cidr = "0.0.0.0/0"
# subdomain_ttl = 60
# ami = "ami-f53d7386"
# instance_type = "t2.medium"
# admin_user = "openvpn"
# admin_password = "openvpn"

The commented lines (starting with #) are left for your own reference, in case you want to change one of the configuration defaults.

Plan, apply and destroy

Ok, we are finally ready to run our Terraform project.

In order to see what changes will be applied to your AWS account, your will need to run:

terraform plan

If you are happy with the displayed changes you can confirm them with:

terraform apply

After few minutes you will have a fully provisioned Virtual Private Cloud with its own VPN access. Try to connect to the domain (https://vpn.example.com) and login in with your admin username and password!

Ok, this was probably just a test project for you so you might fancy cleaning everything up… No problem, just run:

terraform destroy

and everything we just created will be blown away from your AWS account, leaving no trace behind!

Some extra tips

Just few extra tips before closing off…

Let’s Encrypt throttling

Let’s Encrypt allows you to generate a limited number of certificates for a given domain on a given timespan.

Quoting directly from the Let’s Encrypt Rate Limit documentation:

The main limit is Certificates per Registered Domain (20 per week). A registered domain is, generally speaking, the part of the domain you purchased from your domain name registrar. For instance, in the name www.example.com, the registered domain is example.com. In new.blog.example.co.uk, the registered domain is example.co.uk. We use the Public Suffix List to calculate the registered domain.

If you have a lot of subdomains, you may want to combine them into a single certificate, up to a limit of 100 Names per Certificate. Combined with the above limit, that means you can issue certificates containing up to 2,000 unique subdomains per week. A certificate with multiple names is often called a SAN certificate, or sometimes a UCC certificate.

We also have a Duplicate Certificate limit of 5 certificates per week. A certificate is considered a duplicate of an earlier certificate if they contain the exact same set of hostnames, ignoring capitalization and ordering of hostnames. For instance, if you requested a certificate for the names [www.example.com, example.com], you could request four more certificates for [www.example.com, example.com] during the week. If you changed the set of names by adding [blog.example.com], you would be able to request additional certificates.

So, be careful not to blow your production domain while playing with Terraform (or with certbot in general).

There are options to generate test or staging certificates, although I haven’t experimented much with them, so I can only suggest you to have a look at the documentation illustration all the available command line options.

Huge thanks to tialaramex on Reddit for a fruitful discussion and few pieces of advice about this specific aspect.

Certificate expiry and renewal

Let’s Encrypt certificates expire after 3 months, so be sure you enable the auto renewal feature.

In reality, the feature is enabled by default, so what’s left to do is to test the auto renewal process. With certbot you can do that using the following command:

certbot renew --dry-run

There’s another important point here to take into account here: everytime the certificate renewal procedure is triggered, your VPN service gets shutdown for few seconds and all the currently connected users are disconnected.

This will happen only once every 2-3 months, but if it’s a real struggle for your organisation you might want to explore different setups.

One solution could be to setup a reverse proxy like Nginx and use it for the SSL termination. This way during the renewal you will only shutdown nginx and not the full VPN service.

Thanks again to tialaramex on Reddit for raising this discussion and proposing the reverse proxy solution.

Intermittent failures during the provisioning

I had some problems during the installation of certbot on the image used here by default. If this is happening to you as well, you should be able to sort them out by running an apt-get upgrade before trying to install certbot. Since you want to do that in an automated fashion, you’ll probably need a more sophisticated command that takes care of auto-responding to possible prompts: yes | sudo DEBIAN_FRONTEND=noninteractive apt-get upgrade.

Wrapping up

This is just a quick example focused on OpenVPN, but you can use the same approach to generate certificates for other web applications. Consult the Certbot documentation to see all the supported web servers and how to use Certbot with them.

Also, I hope this tutorial gave you a good idea on how to use Terraform to automate your infrastructure provisioning. I am still a bit of noob with it, so if you see I have done something terribly wrong (or something that can be improved), please let me know in the comments.

Thanks to @Podgeypoos79, @techfort, Alan and the rest of Planet 9 tech team for working with me on this cool 💩 (chocolate ice cream!). Also a huge thanks to my friend @gianarb for some precious devopsy bits of advice.

Let me know in the comments if this article was useful for you and if you integrated something similar in one of your projects!

See you in the next post :)

From bare metal to Serverless

Luciano Mammino — Sat, 16 Dec 2017 19:23:20 GMT

Lately, I tried to understand why modern cloud computing brought us to the idea (and growing adoption) of “Serverless”.

In this article, I will illustrate the result of a small research I did about the history of Cloud computing from the age of bare metal to serverless.

“You have to know the past to understand the present.”
― Carl Sagan

At the end of this article, I will also illustrate a definition of Serverless and what are its main characteristics.

The invention of the web

This story starts with Sir Tim Berners Lee, the guy who officially invented the World Wide Web in 1989.

At the beginning, the web was just a simple publishing platform for researchers to share information and publish papers. The first website was actually the CERN website, published in 1991.

The web quickly evolved and grew up out of the research space, to become one of the most mainstream and ubiquitous products of the last decades.

The bare metal age

During the first years of the web, publishing a website wasn’t very easy. We are talking about a period that goes roughly from 1991 to 1995.

At that time you literally had to buy your server machine, configure it, connect it to a stable electricity line and to an internet connection and then run a web server on it. You had to take care of every single aspect of making that server always available.

Can you imagine the effort required when you needed to scale your service and adopt many servers?

The invention of web hosting

During 1995, the idea of the web hosting was popularised by platforms like GeoCities.

Web hosting allowed everybody to publish web pages directly on shared servers at a very low cost (sometimes even for free), removing from the publisher all the struggle and the costs needed with the bare metal approach.

Although web hosting was cheap and easy, most of the time it was very limited and it was possible to publish only static files (HTML, images, etc.) through FTP. Later on, web hosting improved supporting dynamic languages like Perl and PHP and databases like MySQL.

Grid computing

Around 1997, thanks to the astonishing work of the University of Chicago and the Argonne National Laboratory, that together founded the Globus Alliance, the idea of Grid computing was formalized.

In a paper titled “Globus: A Metacomputing Infrastructure Toolkit” by Ian Foster (in the picture) and Carl Kesselman, a new idea of computing, defined metacomputing, was formalized and finally made famous:

Metacomputing: a networked virtual supercomputer, constructed dynamically from geographically distributed resources linked by high-speed networks.

Metacomputing is the foundation for Grid Computing which is generally recognized as the collection of computer resources from multiple locations to reach a common goal. The grid can be thought of as a distributed system with non-interactive workloads that involve a large number of files and tasks.

Grid computing was important for the history of the cloud because it shifted the perception on the power of distributed computing. It was the demonstration that the world didn’t need a single supercomputer in order to solve complex problems, because a well-knitted network of ordinary computers can be orchestrated to achieve complex goals or to sustain sophisticated services.

Software as a Service (SaaS)

In 1999 the web was starting to evolve into something more complex, Amazon.com was already a thing and it was already possible to build complex interactions with the users through the browser.

Marc Benioff of Salesforce was one of the first public figures to strongly state the desktop software wasn’t needed anymore and it could have been replaced by software written to work directly on the web through a web browser.

This principle, lead in the following years to the definition of Software as a Service (SaaS) and it was probably the reason why the web started to play a bigger role, not only as a publishing platform but as a complete runtime to execute all sorts of applications and games.

Server virtualization

In 2001, VMWare releases ESXi and server virtualization becomes a thing.

With server virtualization, it’s possible to divide one physical server into multiple isolated virtual environments.

This way the provider can allocate a number of physical machines in advance in their server farms to create a virtual infrastructure in which a new virtual machine (with variable characteristics) can be initialized and provided as a service in a matter of minutes.

This approach is way more convenient and flexible than the bare metal one.

Interesting thing is that virtualization was a thing way back in the history of computing. For example, CP-40 was a research project back in 1964 that ran on the 360. IBM released a product from that called VM in 1972. This eventually turned into z/VM, which has a long line of products before it.

Infrastructure as a service (IaaS)

From 2002 and 2006 Amazon Web Services, a company spawn up by the opportunity that Amazon saw with renting part of their massive compute power, undergo 3 different “official launch” events.

Only after the 3rd launch, AWS found the right proposition to developers and started to build up significant traction. This successful launch defined the term Infrastructure as a Service.

At the time AWS offer constituted by EC2 (Virtual Machine service), S3 (Scalable storage service) and SQS (message queuing system).

In the years to come, AWS and the competing IaaS platforms would have been associated with the idea of “Cloud Computing”.

Platform as a Service (PaaS)

Heroku was developed by James Lindenbaum, Adam Wiggins (in the picture) and Orion Henry in 2009.

Heroku was originally born as an attempt to create an online editor for Ruby on Rails, but it quickly found it’s product-market fit as a complete platform to deploy and scale Ruby web applications. Support later increasingly extended to all the main languages and frameworks in the market.

Heroku defined the idea of Platform as a Service (PaaS) and it was so successful that Salesforce acquired the company and hired Yukihiro Matsumoto, the inventor of Ruby, as Chief Architect a few years later.

Database as a service

Firebase evolved from Envolve, a prior startup founded by James Tamplin (in the picture) and Andrew Lee in 2011.

The main idea of Firebase was to provide a real-time database as a service through a dedicated SDK that could be easily integrated with websites and mobile applications.

This was one of the first successful instances where a database could have been used with a pay-per-use model.

Firebase was so successful that a few years later it was acquired by Google and evolved again into a complete platform for supporting the development of mobile apps.

Little curiosity: for some reason nobody (as far as I am aware) tried to come up with a short name such as DBaaS or RTDBaaS!

Backend as a service (BaaS)

Parse was founded in 2011 by Tikhon Bernstam, Ilya Sukhar (in the picture), James Yu, and Kevin Lacker.

The firm produced a series of back-end tools for mobile developers to store data in the cloud, manage identity log-ins, handle push notifications and run custom code in the cloud. This kind of product was known at the time as Backend as a Service (Baas).

The company was later on acquired by Facebook that kept it running for a while and then it shut it down. The product was consequently released with an open source license and can be installed on-premise.

Containerization

Docker, launched in 2013 at pyCon by Solomon Hykes, provides an additional layer of abstraction and automation of operating-system-level virtualization on Windows and Linux.

Docker can be used as a way to package applications so that they can be easily executed on various servers without having to worry about the underlying infrastructure.

The funny story is that docker was originally created as a way to abstract the underlying infrastructure at dotCloud, a competitor of Heroku that a few years later bankrupted, while docker became a huge success and it is today adopted or supported by almost every cloud provider.

Containers at scale

The launch of Docker was a blast and its potential was immense.

During the following years, a lot of companies (including Google and Hashicorp) started to invest a significant amount of money and energy to create solutions that leveraged docker to run containers on a large scale, like Kubernetes, Swarm, Nomad and CoreOs.

The idea was to use containers as an abstraction to run processes and applications over a cluster of virtual machines.

Function as a Service (FaaS)

In 2014 AWS launched Lambda, a service that allowed to run “code”, in the form of a function, directly in the cloud.

In Lambda the computation model is event based, which means that a function is executed only if a predefined event occurred.

This product popularised a new class of services called Function as a Service (FaaS).

With the advent of FaaS we started to hear the word “Serverless” for the first time.

What is Serverless

Trying to define what Serverless actually means is always a bit tricky, so I prefer to give you two definitions by two acclaimed industry leaders in this field.

The first lengthy definition comes straight from Amazon Web Services:

“Serverless most often refers to serverless applications. Serverless applications are ones that don’t require you to provision or manage any servers. You can focus on your core product and business logic instead of responsibilities like operating system (OS) access control, OS patching, provisioning, right-sizing, scaling, and availability. By building your application on a serverless platform, the platform manages these responsibilities for you.”

— Amazon Web Services

A much more concise one (that I prefer) comes from Auth0:

The essence of the serverless trend is the absence of the server concept during software development.

— Auth0

I hope you got the point. Serverless doesn’t mean that there are no servers, of course, there’s a server somewhere, but as a developer, you don’t get to worry about it and you can focus as much as possible on the business logic of the application you are working on.

Picture from commitstrip.com

Why Serverless?

The history we explored so far is telling that our industry has always been looking for next higher level of abstraction. Developers need as fewer concerns as possible to be able to quickly release features and deliver value to the customers.

Picture from fullstackpython.com

In this picture, Matt Makai from fullstackpython, illustrates exactly how levels of abstraction and platforms were created to keep removing complexity and leave more time for the actual business value that the app has to provide.

Serverless today is the highest level of abstraction we have. It is not perfect and there are strong trade-offs that a developer has to embrace in order to adopt it, but it definitely removes tons of concerns on the infrastructure layer.

The 4 pillars of Serverless

There are other characteristics that come with Serverless and that make it a valuable option. It’s definitely not only about abstraction and the absence of servers…

No server management: You don’t know how many and how they are configured
Flexible scaling: If you need more resources, they will be allocated for you
High availability: Redundancy and fault tolerance are built in
Never pay for idle: Unused resources cost $0

Is it the right approach for the future?

My personal view, probably yes!

Like any other abstraction, Serverless comes with some strong trade-offs (vendor lock-in, steep learning curve, cold start problem, soft/hard limits, etc.), but it allows us, as developers, to focus more and more on building and releasing value as fast as possible.

We live in a time where the competition on the web is at its highest, there is practically no decent idea that somebody else is not already attempting to implement (or that have already successfully implemented!) and it is critical to be able to be on the market quickly and be able to iterate and improve a product efficiently.

Serverless can give you the needed agility to compete in this market for most of the cases, but you have to be willing to learn it and embrace its trade-offs.

It’s probably not the ultimate solution and I believe there will be other abstraction layers and new platforms in the future, but agility and focus on business logic are definitely some things that we will always want in our future as developers.

References

This article was possible only thanks to these amazing resources:

Also, I had the pleasure to discuss this topic in the recent FrontConf 2017 in Munich in a presentation titled “The future will be SERVERLESS”. Feel free to check out the slides deck and give me your precious feedback.

AWS Solution Architect Associate exam, my notes and tips

Luciano Mammino — Sun, 21 Oct 2018 11:57:00 GMT

In this article, I will share some of my notes and tips that might be useful if you are studying to get the AWS Solution Architect Associate Certification.

I recently took this certification and I have to admit it was a little bit more challenging than I originally expected. I have been using a variety of AWS services professionally in the last 3 years, so I was optimistically expecting this practical experience to be enough. In reality, I had to spend some time to study and fill some gaps about important topics or details that I never had to deal with during my professional experience.

In this article, I will try to recap some of the topics I believe are important to know for this specific certification with a particular focus on things that I struggled a bit to remember or that I generally tripped over during the quiz simulations.

Hopefully, if you are preparing for the same certification, this article will be helpful to you! 😊

Mindset

The exam is a classic quiz where you have to pick one or more answers. It’s very easy to find simulations online, so you might be tempted to spend your time just doing that and use the quiz experience as a way to learn what’s necessary to pass the exam.

While this might be good to understand the style of the questions and the depth of the topics covered, you shouldn’t try to learn specific questions and answers by heart. In fact, studying quiz questions and memorizing answer is NOT a good study technique and it will be likely reducing your chances of passing the exam, rather than increasing it!

All the questions are quite reasonable, if you understood the theory behind all the different AWS services and you know the most important details (in terms of costs, configuration, availability, durability, etc.), you should be able to figure out the right answers.

I admit some questions might be tricky. For instance, you might get questions where more than one answer seems to be correct. In those cases, it might be helpful to try to reason by exclusion and look for all the answers that are definitely wrong. If this is still not helping you to come up to a definitive answer, you can still “flag” the question (yeah, that’s an option in the examination platform) and come back to it later with a fresher mind.

Managing your time will be important too. Try not to spend more than 2 or 3 minutes per question. If you feel you are spending too much time on a question, again, you can flag it and come back to it later when you addressed all the others. In short, make sure you addressed all the questions you feel sure about and save some time to address and review the ones you are struggling with. Sometimes, a bit more reasoning will help you identify some key detail you were initially missing to unlock the right answer.

Study and exercise material

I used a number of different sources to prepare for the exam. The mandatory place to start with is the official AWS Solution Architect Associate certification page. In this page you will find all the necessary details about the certification and links to the two most important official (FREE) resources:

I also used other unofficial resources. A big shout out goes to A Cloud Guru, which offers a subscription that allows to watch all video series for all sorts of AWS certifications. In my personal opinion, the format of the courses is great. It is particularly focused on explaining the fundamental principles of every single service but it is also very practical in underlining what’s really important to know in order to be ready the exam. Their platform also offers per topic quiz and exercises, but also a complete exam simulation app.

In the last few days of my preparation, I also found an interesting (and FREE) Android app called AWS Certified Solutions Architect Associate (by Magic Bytes Soft). The app offers nice features to test your knowledge like flash cards and test sessions.

Finally, when you feel confident enough with the exam simulations, it might be a good idea to tackle the official exam simulation by AWS Training (paid) to have a better feeling of what the platform looks like and what kind of questions can you expect.

Take into account that some practical experience with AWS is very important. Of course you can understand all the concepts only at a theoretical level, but practice will be very important to fix those concepts in your mind and to understand why many details are important. So be sure you spend some time playing around with all the services you might not have used yet. In those cases, I found that trying to build a little side project that involves those services is often the most complete and rewarding experience.

My Notes

Here’s a collection of notes that I took during my studies, somewhat organized by topic/service. Again, these notes don’t aim to be comprehensive but they might still be useful to recap some of the most important aspects of all the different services (that’s at least why I compiled this list for myself).

Cloud Computing and AWS concepts

Durability & Availability

Durability can be described as the probability that you will eventually be able to get your object back from the storage system from one of the stores and archives (objects persist during time).
Availability is the probability that you will be able to get an object back the moment that you ask for it (i.e. the object might still be persistent, but that doesn’t necessarily mean you can read it all the time - think of momentary networking or system issues).

The Well-Architected Framework

The Well-Architected Framework is a guide that proposes a set of questions that you can use to evaluate how well your architecture is aligned to AWS practices. It will help you design architectures that can achieve:

Operational Excellence
Security
Reliability
Performance Efficiency
Cost Optimization

Those are sometimes referred to as 5 pillars.

Sometimes, you’ll see a misleading “Availability” (pillar) in the possible answers of the quiz.

AWS Support plans

There are 4 plans available:

Basic
Developer
Business
Enterprise

In the quizzes, you’ll often see a misleading “Corporate” plan which does not exist

Web console SAML authentication flow

SAML allows configuring federated user access to the corporate AWS account (basically you can re-use your corporate user management system to grant access to AWS and avoid to replicate all your users into IAM).

The flow starts from the corporate portal (not on the AWS console)
The user triggers the auth on the corporate portal which verifies the user identity
The portal generates a SAML authentication response that includes assertions and attributes about the user
The client browser is then redirected to the AWS single sign-on endpoint posting the SAML assertion
The AWS console endpoint validates the SAML assertion and generates a redirect to access the management console (suing STS)
The browser follows the redirect which brings into the AWS console as an authenticated user

More info available in Enabling SAML 2.0 Federated Users to Access the AWS Management Console.

S3

AWS Object Storage Solution.

Consistency model

Read after write consistency:

PUT (New files)

Eventual consistency:

PUT (updates)
DELETE

Updates are atomic: requesting a file immediately after an update will give you either the old data or the new data (no partially updated or corrupted data).

Storage classes

Frequent Access:

STANDARD
REDUCED_REDUNDANCY (RRS): for non-critical and reproducible data, you might be losing up to 0.01% of all your objects per year

Infrequent Access (pay per read):

STANDARD_IA: grant quick reads (small latency), but you pay per read.
ONEZONE_IA: like STANDARD_IA, but stored only in one availability zone, less availability (if that specific zone is down you won’t be able to access the files until the zone is back)

Durability & Availability table

Class	Durability	Availability
`STANDARD`	99,999999999%	99,99%
`STANDARD_IA`	99,999999999%	99,9%
`ONEZONE_IA`	99,999999999%	99,5%
`GLACIER`	99,999999999%	99,99%^*
`REDUCED_REDUNDANCY`	99,99%	99,99%

^* After restore

Durability is always 11 nines (99,999999999%) except for REDUCED_REDUNDANCY, suitable for cases where you might afford to lose files (e.g. you can regenerate them).
Availability is almost always 99,99% except for STANDARD_IA (99,9%) and ONEZONE_IA (99,5%), these are suitable for cases where you rarely have to access the data, but when you need to it has to be as fast as STANDARD (can’t wait hours like with GLACIER).

Encryption features

SSE-S3: fully managed encryption at rest (don’t have to worry about encryption keys)
SSE-C: encryption at rest with custom encryption keys (to be provided together with the uploaded or download the file). The key is not stored by AWS. Use this option if you want to maintain your own encryption keys, but don’t want to implement or leverage a client-side encryption library.
SSE-KMS: encryption at rest using keys managed through the KMS service. Allows to define fine grained permissions based on the permissions of KMS keys. Best solution for compliance (PCI-DSS, HIPAA, etc.).

MFA Delete

For extra security S3 supports the option “MFA on delete” which requests a user to enter an MFA code to be able to delete a file. Protects against accidental or unauthorized deletions.

Transfer optimizations

If latency is an issue you can use S3 transfer acceleration.
If transfer speed is a problem or you have to transfer big files through HTTP you can use multi-part upload.
If you need to transfer massive amount of data into AWS and it might take too long to do that on the wire you can use Snowball.
If your storage needs to exist also on premise (hybrid cloud storage), you can use Storage gateway.
Storage gateway offers 2 main volume modes:
- Cached volumes: stores files in the cloud and keeps a local cache to speed up reads
- Stored volumes: optimized for low latency, stores files locally and asynchronously sends back up point-in-time snapshots to S3.

Cross Region Replication (CRR)

Cross-region replication requires the following:

Both source and destination buckets must have versioning enabled.
The source and destination buckets must be in different AWS Regions (that’s why it’s called cross-region!)
Amazon S3 must have permissions to replicate objects from the source bucket to the destination bucket on your behalf.
If the owner of the source bucket doesn’t own the object in the bucket, the object owner must grant the bucket owner READ and READ_ACP permissions with the object ACL.

Defaults / Limits

Max object size: 5TB
Min object size: 0bytes, except for S3 IA that has a min size of 128kb.
Default maximum number of buckets per region: 100
Bucket/Object url format: http://${BUCKET_NAME}.s3.amazonaws.com/${OBJECT_KEY}

Other

S3 can be used to host (static) websites (URL schemas: <bucket-name>.s3-website-<AWS-region>.amazonaws.com and <bucket-name>.s3-website.<AWS-region>.amazonaws.com). You can use Cloudfront as a CDN to make the website fast globally.
S3 website can be configured to do redirects.
Glacier files can be restored through the web console or the S3 API (RestoreRequest action).
Only the owner of an Amazon S3 bucket can permanently delete a version.
S3 can achieve at least 3,500 PUT/POST/DELETE and 5,500 GET requests per second per prefix in a bucket (you can use random prefixes to increase the throughput in data-intensive applications).
You can write objects directly to an edge location (for instance, this is useful if you are letting users of an app upload files directly to S3 and you want to reduce latency world wide).

SQS

AWS distributed queue service.

Types of queue

Standard
- Extremely high throughput (AWS defines it “nearly unlimited”), highly distributed
- Messages are guaranteed to be delivered “at-least-once” (sometimes a message can be delivered more than once)
- Best-effort ordering (messages are generally delivered in order, but they might be occasionally out of order)
FIFO (First-In-First-Out):
- Limited number of transactions per second (3000 TPS with batching, 300 TPS without)
- Guarantees “in-order” delivery and “exactly-once” processing

Configuration options

Visibility Timeout (default 30s, min 0, max 12hrs): if a consumer receives a message from the queue, that message is made invisible to other consumers as long as the Visibility Timeout option specifies. This is done to prevent other consumers to pick up the same message again and give some time to the current process to complete the processing and delete the message. If the current consumer crashes before removing the message, once the Visibility Timeout expires, another consumer will be able to pick up the message again.
Delay Seconds (default 0, min 0, max 15min): allows to postpone the delivery of a message. Once added to the queue, a message will remain invisible to consumers for the amount of seconds specified in the Delay Seconds attribute. This attribute can be specified globally at the queue level (in this case the queue is referred as Delay Queue) or at the message level.

Delay queues are similar to visibility timeouts because both features make messages unavailable to consumers for a specific period of time. The difference between the two is that, for delay queues, a message is hidden when it is first added to queue, whereas for visibility timeouts a message is hidden only after it is consumed from the queue

Long polling

Long polling helps reduce the cost of using Amazon SQS by eliminating the number of empty responses (when there are no messages available for a ReceiveMessage request). By default, Amazon SQS uses short polling, to enable Long polling you have to specify a positive value for the WaitTimeSeconds attribute in the ReceiveMessage request.

Defaults

Messages Max retention (14 days).

EC2 / EBS / ELB / ECS

Compute solutions (virtual servers, etc.).

Types of virtual machines images (AMI)

HVM (hardware virtual machine): fully virtualised hardware and boot. Recommended for best performance.
PV (paravirtual): uses a special boot loader, can run on hardware that does not have direct support for virtualisation. Recommended for old generation instances.

Metadata API

Prefer instance attached roles and metadata API rather than having dedicated access keys saved into every machine
Public IP is not managed through metadata API (read only)
Metadata API is available at the following endpoint http://169.254.169.254/latest/meta-data/

Dedicated hosting

An Amazon EC2 Dedicated Host is a physical server with EC2 instance capacity fully dedicated to your use. Dedicated Hosts can help you address compliance requirements and reduce costs by allowing you to use your existing server-bound software licenses. There are 2 different dedicated hosting modes (tenancy):

Dedicated Hosts: gives you additional visibility and control over how instances are placed on a physical server, and you can consistently deploy your instances to the same physical server over time. As a result, Dedicated Hosts enable you to use your existing server-bound software licenses and address corporate compliance and regulatory requirements.
Dedicated instances: less configurable (no host billing, no visibility of sockets and cores, can’t add capacity).

You can change the tenancy of Dedicated instances from “dedicated” to “host”. After you’ve stopped your Dedicated instances, you can change the tenancy to “host” using the ModifyInstancePlacement API or the AWS Management Console.

EBS backups and snapshots

You can back up the data on your Amazon EBS volumes to Amazon S3 by taking point-in-time snapshots. Snapshots are incremental backups, which means that only the blocks on the device that have changed after your most recent snapshot are saved. This minimizes the time required to create the snapshot and saves on storage costs by not duplicating data.

A snapshot is constrained to the region where it was created. After you create a snapshot of an EBS volume, you can use it to create new volumes in the same region. You can also copy snapshots across regions, making it possible to use multiple regions for geographical expansion, data center migration, and disaster recovery.
While snapshots are per region, volumes created from snapshots are tied to only a given availability zone in that region, so EBS Volumes cannot be attached to an EC2 instance in another AZ.
To take application consistent snapshots: Shut down the EC2 instance and detach the EBS volume, then take the snapshot.
The most resilient way to backup EBS disks is to take regular snapshots.

EBS encryption:

Snapshots of encrypted volumes are automatically encrypted.
Volumes that are created from encrypted snapshots are automatically encrypted.
When you copy an unencrypted snapshot that you own, you can encrypt it during the copy process.
When you copy an encrypted snapshot that you own, you can re-encrypt it with a different key during the copy process.

Other:

Cannot use pre-existing MS Windows licenses
Important things when selecting an instance: I/O and memory requirements.
ELB: AWS CloudTrail can be enabled to record Application Load Balancer API calls for your account and deliver log files
To create EC2 instances using the VMDK files you can use the VM Import/Export service
Placement groups cannot be extended across availability zones

Limits / Defaults / SLA

Uptime SLA for Amazon EC2 and EBS within a given region is 99,95%
Maximum number of VMWare VMs that can be migrated concurrently is 50

ECS

AWS Containers Service.

With ECS tasks, you can specify an IAM role that can be used by the containers in a task. This is a more fine-grained and secure way to handle permissions at container level rather than at instance level.

Dynamo DB

AWS NoSQL database.

Provisioned Throughput

When you create a table or index in Amazon DynamoDB, you must specify your capacity requirements for read and write activity. By defining your throughput capacity in advance, DynamoDB can reserve the necessary resources to meet the read and write activity your application requires, while ensuring consistent, low-latency performance.

You specify throughput capacity in terms of read capacity units and write capacity units.

One read capacity unit represents one strongly consistent read per second, or two eventually consistent reads per second (you can read twice as much if you opt for an eventually consistent read), for an item up to 4 KB in size. If you need to read an item that is larger than 4 KB, DynamoDB will need to consume additional read capacity units. The total number of read capacity units required depends on the item size, and whether you want an eventually consistent or strongly consistent read.
One write capacity unit represents one write per second for an item up to 1 KB in size. If you need to write an item that is larger than 1 KB, DynamoDB will need to consume additional write capacity units. The total number of write capacity units required depends on the item size.

Operation	Capacity Unit Size
READ	4 KB
READ (eventually consistent)	8 KB
WRITE	1 KB

For example, suppose that you create a table with 5 read capacity units and 5 write capacity units. With these settings, your application could:

Perform strongly consistent reads of up to 20 KB per second (4 KB × 5 read capacity units).
Perform eventually consistent reads of up to 40 KB per second (8 KB × 5 read capacity units, twice as much read throughput).
Write up to 5 KB per second (1 KB × 5 write capacity units).

If you exceed your allocated capacity your will get a 400 Bad Request response (HTTP API) or a ProvisionedThroughputExceededException (SDK). The SDK can offer automatic retry with exponential backoff.

Limits

The cumulative size of attributes per item must fit within the maximum DynamoDB item size (400 KB).

Redshift

AWS analytics database (columnar data storage).

To be able to export data into an S3 bucket from a Redshift instance in a private VPC and make sure data doesn’t leave VPC:

Enable Amazon Redshift Enhanced VPC routing
Create and configure an Amazon S3 VPC endpoint

RDS

AWS Relational Database System. Allows to easily deploy single-machine or distributed instances of the most common databases (Oracle, SQL Server, MySQL, MariaDB, PostgreSQL) and Amazon’s own distributed relational database “Aurora”.

Replication

Read replica replication is asynchronous
Replicas across AZs are synchronous
Read replicas are good for heavy queries (e.g. analytical queries for back office operations)
Read replica is supported for MySQL, MariaDB, and PostgreSQL as well as Amazon Aurora

Encryption

To migrate an unencrypted DB to an encrypted one: create a new DB Instance with encryption enabled and then manually migrate your data into it.

Automatic failover of Multi AZ deployments

Events that would cause Amazon RDS to initiate a failover to the standby replica:

An Availability Zone outage
The primary DB instance fails
The DB instance’s server type is changed
The operating system of the DB instance is undergoing software patching
A manual failover of the DB instance was initiated using Reboot with failover

Limits

Maximum retention period for automated backup: 35 days

Other

Amazon RDS enables automated backups settings depends on the DB engine selected
RDS Provisioned IOPS (SSD) Storage is preferable for high-performance OLTP workloads
default metrics for RDS: The number of current connections to the database

VPC

Amazon Virtual Private Cloud. Allows you to create complex private networks in the cloud.

Availability zones names

Availability Zones consist of one or more discrete data centers. Every account has references to availability zones per regions shuffled. As such, ‘eu-west-1b’ for instance, is not necessarily the same physical location for ‘eu-west-1b’ in another account (important in some latency related questions).

How to enable site to site VPN:

Hardware VPN enabled on VPC
On premise customer gateway
Virtual private gateway
- Peering connections do not support Edge to Edge
- Default of 5 elastic IPs per region
- To enable network access logging you can:
Make use of an OS level logging tools such as iptables and log events to CloudWatch or S3.
Set up a Flow Log for the group of instances and forward them to CloudWatch

You can use a network address translation (NAT) gateway to enable instances in a private subnet to connect to the internet or other AWS services, but prevent the internet from initiating a connection with those instances

5 elastic IPS/region max by default

Cloudwatch

Service that allows you to collect metrics, logs and monitor all the other provisioned services (EC2, Load Balancers, etc). It also allows you to create dashboards and manage alerts.

Defaults

Metrics retention for a deleted EC2 (15 months)

Cloudfront

AWS content delivery network.

Can support multi-language resources or other variation of resources using query string parameters by setting cache based on all (or some) queries bring parameters (e.g. http://d111111abcdef8.cloudfront.net/main.html?language=fr)
There is a cost when clearing cache (path invalidation): first 1,000 paths requested for invalidation each month are FREE. Thereafter, $0.005 per path requested for invalidation

SWF

Simple Workflow Service, allows to define and run background workflow composed by parallel or sequential steps. Also described as a fully-managed state tracker and task coordinator.

Preferred to SQS with cases like order management where there might be manual steps (eg. packing and shipping a parcel) or message delivery needs to be more reliable (only once & in order).
When a workflow is not making progress is probably because of a missing manual interaction.

Closing off

This is all I have for you. Again this is not comprehensive (E.g. I didn’t have notes for services like Lambda or SNS), so please don’t rely exclusively on this (and if you do don’t hold me responsible 😜).

At this point let me just wish you “Good luck” and let me know how it goes.

A huge thanks goes to Padraig O’Brien and Vectra.ai for supporting this achievement! 🙏

2018 - A year in review

Luciano Mammino — Sun, 20 Jan 2019 02:18:58 GMT

It is that time of the year when I have to look back at the previous year and see what were my achievements, my failures and set my expectations for my 2019.

Full disclosure: I write this type of posts mostly for myself, so I expect this to be boring to death and most likely useless for everyone else.

You have been warned, read at your own risk!

Static blog migration

Interesting achievement of the last year is that I finally migrated this blog from Ghost to a static setup using totally free and Open Source tools, which costs me 0.00 $ per month.

I am using GatsbyJS as the engine for the static website.

If you are curious to see my setup (and maybe even correct typos in my articles), I have everything open sourced at lmammino/loige.co.

I don’t think my GatsbyJS config is particularly interesting, but the entire publishing pipeline might be worth some words.

As per January 2019, this blog is published on GitHub pages using the gh-pages branch which I publish using the very handy gh-pages package.

The static website is served through Cloudflare CDN, which takes care of speeding up the delivery across the globe and other handy things like making sure the website runs on https correctly.

Every time I push on master, a CirlceCI build is triggered, and, if the build succeeds, a new version of the website is published automatically.

Checkout my CircleCI config if you want to do something similar.

I wanted to do something like this for a while, so being able to achieve this was a big win for me!

Conference talks

Pretty much like in the last 2 years, I tried to invest some of my time in engaging with the community through conference talks.

This year wasn’t my best, but I still managed to deliver 8 talks and a workshop around the world. For the first time I delivered one outside Europe in the amazing New York.

Here’s the full list of talks for 2018, you can find more details in the usual speaking section with links to slides and videos when available:

“Getting started with Serverless and Lambda Functions” (workshop), Codemotion Rome (April)
“Unbundling the JavaScript module bundler”, Codemotion Rome (April)
“The future will be Serverless”, JS Day Verona (May)
“Cracking JWT tokens: a tale of magic, Node.JS and parallel computing”, Web Rebels Oslo (June)
“Cracking JWT tokens: a tale of magic, Node.JS and parallel computing”, Buzz JS New York (June)
“Unbundling the JavaScript module bundler”, Dublin JS (July)
“Cracking JWT tokens: a tale of magic, Node.JS and parallel computing”, FullStack London (July)
“Unbundling the JavaScript module bundler”, Øredev Malmo (November)
“Processing TeraBytes of data every day and sleeping at night”, Codemotion Milan (November)

Compared to 2017, when I presented 17 talks/workshops, this is a -52.94% “growth” rate. Ouch! 😵

In reality, I don’t mind at all this level of commitment for conferences and workshops and I’ll try to keep 2019 to a similar rate to 2018 and not overcommit.

Career

From a career perspective I spent my 2018 at Vectra, where I worked “automating the hunt for cyberattackers and speeding-up incident response”, which essentially means building software that help companies to identify cyber attacks and resolve them as quick as possible and with the least amount of effort.

One year at Vectra has been a blast and I had the pleasure (and luck 🍀) to be able to contribute to some important success stories:

I was involved in bootstrapping and scaling up a new European site that started from 3 and now counts 20+ people (engineering and sales). All world-class professionals with whom I love to work and from whom I can learn a lot every day.
Built a new product line from scratch. The product is already making revenue! This is a quite complicated product that is able to ingest and make searchable Terabytes of data every day!
Witnessed an astounding 104% growth in annual recurring revenue.

One year in the making, there was no shortage of personal skills development:

Improved my knowledge of AWS, including getting certificated as Solution Architect.
Improved my knowledge of several technologies I love: Terraform, Ansible, Packer, ElasticSearch, Node.js, JavaScript, Python.
Got some new interesting use cases for Serverless architectures.
Started to acquire totally new skills like GoLang.
Deep dived into binary protocols like Capnp and got much better understanding of some low level mechanics like serialization and memory management (thank you @gbinside for your forbearing mentorship on this).
Worked a lot on Node.js performance optimizations.
Learned a bit about network security.
Been exposed to some high level AI/ML concepts.
I have been a team leader and an architect, which meant that I had to engage a lot with peers and customers which probably improved some soft skills: empathy, communication, teamwork, patience, accountability, etc (I hope I also improved my english in the process 😅).

With this premise, I expect my 2019 at Vectra to be even more exciting than 2018. I am sure I’ll learn and grow a ton.

There is a lot of work to do to keep the current growth rate, and I’ll do my best to do my part!

Side projects

In 2017, I launched and contributed to a number of interesting side projects. In 2018, I didn’t launch any new one, but I tried to take care at best of the projects launched in the previous year, so I am going to recap here what happened with each one of them.

Fullstack bulletin

FullStack Bulletin, a newsletter for ambitious FullStack developers, has been ongoing in 2018 with the same format of 2017 except some minor improvement. It recently surpassed 1000 subscribers, all through organic growth.

This projejcts takes me 30-60 minutes of work every week, but I am happy to keep investing the time as it works for me as well to stay up to date with the latest FullStack trends.

I still plan to invest on it during 2019 and maybe try to improve the format (new layout, new books, new quotes, etc.).

Serverless lab

Serverless Lab is a side-project started by Padraig “Podge” O’Brien and myself to help companies to get started quickly with serverless technologies on AWS.

During 2018, we took a chance to improve and update our material, but we used it for only one workshop.

It really takes a lot of time to find interested crowds. So far, all the engagement was mostly organic.

I am not sure yet how much I will commit on this project in 2019. I’ll try to see if there’s interest in the market.

Middy

Middy is a Node.js middleware engine for AWS Lambda. It basically allows you to take care of tedious and repetitive tasks such as validation, input deserialization and output serialization by encapsulating this external logic into reusable and testable middlewares.

In 2018 middy had a decent success. Here some numbers:

almost 500 stars on Github
an average of 6.000 downloads a week
31 contributors over 124 releases.

I am quite proud of this project, even though in the latest months I felt like I didn’t have the time, neither the energies to properly take care of it. My response time has become unacceptable and open PRs and Issues have been accumulating. I am a bit worried that the project will start to drift away.

I recently asked more support to the other contributors, basically stepping away from being the main maintainer.

I have to thank you some people for being there and trying their best to support the project:

Yan Cui (who also wrote an amazing article that mentions middy)
Vlad Holubiev
David Wells

And all the other contributors.

A stable version for the 1.0.0 branch has not been released yet. I’ll try to focus on getting it done in 2019 and to try to be a little bit more responsive in replying to PRs and issues.

Meanwhile I hope new people will be able to jump on board and support the project.

Fastify

Fastify, started by Tomas Della Vedova and Matteo Collina is a fast and low overhead web framework for Node.js.

I have been marginally contributing to this fantastic project, taking care of setting up the official website (including the docs) and its automated pipeline.

In 2018, I did some minor improvements, but nothing major.

I admire all the contributors to the project as they managed to invest an incredible amount of energies in the project and attracted a pool of very talented people creating a strong community of users.

I am glad and grateful that I am being considered one of the core team members, even though I am very far away from putting the same amount of energy that the other contributors do.

I want to narrow my focus in 2019 so I don’t have any special plan to contribute more to Fastify in 2019, but I’ll try to be available to give my support on improving the website where needed.

Blog posts

In 2018 I wasn’t a very active author… So I’ll go straight to the list of new posts published in this blog during the year:

AWS Command line: S3 content from stdin or to stdout (~7.800 views)
AWS Solution Architect Associate exam, my notes and tips (~5.800 views)
Emerging JavaScript pattern: multiple return values (~3.200 views)
A random emoji in your terminal prompt. How and Why! (~1.500 views)

It’s interesting to see that there’s still a lot of the traffic driven by old (and maybe outdated) articles from the previous years like:

Extracting data from Wikipedia using curl, grep, cut and other shell commands (~7.300 views)
Using Let’s Encrypt and Certbot to automate the creation of certificates for OpenVPN (~7.200 views)
Developing a web application with Lumen and MySql (~7.000 views)
Gulp and FTP: update a website “on the fly” (~4.000 views).

The total number of page views amounts to about 65.000 views, which compared to the previous year total of 57.000 is a slight increase of 14.04%!

I guess I have to write more and find more interesting topics if I want to keep this blog relevant in 2019! Also I should probably focus more on one or two topics, rather then writing about random fun stuff that comes into my mind!

External posts

Aside from the articles published on my blog I had the pleasure to collaborate with Avanscoperta and Gojko Adzic writing an interview from the title “Is Serverless the future of Cloud Computing?”.

It’s always fun and rewarding to write guest posts and it’s a nice way to get my name out from my common channels.

I guess in 2019 I should keep doing that and maybe build some new relationship with other publishers :)

Open Sourcing

In 2018, I kept spending some time working out some small open source contributions. Here’s a quick list of my main GitHub contributions:

lmammino/fib-it: 6 ways to generate a fibonacci sequence in JavaScript.
lmammino/twaffle: A TWitter rAFFLE script.
lmammino/get-poop-done: A crappy-but-lovely™ todo list app.
lmammino/invok: Command line tool to Invoke Go lambda functions locally.
salesforce/ja3 (contribution): JA3 is a standard for creating SSL client fingerprints in an easy to produce and shareable way.
felixhageloh/uebersicht (contribution): Keep an eye on what’s happening on your machine and in the world.
gatsbyjs/gatsby (contribution): Build blazing fast, modern apps and websites with React.
tj/node-blocked (contribution): Check if the Node.js event loop is blocked.
lmammino/capnp-stream: A Node.js readable stream for Cap’n Proto encoded binary input.
fastify/fastify-http-proxy (contribution): Proxy your http requests to another server, with hooks.
lmammino/benchmark-distributed-jwt-cracker: A benchmark suite using AWS for distributed-jwt-cracker.
lmammino/metrica: Event Emitter based node library that emits process metrics (uptime, memory, cpu) at given intervals.
zeek/zeek (contribution): Zeek (formerly known as Bro) is a powerful network analysis framework that is much different from the typical IDS you may know.
lmammino/lambda-runtime-node8-capnp: Docker container image that resembles AWS lambda runtime for Node.js 8.10 and CapnProto 0.6.1
lmammino/lambda-runtime-node8: Docker container image that resembles AWS lambda runtime for Node.js 8.10.0
lmammino/openssl-rfc-mapping: A quick’n’dirty script to map OpenSSL RFC to machine parseable data.
lmammino/stream-accumulator: Accumulate all the data flowing through a stream and emit it as a single chunk or as a promise.
maxogden/mississippi (contribution): A collection of useful stream utility modules for writing better code using streams.
lmammino/lumpy: A lumpy and dummy JavaScript module bundler for those who are stuck in the past.
lmammino/tick-stream: A simple transform stream that can log a tick every time data is pushed through a pipe.
nickjj/ansible-docker (contribution): Install / Configure Docker and Docker Compose using Ansible.
jsdoc2md/dmd (contribution): The default output template for jsdoc2md.

👍 Comparing last year (19) to this year (22) I had a +15.79% repo contributions increase. Another pretty good vanity metric!

Previous years goals

Now it’s time to check whether I met the goals I set last year:

✅ MOAR conference talks/workshops (at least 8).
😭 Write at least 8 quality blog posts (excluding this one!): did only 5!
😐 Learn a lot about Security, Machine Learning and networking: I guess I learned a bit, but probably not as much as I was expecting last year.
😐 Keep contributing to Fastify: didn’t do a lot.
😐 Keep working on Middy: Initially well, lately not so much…
✅ Move my blog to a static publishing engine and (possibly) use a serverless hosting approach.
😐 Learn enough about Rust and build something with it: studied for a while, but didn’t really build anything. I feel I kind of dropped it in favor of GoLang.
✅ Learn enough about Elastic Search and build something with it.
😐 Learn enough about Kubernetes and build something with it: Again, studied a bit but didn’t really do much with it.
✅ Do stuff with CSS Grids and Flexbox.
✅ Keep learning AWS and serverless.
😐 More serious and constant BJJ training: kept training, but I should probably be more serious about it!

Expectations for next year

Keep doing conference talks/workshops (at least 8)
Write at least 8 quality blog posts (excluding this one!)
Keep learning about Security, Machine Learning and networking
Release Middy v1.0
Publish a new book (or a new edition ✌️)
Get proficient in GoLang
Learn a bit about Social Media and digital marketing
Keep learning AWS and serverless
Get an advanced AWS certification
Play with new database technologies (e.g. CockroachDB & Clickhouse)
Improve FullStack bulletin
More serious and constant BJJ training!

That’s all… If you are still reading, either you are a bot 🤖 or you are my new hero 💪!

In the latter case, I’d really love you to leave a comment here and let me know something about your 2018 and what you envision for your 2019.

See you in the next post, CIAO!

2019 - A year in review

Luciano Mammino — Sat, 25 Jan 2020 14:18:58 GMT

A little bit later than I expected over the new year, but here we are again, with my usual introspective post to figure out how the hell did the last year went and how to make the best out of the new year.

This is going to be boring (again I do this kind of posts selfishly only for myself), so brace yourself if you really want to keep going on this!

Tech Leadership Training

Possibly the biggest achievement of last year was my contribution as a teacher to the Tech Leadership Training - Taking you from Developer to CTO organised by Codemotion and Facebook Developer Circles.

The course is designed to help developers in their journey to become CTO of startups, combining strong interpersonal and influential skills along with an empathic ear.

The course was made of 3 modules:

Module 1: Defining the CTO Role
Module 2: AI/Machine Learning
Module 3: Software Architecture/DevOps

I presented Module 3, where I essentially covered what happens when a startup scales from thousands of users to millions of users. My track highlights that only a correct and solid software architecture can keep the business safe, but also that lean and agile techniques can allow your product to keep growing over time.

Conference talks

I invested a decent amount of time delivering conference talks and I accrued a total of 13 talks and workshops in 6 different countries.

Here’s the full list of talks for 2019, you can find more details in the usual speaking section with links to slides and videos when available:

“Processing TeraBytes of data every day and sleeping at night”, AWS Usergroup Dublin (February)
“Processing TeraBytes of data every day and sleeping at night”, AWS Usergroup Belfast (February)
“It’s about time to embrace Streams”, DublinJS (March)
“It’s about time to embrace Streams”, Codemotion Rome (March)
“It’s about time to embrace Streams”, CityJS Conf London (May)
“It’s about time to embrace Streams”, Node Ukraine (May)
“It’s about time to embrace Streams (workshop)”, DevIT Thessaloniki (June)
“Unbundling the JavaScript module bundler”, DevIT Thessaloniki (June)
“Processing TeraBytes of data every day and sleeping at night”, InfiniteConf London (July)
“It’s about time to embrace Streams”, Austin Node.js Meetup (August)
“Serverless, the Middy way (workshop)”, AWS Community Day Dublin (October)
“SERVERLESS Panel – Serverless Insights – Taking Serverless to the next level”, AWS Community Day Dublin (October)
“It’s about time to embrace Streams”, MachJS Meetup Manchester (October)

Compared to 2018, when I presented 8 talks/workshops, this is a 62.5% “growth” rate. 👍

Not sure how much do I want to commit for this year. I am not actively applying for conferences, so probably I will limit my participation to events where I get a direct invitation. Also, I will have to start developing some new talks!

Career

From a career perspective, I spent another year at Vectra. As usual, it was an intense and engaging year, full of learnings. Probably most learnings were not as technical as I hoped, but still, I am sure that developing soft skills and business skills is always a precious addition to any technical career.

Surprise now! Next week is going to be my last week at Vectra. I decided to move on and search for new opportunities starting from February 2020.

Hit me up on Twitter if you think I could be a good fit in your company/project and let’s have a chat!

Side projects

Nothing new in terms of side projects, but I am still running and contributing to my usual side projects: FullStack Bulletin, Serverless Lab and Middy.

The work on middy has been a little bit stagnating over the year. Not being able to work on a lot of serverless projects on my day to day made me lost a bit of focus and perspective.

Thankfully the community around middy has been particularly active and involved and a lot of people helped me driving the project towards the long-awaited 1.0.0 beta. This year we will make 1.0.0 stable and start to make plans for the future of the framework for the next big release.

A great shout out to Will Farrell and Sebastian Domagała for their great contributions and the constant support.

One interesting thing is that towards the end of the year I started working on a (currently secret) side project, that should be released this year around May or June. Stay tuned 😇

Blog posts

In 2019 I wasn’t a very active author… It was actually pretty bad with only 3 blog posts published:

JavaScript iterator patterns (~14.600 views)
Fastify and Preact for quick web app prototyping (~4.300 views)
Lean NPM packages (~4.300 views)

It’s interesting to see that my 2 top posts of last year were articles written during the previous year:

Emerging JavaScript pattern: multiple return values (~26.100 views)
AWS Solution Architect Associate exam, my notes and tips (~16.900 views)

The total number of page views amounted to about 120.000 views, which compared to the previous year total of 65.000 views is a great 85% increase.

I should dig in to find out the reason for the increase. I don’t feel like I did anything special (or planned) to drive such improvement.

For this year, I am not sure I am ready to make any commitment. I want to try and find one or two topics that I am really passionate about and that I can be able to write interesting posts about. Let’s see how it goes.

External posts

In 2019, I have been writing 3 posts for a new website that is yet to be published. I will disclose the links in the about section when they are available!

Open Sourcing

In 2019, I did not stop doing my small part to contribute to open source. Here’s a quick list of my main GitHub contributions:

callstack/linaria: Zero-runtime CSS in JS library (contribution).
lmammino/fastify-preact-htm-boilerplate: Quickly bootstrap your next web app with Fastify, Preact and htm.
lmammino/s3-list-bucket-stream: Node.js stream library that allows you to stream a list of objects from an S3 bucket.
lmammino/s3-object-content-stream: A Node.js transform stream that gets S3 object keys and emits their actual content.
lorenzofox3/for-await: operators and stream semantic for asyncIterators (contribution).
lmammino/streams-examples: A bunch of examples on how to use Node.js streams.
olivere/elastic: Elasticsearch client for Go (contribution).
lmammino/lambda-image-colors: An example AWS Lambda written in GoLang to tag a picture with its prominent colors.
lmammino/s3st: A command line utility that allows you to stream data from multiple S3 objects directly into your terminal.
lmammino/rustlings: random exercises to learn rust (mostly to entertain myself on long plane flights ✈️).
lmammino/streams-workshop: A workshop on Node.js Streams.
svgdotjs/svgdotjs.github.io: Documentation and website for SVG.js (contribution).
theKashey/rewiremock: The right way to mock dependencies in Node.js or webpack environment (contribution).
goldbergyoni/nodebestpractices: The largest Node.js best practices list (contribution).
Jason3S/xxhash: Pure JS Implementation of xxhash (contribution).
fastify/fastify-etag: Automatically generate etags for HTTP responses, for Fastify (contribution).
fastify/releasify: A tool to release in a simpler way your module (contribution).
fastify/fastify-oauth2: Enable to perform login using oauth2 protocol (contribution).
lmammino/public-transport-ireland: Node.js module with utility functions to get real time data for Irish public transport (Irish Rail, Luas, Dublin Bus).
lmammino/md-article-template: A ready-made setup to write articles in Markdown and be able to see live preview and export the article in HTML.
lmammino/realtime-transport-dashboards: Serverless APIs for AWS to build and display public transports real time data (Serverless application example).
fourTheorem/slic-starter: A complete, serverless starter project (contribution).
lmammino/netcore-node-azure-func-container: a repository for Debian 9 containers including net core sdk, node.js and azure functions CLI.
nodejs/node: yes, that’s the Node.js repository! I did a small contribution, but still cool to have one there.

👍 Comparing last year (22) to this year (24) I had a +9% repo contributions increase. Quite stable I’d say!

Previous years goals

Now it’s time to check against my previous year goals:

✅ Keep doing conference talks/workshops (at least 8).
😭 Write at least 8 quality blog posts (excluding this one!): did only 3…
😐 Keep learning about Security, Machine Learning and networking: I guess I did learn something more, but nothing worth a mention…
😐 Release Middy v1.0: only the beta for now!
😐 Publish a new book (or a new edition ✌️): …
😐 Get proficient in GoLang: I didn’t have many chances at work to keep developing this skill and I ended up investing a bit of my free time on Rust instead…
😐 Learn a bit about Social Media and digital marketing: read only few blog posts…
😐 Keep learning AWS and serverless: didn’t do much serverless, or at least not as much as I expected; I barely kept up with the various Serverless news.
😭 Get an advanced AWS certification. Nope…
😭 Play with new database technologies (e.g. CockroachDB & Clickhouse): this was drive by expected work needs that did not go as expected. Totally dropped it.
😐 Improve FullStack bulletin: I did some improvements under the hood, but nothing visible outside (as I wanted).
😐 More serious and constant BJJ training!: not too bad, but could have been better!

Yep I did underdeliver or fail most o my goals, maybe I should set more realistic and coherent goals for this new year!

Expectations for next year

Ok, let’s try to do that then:

Get a new job 😛
Do only few (invitation based) conferences (3 or 4)
Write some good quality blog posts (at least 4)
Release Middy v1.0 stable and have a plan for the future of the framework together with the community around it.
Publish a new book (or a new edition ✌️)
Learn more about Rust
Learn more about Kubernetes
Keep learning AWS and serverless
Get an advanced AWS certification
Improve FullStack bulletin
More serious and constant BJJ training!

That’s it for this year review!

I am not sure what to think if you are still reading… Either you are a stalker or a bot! Or maybe a recruiter, or, hopefully a fellow developer who is trying to make similar career plans.

If you classify yourself in the latter category, I would really love to know what was your biggest achievement in 2019 and what do you plan to achieve in 2020. If you want to mention some failure as well, that would be super interesting to hear too, especially if you can turn it into a littre retrospective 🙃

All the best, CIAO!

2020 - A year in review

Luciano Mammino — Sat, 02 Jan 2021 12:50:00 GMT

As we transition into 2021 and we leave this crazy 2020 behind, it is a great time for me to reflect on my what happened in 2020 from a career perspective and make some plans for the year to come.

Usual spoiler alert: this is the fifth year in a row for me writing this kind of post. This is going to be a super boring one that I write for myself to reflect on where my career is going and what I am learning every year.

I hope I am setting the right expectation here. If you decide to go ahead and read this, get comfortable and grab a coffee because it’s going to get quite BOOORING.

Yeah, this is a boring GIF. A silly attempt to warn you and (hopefully) to make things slightly less boring! 😜

An important reflection

Before getting started, I want to make a small but important reflection about this crazy 2020, signed by a worldwide pandemic and a global crisis. And, by the way, I dare to remind you that none of this will be magically over just because we are in a new year now… 😟

This is something that no one could have predicted and it should rightfully shift our perspective towards society and ourself as individuals.

Of course, I am not going to tell you how you should reconsider your life, but I want to share how I am reconsidering some small elements in mine.

In the last few years, I have been quite eager to grow as a professional and I tried to accomplish more and more every year, sometimes at the expense of time spent with the people that are dear to me. In retrospective, I recognize that sometimes I might have been a bit too selfish, and I should try hard to change that…

I have been quite lucky in 2020 because even in these strange times, I have been one of the privileged ones. I work in an industry that it is still thriving (now more than ever) and I didn’t lose my job, nor did I have to take a pay cut. I somehow managed to stay healthy throughout last year and so did most of the people dear to me.

I hope that most of the people reading this could say the same, but we all can’t ignore the fact that the majority of people out there are struggling and suffering.

I don’t think, as an individual, that I can have any significant impact to make things better on a large scale, but I can have an impact on my local community and with the people I care about.

I am not planning anything big here. I want to try to be more aware and careful with the people around. A phone call or just asking “how are you today?”, “what are you doing?”, “I was thinking about you…” or “can I help you with anything?” could be small gestures but they might mean a lot for the people you care about, even at distance.

I have tried to do these things a little bit this year, but I’ll try to it more next year. I want to encourage you to do the same.

In short, I am wishing for 2021 that we can be more compassionate and support each other more.

Together we can face any challenge as deep as the ocean and as high as the sky.
— Sonia Gandhi

Node.js Design Patterns - Third edition

Mario Casciaro and I have been working on the third edition of Node.js Desing Patterns (the book) since September 2019. This new edition finally came out this year at the end of July 2020.

We have been spending countless hours working in tandem on this new edition, which resulted in a full rewrite of the previous edition and a lot of additional new content. The print edition turned out to be massive counting more than 660 pages!

A lot has changed in the Node.js landscape since 2016 (when we published the second edition) and in this book, we tried to take full advantage of new interesting capabilities like async/await and ESM. We have also grown a lot as engineers in the last 4 years and this helped us to put things on a different perspective and improve the overall quality of the book compared to the previous editions.

We re-imagined the book as a learning path and we decided to add exercises at the end of every chapter. This allows readers to test their understanding of the main concepts. In this perspective, we also created a wiki and a discussion board for readers to exchange opinions and collaborate towards writing their own solutions.

I am quite proud of the final result and I consider this book one of the greatest achievements of my career so far. A mandatory “thank you” goes to Mario for involving me in this awesome project!

But this result is not something that can only be attributed to our hard work as authors. It is important to shine a light on all the people that contributed and helped us to make this new edition a reality.

First and foremost, all our readers from the previous editions. In the last few years, we have collected tons of feedback and ideas on what could have been improved and how. I want to dedicate this new edition to all of them and I hope this book will bring value to the Node.js community as a whole.

Our publisher, Packt, has been helping us a lot. Packt made sure that this new edition would result in a higher quality product compared to the 2 previous editions.

Last but not least, the incredible team of technical reviewers made up by some of the most talented engineers I had the pleasure to engage with in my career. This has been the tip of the spear for us and the element that contributed the most to the final quality of the book. I want to thank the reviewers one by one (in random order): Romina Miraballes, Roberto Gambuzzi, Kyriakos Markakis, Liran Tal, Tomas Della Vedova, Peter Poliwoda, and Minwoo Jung. If you are reading this, make sure to follow all of them on Twitter… Trust me, you won’t regret it. 😇

Unfortunately, I can’t disclose any data, but this new edition has been doing quite well so far. We got some good recognition (if you need a proof of that, you could check out the reviews on Amazon) and it has been ranking quite high on Amazon in several categories (including JavaScript).

Thanks a lot to Goran Jawdat for sending us this amazing screenshot!

If you are interested in finding out more about Node.js Design Patterns, make sure to check out the official website. Also, we recently launched a blog section that you might want to check out as well!

These are not books, lumps of lifeless paper, but minds alive on the shelves.
— Gilbert Highet

Conference Talks

After a few years involving travelling a lot to speak at many different conferences around the World and with all the effort I put on the third edition of Node.js Design Patterns, I didn’t feel like I wanted to do any talk this year. So, I did not apply for any conference or meetup.

With that being said, I still ended up delivering 13 different talks, which put me exactly on par with what I did last year!

I was extremely surprised and flattered to have received a lot of invitations to talk at several events. And, yes… when you get an invitation, it’s very hard to say NO… Needless to say that, in 2020, most of these conferences were remote.

What made me particularly happy is that some of these talks are not regular tech talks, but interviews about my career path. I am quite happy that I had the opportunity to tell my story and, through my failures and successes, to have an opportunity to inspire other people to try their best every day.

Thanks to Francesco Sciuti, Simone Torrisi, Rafael Del Nero, Francesco Ciulla for interviewing me and everyone else that invited me to speak at their own meetups and conferences! Seriously, THANK YOU!

This is the full list of talks for my 2020. You can find more details and links in the speaking section of this blog:

“Middy.js: a powerful middleware framework for your Node.js lambdas”, Dublin GCS Connect Serverless (Dublin, Ireland)
“Writing a tech book: it’s doable! (Interview — in Italian)”, Acadevmy - Uno di frontend (remote)
“Is the pot full of gold really in Ireland? (Interview — in Italian)”, Acadevmy - Chiacchiere per DEVulgare (remote)
“An overview of JavaScript and Node.js”, Liferay Universities Series (remote)
“Node.js scalability tips”, Shift conference (remote)
“Unbundling the JavaScript module bundler”, Road to Coderful (remote)
“Node.js scalability tips”, CityJS 2020 (remote)
“Tomorrowdevs interview (#storiedidevelopers)”, Tomorrowdevs (remote)
“An Intro to Eleventy: Static Site Generation Made Fun Again”, CorkDev (remote)
“The path to become a Principal Engineer - interview with Java Challengers”, Java Challengers (remote)
“Node.js scalability tips”, Codemotion online 2020 (remote)
“An Intro to Eleventy: Static Site Generation Made Fun Again”, MancJS (remote)
“The process of writing a successful programming book”, Francesco Ciulla’s Channel (remote)

After having delivered a total of 54 talks and workshops in the last few years, I have to admit I feel like I am running out of things to say… Maybe it’s just my imposter syndrome talking, but I am not sure I want to commit to doing any talk for next year. So, I will probably keep this open-ended. If some good opportunity presents itself, I’ll take it…

I believe in the power of ideas. I believe in the power of sharing knowledge.
— Ory Okolloh

Career

In 2020 I joined Fabfitfun as a Principal Software Engineer.

Fabfitfun is a very interesting company with a lot of technical challenges, regarding very spiky traffic (serving millions of users on a regular basis) and an evolving architecture where we are transitioning from a legacy monolith to a set of decoupled microservices.

The main headquarter is in Los Angeles, but I joined a new team based in Dublin, which was formed only at the beginning of 2020.

Even if I have been at Fabfitfun for less than a year I feel like I had several opportunities to have an impact.

I am currently spending a lot of time wearing many different hats and trying to help in different areas. From building new services (mostly using Python and the excellent fastapi framework), building new integrations and automation around our corporate Slack, optimizing existing solutions around scalability, security and authentication and helping to decouple the legacy monolith into new services deployed on Kubernetes. In the last few months, I have also been organising an internal monthly meetup to help different teams located worldwide to have a place where they can share their passion for technology.

I admit it has been a bit of a bumpy ride at times. I suppose that starting a new remote office in a crazy unpredictable year like 2020 has been the main reason for some hiccups, but things are getting better as we integrate more with the rest of the company and I am definitely blessed to be working with a world-class team of professionals and to be able to share my expertise every day.

On top of this, I have been quite lucky I had the chance to visit the headquarter in Los Angeles, just before the whole pandemic exploded.

Let me prove that to you with a couple of pictures:

This is me at the Hollywood sign!

From the left to the right: this is Padraig O’Brien, me and Roberto Gambuzzi at the Griffith Observatory (yes the one from La La Land).

And yes… in case you are wondering, we also did a lot of useful work while we were in LA! 😇

All things are difficult before they are easy.
— Thomas Fuller

Side projects

In terms of side projects, 2020 has been an interesting year. I already spoke at length about Node.js Design Patterns, so now let’s discuss Middy, FullStack Bulletin and new project: Linkerflix.

Middy

Middy is a Node.js middleware framework for AWS Lambda. I have been working on this project since the early days of Lambda (even though the first public commit was made the 3rd of August 2017).

In April 2020 we finally launched Middy 1.0.0! After a few years of development and contributions from many awesome engineers, this was a pretty big milestone and I am quite proud of it!

What’s even better is that throughout the year I found out that several interesting big companies are actually using Middy on a daily basis. Unfortunately, I don’t have an official list, but here I have some proof that the LEGO Group does use Middy!

This is Hannah Tucker McLellan (@hannahintech) on the ServerlessDays Virtual October 2020 Discord channel. Thanks to @eoins for sending me this amazing screenshot and, of course, to Hannah for disclosing this info!

Interesting enough, I think I reached this milestone a bit burned out regarding the whole project. I haven’t been working a lot with Lambda and serverless in the last couple of years (definitely not as much as I used to do before), so I felt like I was dragging my heels to keep the project alive while I wanted to focus my attention somewhere else.

Because of this, my release post on this blog turned out to be a little bit of a cry for help to keep the project alive.

Thankfully the community around the project has been very patient and supporting. In particular, Will Farrell stepped up and offered to become the lead maintainer of the project. Since then, he has been doing an incredible amount of work releasing other 5 minor versions of the framework and he is already working on a new major release!

If you have been using Middy, please consider joining the community and giving back to the project! You can also sponsor Will to support the time he is investing in the project!

Thanks a lot, Will! 🙌

One final observation about Middy is that the project has kept growing steadily throughout 2020, surpassing the astonishing number of more than 1 million downloads for the @middy/core package only.

These are not crazy big numbers in open source, but they are not small either. Especially if we consider that serverless is still new and there is a lot of potential for growth. If nothing else, these numbers make me proud to have started a project that is giving value to development teams.

UPDATE: If you want to learn more about what happened for Middy in 2020, Will has done a great recap on GitHub.

FullStack Bulletin

FullStack bulletin is a free weekly newsletter about full stack web development. I have been running this project with my dear friend Andrea Mangano for the last 3 years. The project is totally open source and uses tons of automation to minimise the amount of manual work.

In 2020 we didn’t change anything big in the format but the audience kept growing steadily:

One interesting thing is that toward the end of the year we surpassed some of the thresholds around free plans for some of the services we use, so we decided to start asking our subscribers for help.

If you are interested in supporting FullStack Bulletin, you can do so by donating on Patreon or sponsoring one of the next issues.

There are probably many ways we can improve FullStack Bulletin, so if you have any suggestion, please do let us know!

Linkerflix

In 2020 I decided to start a new small side project: Linkerflix.

Linkerflix is an idea that was born from the frustration I had while trying to create links to allow readers to buy the book from Amazon. I wanted to have a system that, by using only one link, would automatically redirect the user to the closest Amazon shop. For instance, a user clicking from Italy should be redirected to the book page on Amazon.it while someone in Ireland would go to Amazon.co.uk (we will see if this changes with Brexit, but that’s another topic 😅).

I have been researching a solution for this for a while until I found out that there’s an official solution from Amazon called OneLink.

Unfortunately, I didn’t have a great experience while trying to setup OneLink for my account and in the end, I did realize that OneLink only works on a pre-determined website that you have to pre-configure with Amazon (e.g. this blog). Since you have to prove that you own every website you link, you cannot use OneLink on social media channels like Twitter or Youtube… such a bummer!

To overcome these limitations, in August 2020 I created 2 small alternative open source solutions: not-one-link and not-one-link-lambda. Yes, the name is a pun! Sorry Amazon… 😇

Later in the year, as I noticed many content creators suffered the same struggle I faced with sharing links to products on Amazon, I thought it might be a cool idea to try to build a small SaaS service to make it easier to adopt the solutions I already created with not-one-link. So I took one week off of work and I tried to build a super quick MVP. I documented the entire week on Twitter. If you are curious to see which technologies I used, how I structured the work and what kind of challenges I faced, you can read all the tweets on Threader:

The next big piece of work in line is more product facing. Now that I have an MVP I have to engage with potential users and try to get their feedback. Hopefully, at the end of this process, I will have found some degree of product market fit… or some new ideas for a pivot!

I want to dedicate some of my free times in 2021 to focus on this. Wish me luck!

Your first projects aren’t the greatest things in the world, and they may have no money value, they may go nowhere, but that is how you learn - you put so much effort into making something right if it is for yourself.
— Steve Wozniak

Blog

If I have to be honest, I feel like I have been failing this blog a little in 2020. The main reason why I think that is because I have published only 3 new articles:

Well, it’s easy to say I wasn’t a very prolific author… But let’s see some numbers to check the impact on the blog.

As you can see from this graph, I got a total of 77.886 pageviews (72.043 unique). Compared to last year, where I had almost 120.000 views, it is a significant drop (~35% decrease).

It is also interesting to see that what’s driving most of the traffic is old articles:

And, also that visitors are coming mostly through organic search:

I am not a consistent author and I have been struggling for years to come up with a backlog of article ideas to pursue. I am not sure I want to make a promise to change this in 2021. Maybe I should just accept that I will keep blogging randomly and that this is not an area where I need to be consistent.

I’ve never managed to keep a journal longer than two weeks.
— J. K. Rowling

External blog posts

While I haven’t written a lot of new blog posts on this blog, I actually ended up publishing a decent number of articles on other sites. 7 new posts to be precise:

I am actually lying a little bit here… The two articles about AWS Lambda and Go are something I wrote during 2019 and they just got published this year… So, in reality, I have been publishing only 5 new posts.

Still, I don’t feel like this is to bad and I don’t want to commit to doing more for next year. At this point, I should embrace the fact that I will write blog posts only when there is something cool that I want to share.

There is maybe one exception I will try to make about this. I have been reading a bit about SEO during 2020 and I started the blog section on Node.js Design Patterns website to experiment with some of the ideas I had while learning more about SEO.

If I have to be serious about this I will have to publish a few more articles. At least one every month for the next 12 months. I already came up with a list of potential articles, so I am already laying the groundwork here…

I suppose, at the end of the year, we should be able to review my commitment and see if I managed to stay consistent here! 🤞

I don’t journal to ‘be productive.’ I don’t do it to find great ideas or to put down prose I can later publish. The pages aren’t intended for anyone but me. It’s the most cost-effective therapy I’ve ever found.
— Tim Ferriss

Building a Twitter audience

Another area that kept me busy in the second half of 2020 was working on building a Twitter audience. This is nothing new for me. I have been trying to do this for years since I started this blog and joined Twitter.

What’s changed this year is that I have been trying to take a slightly more conscious and structured approach.

How am I structuring things?

Well, before getting into that, let me give some credit to Daniel Vassallo and his great Everyone Can Build a Twitter Audience course. I watched his course this year and it helped me realise I did not have any plan or strategy.

In fairness, I don’t think I have come up with a particularly sophisticated plan now, but I have definitely identified some mistakes I was doing and I have tried to correct them. Let me list some of them.

I was tweeting a lot of links to articles (like a lot!). This is mainly because of the automation I set up for FullStack bulletin. Whenever I found an interesting article related to full stack web development I was tweeting it to add it to the list of selected articles for the next issue of FullStack bulletin. I don’t think tweeting articles is necessarily a bad thing, but it does not help me to express my personal opinion about things or to add colour to interesting tech topics or projects. Since I realised this I moved all these tweets to FullStack Bulletin’s Twitter account. Now, whenever there is something article I feel like I want to tweet I ask myself “can I add any valuable comment to this tweet?”. If the answer is yes, then I do share the article with some additional comment, otherwise, I just share it through the FullStack Bulletin profile. I am extending this logic to any retweet. I have stopped almost all retweets unless I feel like I can add some colour to the original tweet.

I was not giving enough. Building an audience is mostly about giving and very little about taking. What I mean by this is that revisiting some of my previous tweets I realised I was not giving my audience many reasons to follow me. Since I realised this mistake I am actually tweeting less, but when I do try to post something meaningful and useful. It can be a quick tip about something I just learned and that might be useful to other engineers. It might be an opinion about a new technology I have been playing with… This also helps to show interest and expertise on a given area and this should encourage people with similar interests to engage more.

And this brings me to another important mistake: I was not engaging enough with people with similar interests. This is an easy one to fix in principle, but it is one that takes a lot of time. It means you have to spend a lot more time on Twitter listening rather than just talking.

To be fair, this one last point was a bit illuminating to me. When I realised this mistake I felt like I have been using Twitter in the wrong way for so long… In fact, I have been using Twitter more as a news broadcasting channel rather than a place for conversations.

Realising all these mistakes made me change the way I used Twitter and I have a feeling that now I can use it to have better conversations and engage more with people and learn from them.

But let’s have a look at the numbers to see if all of this had an effect on my followers base:

This graph represents the number of new followers month by month. I started changing my behaviours around the end of July 2020 and you can see there is a visible inflexion point there.

I know that this is not a sophisticated analysis, but I would like to think that I am onto something here. I’ll keep collecting raw data and eventually do a better analysis to try to find some interesting correlation between actions and the number of new followers per month.

Meanwhile, if you are curious, you can have a look at my raw data.

If you think you’re boring your audience, go slower not faster.
— Gustav Mahler

Open Source

In 2020, I did not stop doing my small part to contribute to open source. Here’s a quick list of my contributions on different projects on GitHub:

Middy: Node.js middleware framework for AWS Lambda.
Fastify website: Fastify’s website.
jwtinfo: A command-line tool to get information about JWTs (Json Web Tokens) - my first open source Rust project!
PacktPublishing/Node.js-Design-Patterns-Third-Edition: Repository with all the code examples from the third edition of Node.js Design Patterns.
minikube-helm-init-container: Sample helm based web application using an init container.
univ: Universal JavaScript application example without server side transpilation (React + Fastify).
unsplash/comment-on-pr: A GitHub Action to comment on the relevant open PR when a commit is pushed (contribution).
luciopaiva/heapify: The fastest JavaScript priority queue out there. Zero dependencies (contribution).
Kevinrob/guzzle-cache-middleware: A HTTP Cache for Guzzle 6. It’s a simple Middleware to be added in the HandlerStack (contribution).
guzzle-fs-cache-middleware: A simple FS backed guzzle cache file middleware compatible with php 5.6.
fastify-preact-htm-boilerplate: Quickly bootstrap your next web app with Fastify, Preact and htm.
streams-examples: A bunch of examples on how to use Node.js streams.
guzzle-apcu-fs-cache: A zero config cache storage for guzzle-cache-middleware that tries to cache on Apcu and fallbacks to filesystem.
algorust: Randomly playing with algorithms and Rust.
DataDog/java-dogstatsd-client: a java statsd client library by Datadog (contribution).
nodejs/undici: An HTTP/1.1 client, written from scratch for Node.js (contribution).
jsmrcaga/action-netlify-deploy: a GitHub Action step to easily deploy on Netlify (contribution).
financial: A Zero-dependency TypeScript/JavaScript financial library (based on numpy-financial) for Node.js, Deno and the browser.
nim-node-hello-world-http-bench: A super simple benchmark to compare Nim raw http performance against Node.js.
denoland/deno_website2: deno.land website (contribution).
react-sca: A super simple scaffold for React projects for those who believe CreateReactApp is bloated.
11ty/eleventy-img: Utility to perform build-time image transformations in Eleventy (contribution).
not-one-link: A simple Node.js library that allows you to re-map Amazon links from one country to another (Like Amazon OneLink™️ but simpler to use).
not-one-link-lambda: An AWS lambda that allows you to host your own alternative of Amazon OneLink™️ to rewrite Amazon URLs.
kryz81/awesome-nodejs-learning: About A list limited to the best Node.js Learning Resources (contribution).
mre/idiomatic-rust: A peer-reviewed collection of articles/talks/repos which teach concise, idiomatic Rust (contribution).
11ty-sample-project: An example project to illustrate the main features of Eleventy.
mattermost/mattermost-webapp: Webapp of Mattermost server (contribution).
xdesro/personalsit.es: A little directory of people’s personal sites (contribution).
next-sample-file-upload-app: Playing with Next.js and building a sample file upload app for educational purposes.

This is a total of 30 projects I have been involved with in 2020. I used to compare this metric year by year, but I am finally realising that it might not make a lot of sense to do so. Some of these contributions are quite small (e.g. a README fix) while others required me a significant investment of multiple hours, days or even weeks.

I don’t know if there is any decent way to compare my involvement in Open Source between years, but from now on I will be happy as long as I can write something in this section.

One project that is definitely worth discussing though is Fastify website. This year I didn’t have a lot of time to invest in it and I realised that I have become a blocker for the project. This probably shows that I haven’t structured and documented the project well enough to allow other people to contribute. Of course, this is bad, so definitely an area to improve on for 2021.

Each of us is a unique strand in the intricate web of life and here to make a contribution.
— Deepak Chopra

Previous year goals

Now it’s finally time to check how I performed against my previous year goals:

✅ Get a new job
✅ Do only a few (invitation-based) conferences (3 or 4): I smashed this one (unexpectedly)!
😐 Write some good quality blog posts (at least 4): I think I failed at this, unless we want to cumulative count all the blog posts, even outside this blog…
✅ Release Middy v1.0 stable and have a plan for the future of the framework together with the community around it.
✅ Publish a new book (or a new edition!)
✅ Learn more about Rust: I still feel like I am only at the early days, but 2020 was definitely the year were I felt Rust started to click for me!
✅ Learn more about Kubernetes: I definitely did learn a ton about Kubernetes in 10 months at Fabfitfun.
😐 Keep learning AWS and serverless: I definetely didn’t do much around this area except maybe trying to read some relevant news and articles.
😭 Get an advanced AWS certification: Definitely failed at this one. I did renew my subscription for A Cloud Guru but then I did not spend enough time to be able to try to get an AWS Solution Architect professional certification…
😐 Improve FullStack bulletin: FullStack Bulletin definitely kept growing steadily but I feel I didn’t invest any time trying to figure out what could have been improved.
😭 More serious and constant BJJ training: well… I did stop this one entirely and you can all imagine why in 2020… I compensated it a bit by going for running and a great thank you goes to my dear friend Luca (@lucamarchesotti) for motivating me!

Looking back, I think I have actually achieved more than ever this year. I am not going to lie, this comes a bit unexpected…

One thing is certain though. I should try to re-asses these goals on a more regular basis, rather than just doing it at the end of the year. I think doing it maybe on a quarterly basis can help me to see if I am on track or not and where it makes sense to spend most of my time.

Exercise to stimulate, not to annihilate. The world wasn’t formed in a day, and neither were we. Set small goals and build upon them.
— Lee Haney

Expectations for 2021

Getting close to the end of the year I have been spending a lot of time trying to think where I want my career to go and which areas I might want to invest my time.

Before giving you my conclusion, let me thank Padraig O’Brien for spending a lot of time mentoring me on this and for triggering some great reflections.

What I realised is that I want to grow in 3 different directions:

Become a better Software Engineer: learn new technologies, step out of my comfort zone of FullStack wen development.
Become a better Cloud Architect: I think I am currently junior/mid-ish when it comes to cloud architectures. But this is an area where I enjoy a lot spending my time and one where there is an increasing number of opportunities.
Become a more serious Indie Maker and startupper: I have launched (and failed!) startups before. I have also launched several products. This is not a new one for me, but this year I have been realising that I have always approached this topic mostly from a technology perspective. For this reason, I think I have always delivered well on the tech side, but quite poorly on the product, marketing and overall strategy. I definitely want to have excuses to explore these areas more and become a more complete individual when it comes to build and launch a product.

So, based on these 3 broad topics, this year, I want to try to organise some more practical goals in categories:

Get better at Software Engineering

Continue to get better at Rust: I think the language it’s just a tool, but it can definitely help to step outside the comfort zone of higher-level interpreted languages and learn a lot more about lower-level concepts like memory management.
Improve Fastify website: make sure that it is easy for other people to join the project and contribute.

Become a better Cloud Architect

Keep learning about AWS and cloud deployments
Get a new certification
Keep learning about Kubernetes

Become a more serious Indie Maker

Complete the research around Linkerflix
Launch a new product (either Linkerflix or something else)
Keep building an audience
Publish 12 articles on Node.js Design patterns blog

In between goals is a thing called life, that has to be lived and enjoyed.
— Sid Caesar

Conclusions

Woah! Have you really been reading all this stuff?! At this point, you probably know me better than my mom! 🤣

Well, what can I say? I am flattered, but also curious to know what you think about all that I have been blurting here…

Definitely do leave a comment and let me know, but also please share what will be your career goals for 2021… I am really curious to find out what you have in mind for yourself!

Ciao! 👋

What's in a JWT (Json Web Token)?

Luciano Mammino — Mon, 05 Oct 2020 09:00:00 GMT

If you ended up on this article, chances are that you have been seeing JWTs (Json Web Tokens) for a while and you are curious to find out what they really are, what’s inside of them and how they actually work! Why did they get so mainstream anyway?! 🤔

In this article we will be trying to address this topic in a short but (hopefully) effective fashion!

It’s just a string with a well-defined format

JWT stands for JSON Web Token and such a token is just a string composed by 3 parts:

a header
a body (sometimes also referred to as payload)
a cryptographic signature.

Let’s have a look at a more concrete example, a JWT looks like this:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJoZWxsbyI6ImZyb20gSldUIn0.XoByFQCJvii_iOTO4xlz23zXmb4yuzC3gqrWNt3EHrg

The three parts are separated by a . (dot) character:

header: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9
body: eyJoZWxsbyI6ImZyb20gSldUIn0
signature: XoByFQCJvii_iOTO4xlz23zXmb4yuzC3gqrWNt3EHrg

Header and Body are JSON strings (yeah, that’s why the are called JSON web tokens…). This is somewhat hidden, because those two strings are encoded using the Base64Url algorithm (a URL-safe variation of the standard Base64 encoding algorithm).

If we decode them this is what we get:

header: { "alg": "HS256", "typ": "JWT" }
body: { "hello": "from JWT" }

Which now looks like proper JSON!

The signature part contains raw bytes which represent a cryptographic signature of header and body. The signature is also encoded using Base64Url and it can be used to verify the authenticity of a token.

If you try to decode the signature and visualize it a s UTF-8 string you will probably see some rubbish like this:

Again, this is because the signature just contains raw bytes which do not represent a valid UTF-8 string. So yeah, there is no point in trying to decode and visualize a signature. 🤗

When are JWTs useful

JWTs are mostly used as a mechanism for “stateless” claims exchange. We can define a claim as some piece of information that we want to certify and propagate through various systems.

A good example is authentication and authorization, were our claims could be information about the user session (user id, roles, etc). Let’s try to discuss this example with a (rather crappy) illustration:

In this picture:

John is authenticating against an auth server. The auth server recognizes his credentials and gives him back a token
John can now use the token to connect to specific services, for instance the pizza service.

Every time John makes a request to a service, he will need to attach his token to the request. The service can look at the token to understand if the request can be authorized. The service can read the information embedded within the token to understand that the request is coming from John and can verify that the signature was applied by the Auth server.

This process is “stateless” because this validation can be done without having to make an explicit request to the Auth server. Of course every service needs to know the Auth server public key to be able to validate the token signature.

A stateless mechanism like this, is great for distributed systems or, in general, systems that deal with a high load of requests.

Authentication is not the only use case for JWT. For instance a JWT can also be used to generate confirmation links like “confirm your subscription to this newsletter” or to generate “password reset links”.

Now you can be nosy

Now that you know how a JWT token is made you can start to be nosy and look into the JWT tokens you bump into.

You can copy-paste tokens into jwt.io or use a command-line tool like jwtinfo to read the content of the header and the body in plain text.

Trust me, if you do that you will be surprised by how many interesting information you can find there!

Closing notes

JWT tokens are an interesting approach to authentication and they are particularly convenient in distributed systems when we want to minimise the communication and the amount of load to the authentication provider. Once a token is created, it can be validated without the need to call the authentication provider again.

If you are curious to understand more about how JWT works (and even how you could crack tokens using Node.js 😼) check out this talk I gave a couple of years ago at BuzzJS 2018 in the beautiful New York City:

While you watch that, you can have the slides too! 🤗

That’s all for now.

Thank you and see you on the next post! 👋

Middy 1.0.0 is here

Luciano Mammino — Sun, 26 Apr 2020 15:25:00 GMT

Middy, the Node.js middleware framework for AWS Lambda, has finally graduated to 1.0.0! A long awaited milestone for the project.

In this post we will discuss what middy is and what are the main features of this first stable release.

What is middy

If you are hearing about middy for the first time, middy is a middleware framework for AWS Lambda written in Node.js.

Middy has the goal to simplify the way you write Lambda code, essentially by providing a convenient middleware abstraction (similar to the ones you find in frameworks such as Express or Fastify). A middleware framework can simplify code re-use, testing and avoid code duplication. Middy also provides a collection of middlewares that can be installed and configured to fullfil very common use cases (authentication, authorization, caching, data validation, deserialization and serialization, etc.).

Just to give you a feeling of what using middy looks like, here’s a snippet of code:

// import core
const middy = require('@middy/core')

// import some middlewares
const jsonBodyParser = require('@middy/http-json-body-parser')
const httpErrorHandler = require('@middy/http-error-handler')
const validator = require('@middy/validator')

// This is your common handler, in no way different than what
// you are used to doing every day in AWS Lambda
const processPayment = async (event, context) => {
  // we don't need to deserialize the body ourself as
  // a middleware will be used to do that
  const { creditCardNumber, expiryMonth, expiryYear, cvc, nameOnCard, amount } =
    event.body

  // do stuff with this data ...
  // e.g. send it to a payment gateway and record the transaction

  return { result: 'success', message: 'payment processed correctly' }
}

// Notice that in the handler you only added base business logic (no deserilization,
// validation or error handler), we will add the rest with middlewares
const inputSchema = {
  type: 'object',
  properties: {
    body: {
      type: 'object',
      properties: {
        creditCardNumber: {
          type: 'string',
          minLength: 12,
          maxLength: 19,
          pattern: 'd+',
        },
        expiryMonth: {
          type: 'integer',
          minimum: 1,
          maximum: 12,
        },
        expiryYear: {
          type: 'integer',
          minimum: 2020,
          maximum: 2027,
        },
        cvc: {
          type: 'string',
          minLength: 3,
          maxLength: 4,
          pattern: 'd+',
        },
        nameOnCard: { type: 'string' },
        amount: { type: 'number' },
      },
      // Insert here all required event properties
      required: ['creditCardNumber'],
    },
  },
}

// Let's "middyfy" our handler, then we will be able to attach middlewares to it
const handler = middy(processPayment)
  .use(jsonBodyParser()) // parses the request body when it's a JSON and converts it to an object
  .use(validator({ inputSchema })) // validates the input
  .use(httpErrorHandler()) // handles common http errors and returns proper responses

module.exports = { handler }

From the example above, you can see that middy helps you to keep the handler code (the core of your lambda function) as clean as possible. Your handler is now free from having to handle all sorts of pre and post conditions such as data serialization and validation. Managing these pre and post-conditions can be done with the help of default of custom middlewares and those middlewares can easily be reused across different lambdas as needed.

This is nothing new if you come from frameworks such as Express or Fastify, but surprisingly, this is yet not so common in Lambda-land and middy is the first (and one of the very few) frameworks that offers you this capability in this context.

The best feature of middy is that, since it focuses only on the code layer, you can use it with any other infrastructure-level tool you commonly use, including Terraform, Cloudformation, SAM and the Serverless Framework. The integration is seamless, just install and use the library in your code (as you do with any other external dependency from npm).

Middy is not so new (in open source terms). It has been around for almost 3 years receiving contributions from more 50 different people, it has almost 1500 GitHub stars (please give it some more if you are reading this 😇) and 5 digits downloads per week. Not impressive numbers, but, in my humble opinion, these are numbers good enough to demostrate that there is interest and maturity around the project.

If you want to find out more about middy from a technical perspective, check out the official website at middy.org.js.

What’s in the 1.0.0

The main change in version 1.0.0 is that now the core and the official middlewares are distributed in separated NPM packages. This allows you to install only the packages you actually need, which helps with keeping your lambdas slim and allow them to bootstrap fast.

Other interesting changes are new official middlewares, bug fixes and a number of improvements in terms of features, stability and usability.

You can read the full list of changes in the release page on GitHub.

If you are already using middy version 0.x, you can read the dedicated migration guide.

The community behind middy

I am personally quite impressed and honoured about all the contributions spontaneously provided by the growing community around middy. I want to personally thank you and celebrate all the people that contributed to the project with the following video (click on the picture to see the full video on Youtube).

Thank you!

What’s next for middy

This is the hard question for me. The community is definitely asking for a number of interesting things like:

Better integration with TypeScript (#203, #300, #316, #373, #506)
Better (and more clear) support for promises and Async/Await (#392, #414)
Better documentation (#173)

All in all, my feeling is that the framework can still be improved a hell of a lot in terms of usability.

Follow the list of issues on GitHub to find out what are the common topics and feel free to open new issues if you want to suggest new features or improvements.

Looking for help

Here’s the moment where I am to be honest with myself…

As a maintainer of the project I don’t feel like I did a great job to make this project progress. I limited myself to doing code reviews and merge feature requests quite passively. I have been lacking vision and initiative to be able to steer the project forward and bring it to the next stage.

The truth is that I haven’t been as involved with serverless as I used to be in the last 2 years. I have probably been lacking perspective and I lost pace with the evolution of serverless and Lambda, so it’s hard for me to be proactive and have a solid vision for the project.

I want to shed some light on some people that helped carry the project forward and I want to give them a special thank you (in no particular order):

If you use middy, don’t be shy and please show yourself on GitHub. I created a new repository just to foster conversations at github.com/middyjs/discuss. Open an issue just to say hello and tell us how and why you are using middy and what you would like to see happening next.

I’d love to be able to create a more cohesive community and possibly to pass the torch to a group of people that is actively involved in serverless and that can drive the project forward better than I am currently doing.

Alternatives to middy

In case you don’t like middy or you just want to explore possible alternatives, these are some of the alternatives I know about:

curveball by @evert
lambcycle by @juliantellez
lambda-middleware by @dbartholomae
aws-serverless-express by AWS itself
aws-lambda-fastify by the fastify community
aws-lambda-middleware
nodejs-lambda-middleware
lambda_decorators (python)
beeline (python)
vicuna (rust)
aws-lambda-go (go)

Feel free to suggest me other alternatives in the comments and I’ll happily add them to the article.

Closing notes

That’s it. I feel quite happy and relieved to know that we finally reached this milestone and I look forward to seeing what’s next for the project.

All the best and please let me know in the comments what do you think.

Regards

PS: thanks to @theburningmonk, @StefanoAbalsamo, @PadraigOBrien and @quasi_modal for feedback and reviews.

Lean NPM packages

Luciano Mammino — Sun, 08 Sep 2019 16:40:58 GMT

Every developer on the planet knows how modular Node.js and the JavaScript ecosystem have become. This is probably due to the great job that package management systems and registries like bower (discontinued) and npm carried over in the last few years. I personally believe that this is also a consequence of the “many small modules” philosophy that has been popularised within the JavaScript ecosystem.

This is great, but all that glitters is not gold… Look, for instance, at this picture for a second:

Yeah you have probably seen this picture before and it’s probably not funny anymore… Anyway, this picture is a good summary right there on how this “many small modules” idea got a little bit out of hand within the JavaScript ecosystem.

Every time you run npm install you basically start to get so many files that you might feel like you are downloading the entire world wide into your hard drive! 😰

There are even tools that try to scout for node_modules folders in your system and get rid of them (E.g. wipe-modules). There are also some developers who showed how all the node_modules folders in their system is making their backups too slow (see tweet)!

Some like to make fun of this issue or they just complain about it. In this article, I don’t want to do any of those things. I’d rather prefer to be a little bit more constructive and try to share some simple techniques to keep your NPM modules as lean as possible, so that other developers will save bandwidth and time when pulling your modules from NPM!

Repository vs Registry

In some languages like Go or PHP, what you have in a module repository is exactly what you get through the package manager when trying to install the module. This is because the code you download through the package manager is actually coming straight from the repository (or from a proxy that keeps a copy of the repository). In this cases, the structure of your repository is fundamentally tied to the file structure of your module: what you get by installing a module is pretty much what you would get by cloning the repository.

NPM doesn’t work this way. In fact, NPM allows you to selectively push files into the registry, so you might end up with a very different file structure compared to what you have in your git repository.

While this interesting property of the system have caused some security issues in the past (see the event-stream module incident if you are curious), it also offers us an opportunity to be very selective with what we publish and keep the module lean.

This is especially important if you “build” your JavaScript code (e.g. using Typescript, Babel or a module bundler), so that the “distribution” (dist) version of your module is the result of a compilation/transpilation/bundling process. In such cases, you don’t need to publish the entire codebase on NPM as your users will be using only the dist version of your code. The same goes for tests, documentation, images and other files that won’t be used by the users of your module in their codebase, you should keep them only in your repository and avoid to publish them in the registry.

Conversely, you probably don’t want to keep dist code in your repository. This code can easily be regenerated by the build toolchain when necessary and there’s no point in tracking changes on the dist files when what you are really changing over time is the source code. In git you can use .gitignore to make sure dist files are kept out of the repository.

In short, registries are for production-ready code (dist) while repositories are for development code (src).

In the rest of this article we will see some ways to configure an NPM package so that all the unnecessary files will be excluded from the registry.

Publishing on NPM

With the NPM command line, npm publish is the de facto way of publishing new modules (or new versions of a module) into the NPM registry.

An NPM module is nothing else than a folder with a valid package.json file in it. It doesn’t have to be a git repository (in reality the definition of what an NPM module can be is a little bit more complicated, to get the full spiel, check out the official NPM documentation).

By default npm publish will publish all the files in the package directory (including subfolders recursively).

So the first thing to do is to be careful and make sure that you don’t have sensible files containing passwords, tokens or other sensible information in your project folder. It’s generally a good idea to keep those away from the module folder, just in case…

You should also try to avoid to keep unrelated files in the same folder. Yeah, I admit that many times I did some quick n’dirty wget to get something I needed while I was working on a module and ended up with a lot of unrelated stuff published in my module. Please be smarter than me, don’t do that! 😜

Default rules

Before starting to deep dive into the different ways you can specify the files to be included/excluded when you publish your package, let’s see first what are the default rules.

No matter what you do, there are some files that are always excluded:

.*.swp
._*
.DS_Store
.git
.hg
.npmrc
.lock-wscript
.svn
.wafpickle-*
config.gypi
CVS
npm-debug.log

Similarly, there are files that are always included:

package.json
README.md (and its variants, like README.markdown or README.rst)
CHANGELOG.md (and its variants, like CHANGELOG.markdown or CHANGELOG.rst)
LICENSE and LICENCE

Note that package-lock.json is NOT automatically included.

`.gitignore` & `.npmignore`

The first interesting property of npm publish is that, if your folder is also a git repository and you are using a .gitignore file, all the patterns listed in it will be used to exclude files.

So, for instance, if you have *.cache pattern in your .gitignore, all the files matching the pattern won’t be published in the registry.

We discussed already that you might want to have different rules between what you track in your repository and what you publish to the registry, so relying on one configuration to ignore files for both targets might not always be a good idea.

In those cases you can create a more specific file called .npmignore (which supports exactly the same syntax as .gitignore). If this file exists, npm publish will use that to exclude files, rather than using .gitignore.

This means that there’s no inheritance, the two files are totally independent. If you want a pattern to exclude files for both your repository and your registry, you will have to put the pattern in both configuration files.

One interesting lesser known (and rarely used) tip is that you can put .npmignore files also in subdirectories. The patterns specified in these files will apply only to the subtree of directories where the .npmignore is found.

The `files` field

If you don’t like the idea of blacklisting some files (in fairness, you might forget to exclude a file with some sensitive information in it…) you can also follow a whitelisting approach.

In fact, NPM allows you to use a field called files in your package.json to specify an array of file patterns to include in the package.

From the official documentation:

The optional files field is an array of file patterns that describes the entries to be included when your package is installed as a dependency. File patterns follow a similar syntax to .gitignore, but reversed: including a file, directory, or glob pattern (*, **/*, and such) will make it so that file is included in the tarball when it’s packed. Omitting the field will make it default to ["*"], which means it will include all files.

One important rule is that files included with the files field cannot be excluded through .npmignore. In other words, the files field has higher priority than .npmignore.

`.npmignore` vs the `files` field

As we said, .npmignore is effectively a blacklist of files, while the files field acts as a whitelist.

This means that, if the files field is populated, everything is excluded by default and only those files explicitly listed will be included in the packaged tarball.

You are probably wondering now, should I use the files field or the .npmignore file?

To be honest, I don’t think there’s a silver bullet here. Just pick the mental model (whitelist vs blacklist) that comes easier to you.

An example

I generally prefer to keep my folder structure simple and explicit by having folders for source (src) and distribution files (dist).

With this approach you can simply say that src is what you want to keep in your repo (excluding dist) and, viceversa, in dist is what you want to publish on NPM (excluding src).

Just to make a very simple example, let’s say we are building a new library and our code base contains the following files:

src/index.js: source code for our module logic (using ES2019 syntax, because we like to be cool! 😎)
src/index.test.js: unit test file
dist/index.js: distributable version of our module (transpiled to ES5 with babel)

Now we want to keep src/index.js and src/index.test.js in our repository (but not in your final package) and dist/index.js in our package (but not in our repository).

One way we can achieve this result is by adding dist/ to our .gitignore, this will make sure we never commit files from the dist folder to the repository. Then we can either use the .npmignore file or the files field to specify what goes in our package.

I personally prefer to use the files field, which in this case will be super simple.

{
  "name": "some-test-package",
  "version": "1.0.0",
  "main": "dist/index.js",
  "files": [
    "dist/"
  ]
}

Notice that I am also pointing the entrypoint (main) to our index.js file in dist. This is what will be used when our module is imported.

With this approach I can add all sorts of other files to my repo (e.g. integration tests, functional tests, images, documentation, etc.) and I won’t have to worry about polluting my final package and making the end user download a lot of stuff that they won’t need!

Testing the package files

But how do we know if our setup is correct? We don’t want to publish the package just to see if our setup is correct.

Thankfully there are at least 2 ways to preview what’s gonna end up in the registry with npm publish without having to actually publish anything.

The first way is npm pack, this command will create a tarball that contains all the files that will be published in the registry.

The output is actually pretty nice and it will list all the included files.

If we run npm pack on the package folder from the example above we should see something like this:

npm notice
npm notice 📦  some-test-package@1.0.0
npm notice === Tarball Contents ===
npm notice 74B  dist/index.js
npm notice 266B package.json
npm notice 13B  LICENSE.md
npm notice 39B  README.md
npm notice === Tarball Details ===
npm notice name:          some-test-package
npm notice version:       1.0.0
npm notice filename:      some-test-package-1.0.0.tgz
npm notice package size:  428 B
npm notice unpacked size: 392 B
npm notice shasum:        738776acad3cb41c549a884c6f9e946e7f367657
npm notice integrity:     sha512-QQS68QqFtfTGE[...]XmPGJpSYqmpKw==
npm notice total files:   4
npm notice
some-test-package-1.0.0.tgz

Note that only 4 files have been included:

dist/index.js
package.json
LICENSE.md
README.md

An alternative approach is to run npm publish in dry run mode with the flag --dry-run. With this approach no tarball is created but you will see the output of all the files that would be published with a normal npm publish run.

Conclusion

In summary, these are the main points I wanted to get across with this article:

What you have in your repository can (and probably should) be different from what you publish in the NPM registry.
You can exclude files by specifying patterns in .npmignore (similarly to .gitignore)
Alternatively, you can whitelist files by specifying patterns of files to be included in the files field in your package.json
There’s a list of files that are always included and, similarly, a list of files that are always excluded (see list above).
Be smart and only publish the bare minimum needed for people to use your library: keep your NPM package lean!

With these advices we are probably not going to solve the node_modules drama, but at least we can do our part to make it a little bit more bearable.

Please, let me know what you think about these advices here in the comments. Did you know about these configuration options? Did you use other strategies to keep your NPM packages lean?

I’ll see you in the next article. Until then, keep your NPM modules lean! 🤗📦

CIAO 👋

Where to go to learn Rust in 2021

Luciano Mammino — Sun, 28 Mar 2021 19:55:00 GMT

In this article, we want to provide a list of some free and paid resources that we loved the most in our journey to learning Rust.

Rust is certainly not the easiest of programming languages, especially at first glance, but once you can overcome the initial “wall of fear” and start to grasp some of the key concepts, Rust becomes a language that you are going to love and you will probably be looking for more and more excuses to use it and learn it further. For this reason, we wanted to collect a list of resources that can help new Rust adventures to find their path towards becoming real “rustaceans”.

It’s important to mention that this list is totally subjective and not comprehensive. We are only listing material that we had a chance to explore and that we enjoyed. We are sure there is still a lot of great content out there that we haven’t found yet! So if you think something is missing here, let us know in the comments box below! We will also be mentioning some paid content, but we are not receiving any fee or using referral links when we mention these resources.

Who are we

Just to put this in context, this article is written by Luciano and Stjepan, 2 software engineers with different backgrounds and expertise who came to love Rust. If you want to know more about us and understand why we are enjoying Rust, check out our bios at the end of this post.

Free material

Official guides

The Rust Programming Language - a.k.a. “The book”

The obvious place to start is the official Rust book! The Rust Programming Language is an open-book written by Steve Klabnik and Carol Nichols is a must-read for everyone starting their journey with Rust. It is a quite comprehensive document that explores all the main Rust concepts with very clear examples and a few interesting projects spanning multiple chapters: from a guessing game on the CLI to a multi-threaded web server. The best part of it is that it is a free resource, maintained by the Rust community itself. It has been kept up to date during the last few years, so it’s a resource you can always come back to if you want to review some of the topics you are struggling with.

Rust by example

Rust by example is another wonderful official resource. It is structured as a book, but it feels more like a cheat sheet. It is divided into small chapters and subchapters. Every section is something you can read in about 5 minutes to learn a new Rust concept. You can treat it as reference material and you can get back to it anytime you struggle to remember some specific details like how do you write a unit test or what’s the exact syntax of a match block. The reason why we love this resource is that it puts a lot of focus on the code. There is certainly more code than text and you will be learning by reading code and comment blocks embedded in the code; a great way to get used to the Rust syntax while you learn the concepts around the language.

The Rustonomicon

The Rustonomicon is a free book for those who are already pretty confident with Rust and want to learn the dark arts of unsafe Rust! Rust can have portions of code that are unsafe. What that means is that the compiler will let you do things that might be unsafe and it’s up to you to use this power responsibly. Unsafe code is not necessarily bad but needs to be used with caution and this book will guide you through all the nitty-gritty details that you need to be aware of before starting to write unsafe Rust. Read it if you want to avoid unleashing indescribable horrors!

Non-official guides

Rust Design Patterns

Rust Design Patterns is another open-book that is focused on teaching idiomatic Rust. It explores some Rust specific concepts but also revisits the classic behavioural, creational and structural design patterns and adapts them to the Rust idioms. Of course, it also takes the opportunity to explore idiomatic Rust patterns such as RAII and NewType. As you might have noticed if you know us, we are real fans of design patterns, so we couldn’t skip this book! Check it out after you have completed The Book and Rust by example.

Possible Rust

Possible Rust is a beautifully designed website that talks about “what’s actually possible with Rust”. The website is divided into 2 main sections: guides and patterns. Guides aim at explaining interesting Rust concepts like Foreign Function Interfaces (FFI), Traits, Enums, etc, while the Patterns section tries to explore interesting gotchas like “what can you do when you can’t make a trait object”. There isn’t a lot of material just yet, but we are sure that this resource is worth checking out and that it will grow over time. Keep it in your bookmarks or feed reader!

Easy Rust

Did we already say that Rust is not the easiest language to learn? Well, that’s actually the same premise from which Easy Rust is starting from. This is another open-book that tries to approach Rust concepts in a simple way, making them much more approachable and digestible, especially to people that are approaching Rust for the first time or engineers coming from higher-level languages. We especially loved some of the analogies in the book, for instance, pointers are compared to entries in a table of contents of a book. In our opinion, it is a great complement to the official Rust Book, but what makes it even more interesting is that recently the contents of this book were made available also in the form of videos available on YouTube!

Learning Rust

Learning Rust is another well-rounded open guide on Rust. With its beautiful website divided into a few major sections, it explores the basics of Rusts but also more advanced concepts such as ownership, borrowing, generics and traits. This resource focuses a lot on providing simple explanations and incentivises learning through reading Rust code with its rich set of code samples.

Rust Cookbook

Also called “Cookin’ with Rust”, Rust Cookbook defines itself as being “a collection of simple examples that demonstrate good practices to accomplish common programming tasks, using the crates of the Rust ecosystem”. As many of the resources already presented so far, Rust Cookbook is built in the form of an open-book (using mdbooks) so it is completely available as an open-source project. The focus of this book is to complement many of the resources we already described above. It does not try to explore the basics of the language but instead, it takes a practical stance against common everyday problems such as concurrency, compression, cryptography and data structures. For every one of these areas it provides a number of interesting examples and suggests some of the most common third-party libraries (crates) that are generally used to tackle these problems.

Learn Rust With Entirely Too Many Linked Lists

Linked lists are… fun! I mean sort of! Definitely, the kind of thing you have been bogged down implementing if you took a computer science degree. In fairness, there is nothing wrong with linked lists even though they are no one’s favourite for sure!

So why should we care about linked lists in Rust? Well, Learn Rust With Entirely Too Many Linked Lists might provide a good answer to this question.

Linked are built heavily around references so, if you grasp a little bit of Rust already, you might guess that by building a Linked List in Rust you will have to engage in a bloody fight with the borrow checker! This open-book illustrates a number of different approaches to tackle this challenge and, through a series of failed attempts, you will learn many interesting details about Rust and the borrow checker.

This resource is definitely recommended if you are starting to feel confident about your Rust knowledge and you want to start exploring more advanced topics in a practical way.

Blogs

Amos (fasterthanlime)

If you want to learn more about Rust, why it is a fantastic language, why and when it might not be so good, and how it is becoming even better, the best place to get into that is Amos (fasterthanlime)‘s blog.

Amos has a very engaging writing style. He can write very lengthy posts but they are always extremely enjoyable. Maybe because of his extensive in-depth knowledge, maybe because of his entertaining writing style and the support of the Cool bear!

If there one article you shouldn’t miss in this blog that certainly is Half an hour to learn Rust.

Yoshua Wuyts

Yoshua has been quite an influential developer in the JavaScript community and we had a chance to meet him at several conferences. Always inspiring, Yoshua has been transitioning more and more to Rust in the latest years and he has been documenting his journey in his great blog.

His blog is a very good place to learn about async Rust, streams, iterators, etc. Yoshua works on several open-source projects and gives a lot of insight into the pros and cons of async in general.

Given our background as web developers, our favourite article so far is Async HTTP.

Steve Klabnik

Steve Klabnik is one of the most influential people in the Rust community. No surprise that some of the most influential Rust posts come from his blog. In this blog, there’s a lot of content on software engineering in general, and on how other languages and their problems (and solutions) compare to Rust.

If you don’t know where to start, we recommend Rust has finally outgrown me.

Tyler Neely

If you’re into databases, performance optimization, distributed systems, check out Tyler Neely’s blog and look no further. The author writes about low-level problems, tips and tricks, linux, etc. Their project, Sled is a fantastic tool if you want to experiment with embedded databases and Rust.

Luca Palmieri

A learning journal by Luca Palmieri is another valuable resource to find Rust learning material. Luca is the author of the in-progress book Zero to Production in Rust (more on this later) and some of the chapters of the book have been published in the form of blog posts. Luca is a very prolific author and there is really a lot of good material, mostly around web development with Rust. One of our favourite articles is Choosing a Rust web framework, 2020 edition which compares the major Rust web frameworks and provides pointers on which one you should choose for your next web project.

Streams (youtube & twitch channels)

If you prefer a more visual way to learn or if you simply enjoy video material, here’s a lit of the Youtube channels and Twitch streams dedicated to Rust that we have been enjoying the most.

Streaming Rust with Ryan Levick

One of our favourite Rust YouTube channels is Streaming Rust with Ryan Levick. Ryan is a Microsoft engineer who has been spending a lot of time streaming Rust material. His content is quite varied, there are intro videos for beginners but also more advanced videos like static VS dynamic dispatch or FFI. What we like the most about Ryan is that he is great at explaining concepts, both simple and complicated ones. Every video is a little gem and the examples are always spot on.

If you prefer to watch the live versions of his videos you can also find Ryan on Twitch.

Jon Gjengset

Jon Gjengset is another brilliant author who is streaming about Rust. He is also another content creator that we admire for his capacity of making complex topics easy and understandable. He has 2 main series about Rust right now:

Crust of Rust: a very good series for the mid-level Rustucean who wants to get better with more advanced concepts such as iterators, smart pointers, lifetime annotations, etc.
The Unsafe Chronicles: A deep dive into the world of unsafe Rust.

Currently, The Unsafe Chronicles series has only one video, but the author recently posted on Twitter that we should be expecting a lot more of his time spent on creating new exciting content! How cool is that?

If you prefer to watch Jon live, you can find him on Twitch.

Tim McNamara

Tim is the author of the book Rust in Action (more on this later), but he is also a prolific video streamer. He has been doing an excellent introduction to Rust video series called Learn Rust Programming. We think that series is a pretty great place to start with Rust. One of the videos we loved the most is about error handling in Rust.

Also, special love goes to Tim for doing a live review of Luciano’s first crate (jwtinfo), which is a testament to how much Tim cares about educating people about the Rust programming language.

If you want to follow Tim (you should!) you can find him on YouTube and Twitch.

Genus-v programming

Genus-v programming is another great YouTube channel with some interesting Rust material. It is mostly focused on web development with Rust and that’s probably why it fell under our radar.

One of our favourite playlists (unsurprisingly) is Web Development with Rust which shows how to use Actix to build a GraphQL API and an authentication server.

Systems with JT

Systems with JT is the YouTube channel of Jonathan Turner. Jonathan has been a very active Rust developer. He is the author of Nushell and, of course, in his channel, he has a good number of videos dedicated to Rust. Right now, our favourite series is the one where Jonathan explains how to create a line editor in Rust.

Newsletters

The two main Newsletter we follow and recommend are Awesome Rust Newsletter and This Week in Rust. Both are excellent resources to keep yourself up to date with the latest news in the Rust world.

Another resource that is worth recommending, even though it is not strictly a newsletter, is Read Rust, which provides a collection of really high-quality articles about Rust. You can subscribe to the RSS feed if you want to receive instant notifications for new content!

Podcasts

As far as we are aware, there isn’t a hell of a lot of Rust podcasts (yet). The two we have been sporadically listening to are The Rustacean Station Podcast and New Rustacean.

Another one that is quite good, although not strictly about Rust, is The Bikeshed which occasionally contains episodes that explore Rust (see #133 and #134).

Communities & Meetups

If you like to interact more with the Rust communities there a few “places” we can certainly recommend!

The first one is the official Rust Discord community.

Then, there are a few meetup groups that we have been sporadically attending (all virtual at this time): Rust London, Rust Berlin and Rust Dublin.

Exercises and Workshops

A most recommended way to learn Rust is… practice, practice, practice! But it is possible that you might not have ideas on what to build right now… and, if you do, the specific idea might not provide the easiest of learning curves!

There are certainly a few resources that we can recommend if you are willing to write more and more Rust code to get better at Rust!

The first one that we should absolutely mention is Rustlings. This is an official resource that allows you to practice Rust concepts by giving you exercises and tests (that you can use to validate your solution). The exercises are related to the topics discussed in the official Rust book, so you can do them as you progress through the book.

Another interesting resource is a workshop by Luca Palmieri called Build your own Jira with Rust. In this workshop, you start with a semi-structured project and you have to fill in the blanks. Every file is a small Rust lesson and you’ll get to learn a new concept or two by providing a correct implementation.

Finally, when you start to feel a bit more comfortable with Rust, we do recommend you try to solve the excellent coding challenges proposed by Advent of Code in Rust. We are actually doing that and so far it has been a lot of fun!

Open-source projects

Another interesting thing you could do to solidify your understanding of Rust and learn to write more idiomatic code is to look at the code of famous open-source libraries.

You don’t have to read all the code line by line (some projects are massive!) but, just having random glances here and there in the code or simply looking at the code structure and the documentation can be very beneficial and you could learn a lot from it.

Here’s a list of libraries we do recommend you to take a look at:

Tokio, Async-std and Smol (async runtimes)
Serde (serialization library)
Clap (CLI helper library)
Rand (random value generation)
Chrono (date and time library)
Nannou (a creative coding framework)
Amethyst and Bevy (game development)
Sled (embedded database)
mdbooks (create a book from markdown)

Paid material

Books

If you like learning by reading books, this list should have you covered.

Rust in Action (Tim McNamara)

Rust in Action by Tim McNamara (Manning) is probably one of our favourite books. It has been in the works for a good number of years but it’s very very close to hitting the press. The book can be already purchased through the Manning MEAP program (you get the digital version and all the updates, then when the print edition is finally available, you’ll receive it by post).

The reason why we enjoyed this book so much is that it takes a very practical stance. Rust concepts are taught by building interesting projects such as your own floating-point number implementation, a CPU emulator, a custom file format, a key-value store database, etc.

Zero to Production in Rust (Luca Palmieri)

Zero to Production in Rust by Luca Palmieri (self-published) is a really good book about backend web development in Rust. We particularly like this book for various reasons.

First of all, it’s clearly a work of dedication and love. We had more than one opportunity to speak with Luca and it’s clear how much time and dedication he is putting into this project.

Since the book is a work in progress, you can read the new chapters as soon as they are available. It’s great to see how the community has been engaging around the project helping the author to make the best out of this book.

Luca is putting together a book that revolves around one main example: building an e-mail subscription platform. At first glance, one might think this is a very simple example, but in reality, there’s enough complexity around this domain to fill an entire book and, most importantly, learn Rust in the process!

In addition to this, we love the particular style that the author uses throughout the book. In every chapter, before any piece of code is written, there is an entire section about what the requirements are and why we really need to build a certain feature. Then, every piece of code has its own test and changes are actually introduced with a strict test-driven approach. This is quite unusual for a technical book, but that’s actually why we like this book. While reading it, it seems like you are having a conversation with your product manager or your team lead and you realise you could actually use Rust at work, no big deal!

We think there is a lot to learn from this book, not just about Rust but even about product development and good software engineering practices. Highly recommended!

Programming Rust

Programming Rust by Jim Blandy, Jason Orendorff and Leonora F.S. Tindall (O’Reilly) is one of the most popular Rust books out there. We had an opportunity to read the first edition and it’s a very solid book. The second edition should be coming this summer.

In the first edition, there were a few sections (especially in the first chapters) that were a bit hard to get, but we heard that the authors have been putting a lot of effort in listening to readers feedback and making sure the second edition will come out much more polished and approachable.

Given the success of the first edition, we are simply looking forward to get our hands on the second edition!

The Rust Programming Language (Steve Klabnik, Carol Nichols)

The Rust Programming Language is the print edition of the actual “official” Rust book that we discussed at the beginning of this post. There isn’t much else to add, except maybe that if you enjoy reading printed books (and want to support the authors), this is a great book to have!

Video Courses

To conclude this article, in this last section, we will explore some paid video courses that you can check if you prefer videos as learning material.

Rust in Motion

Rust in Motion by Carol Nichols and Jake Goulding (Manning) is a great video introduction to Rust. The material is geared towards beginners so it’s a great way to start learning Rust.

One of the things that we have been enjoying the most from this video course is the explanation of Rust lifetimes. We particularly liked how lifetimes are visualised and all the examples presented in the chapter make it easy to understand this fundamental concept of Rust.

Building web APIs with Rust

Building web APIs with Rust by Paris Liakos (Udemy) is a quite good beginner-level introduction to building web applications with Rust using Rocket for web framework and Diesel as the ORM.

In about 2 hours and a half, you could have a feeling for what it means to do web development with Rust.

Conclusion

This concludes our list of resources for learning Rust.

We really hope you found all the material here useful and we look forward to hearing how your journey to becoming a “rustacean” is going!

Let us know in the comment which material you are enjoying the most and if you come across some other interesting material that we didn’t mention in this post.

Until then… enjoy writing Rust code!

CIAO 👋

About the authors

Stjepan Golemac

Stjepan is a full-stack engineer with interests in machine learning, high-frequency trading, p2p, decentralization, and Rust. Working mostly with JavaScript and TypeScript in the past, he became a huge fan of Rust mostly because it enables humans to produce safe and usually correct code, but also because of its latency and performance guarantees. If you want to see what he’s been up to, or just want to say hi, you can check out his blog or send him a message on Twitter.

Luciano Mammino

Luciano is the owner and main author of this blog. He is a full-stack developer who, in the latest years, has been focusing more and more on scalable cloud architectures. He has been attracted by Rust because it is a great language to learn and appreciate interesting lower-level details of the programming world like memory management and thread-safety. If you want to find out more about Luciano you can check out the about section of this website or engage with him on Twitter.

How to send gzipped requests with boto3

Luciano Mammino — Tue, 22 Jun 2021 18:20:00 GMT

I recently needed to send a big payload to CloudWatch and I managed to increase my chances of staying within the AWS payload size limit by using gzip encoding on the request body with boto3.

Unfortunately, with boto3, gzip encoding is not enabled by default and it can’t be turned on with some simple boolean flag. To make things worse, as of today, there isn’t a great body of documentation or examples available on how to achieve this. Or maybe I am just terrible at googling…

I had to go down the rabbit hole to figure out how to support this use case and, in this article, I want to share what I learned with you.

Send gzipped metrics to CloudWatch using boto3

UPDATE (2024-02-26): AWS has recently announced automatic GZip support for PutMetricData requests in the CloudWatch API. This means that, for the specific case of the PutMetricData, the approach described in this article is not necessary anymore. You can find more details in the official announcement. This article might still be interesting if you want to learn more about the internals of boto3 and how to use its event system.

Ok, this is the TLDR; a little gift for the ones in a rush and who are looking for a quick copy-pastable solution.

Be aware that the various AWS SDKs are just a convenience layer in front of the AWS HTTP API. Python and boto3 are no exception. Every time you are calling a method on a boto3 client, you are just sending HTTP requests to AWS behind the scenes…

This is how you can intercept and modify such HTTP requests before they are sent to AWS:

boto3 has a built-in event system that, among other things, allows you to intercept (and even modify) HTTP requests that are ready to be forwarded to AWS.
By using this event system, you can implement an event handler that takes the payload of an outgoing request and gzips it. The same handler can also alter the set of outgoing HTTP headers so that it can indicate the request is gzipped by adding the header Content-Encoding: gzip.

Here is a code example that uses a CloudWatch client, intercepts calls to the PutMetricData API, and gzips the request payload:

import boto3
import gzip

# Create a CloudWatch client
cw_client = boto3.client('cloudwatch')

# Get a reference to the event system for that client
event_system = cw_client.meta.events

# Event handler that takes an arbitrary request and gzips its body
def gzip_request_body(request, **kwargs):
    gzipped_body = gzip.compress(request.body)
    request.headers.add_header('Content-Encoding', 'gzip')
    request.data = gzipped_body

# Register the event handler (only for CloudWatch PutMetricData!)
event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body)

# Sends a request
cw_client.put_metric_data(
    MetricData = [
        {
            'MetricName': 'KPIs',
            'Dimensions': [
                {
                    'Name': 'PURCHASES_SERVICE',
                    'Value': 'CoolService'
                },
                {
                    'Name': 'APP_VERSION',
                    'Value': '1.0'
                },
            ],
            'Unit': 'None',
            'Value': 217
        },
    ],
    Namespace='CoolApp'
)

You are welcome! 😜

Now, if you are curious to know more about my use case and how the boto3 events system works you are more than welcome to keep reading the rest of the article.

The use case: sending custom metrics to CloudWatch

Last week, during my work at fourTheorem, we started to get intermittent alarms for a Lambda in our stack that was failing because of requests to CloudWatch being occasionally throttled.

Once we looked at the error, the problem was actually quite obvious, but let me give you a simplified overview of what we are doing with this Lambda.

This Lambda is triggered by a Kinesis stream in which we publish custom metric data from all the other components of our application. The Lambda reads these metrics and publishes them to CloudWatch using the PutMetricData API.

Note that this idea is coming from Lumigo’s SAR async Lambda metrics.

The issue is that in our original implementation we took the naive approach of submitting 1 data point at a time. Therefore, under load, we would be sending a large number of HTTP requests per second to AWS and we might end up being throttled.

The solution to the problem is actually quite simple: we can reduce the total number of HTTP requests by sending the data in batches containing multiple data points, rather than sending data points one by one.

This is actually possible by using the same PutMetricData API from CloudWatch.

CloudWatch `PutMetricData` limits

So apparently the solution is simple: we just need to send multiple data points for every single PutMetricData call. But how many?

A call to this API accepts an array of metrics that looks like this:

[
    {
        'MetricName': 'SomeMetric1',
        'Dimensions': [
            {
                'Name': 'Dimension1Name',
                'Value': 'Dimension1Value'
            },
            # Up to other 9 dimensions here ...
        ],
        'Unit': 'Count',
        'Values': [217, 220, 221], # Up to 150 items here ...
        'Timestamp': 1624290910000
    },
    # Up to other 19 metric items here ...
]

More details can be found in the documentation for the MetricDatum type.

As you can tell from the comments in the snippet above, the API has some interesting limits to be considered:

Up to 20 different metrics
Up to 10 dimensions per metric
Up to 150 values per metric
Up to 40 KB in size for HTTP POST requests

Update (2022-08-04): AWS has recently incresed all these limits (by a lot), check out the new limits in their official announcement post.

So, the best case scenario to reduce the number of PutMetricData requests is to have batches of 20 metrics, but we need to make sure that all this data, once encoded, fits in 40 KB.

Splitting the metrics in chunks of 20 entries per request is not a big deal. But how do we make sure all the other constraints are respected. Especially the payload size one.

The PutMetricData documentation page adds an interesting bit of information:

You can send a payload compressed by gzip.

It would be convenient to use gzip compression to increase our chances to stay within boundaries.

At this point, I thought “Ok, probably boto3 is automatically doing the compression for us, because why shouldn’t it?”

But if you have used AWS for long enough, you learn not to give too many things for granted, so… Let’s test this assumption, first!

Testing `boto3` default behavior

So, how can we see if the by default boto3 is already gzipping our requests? Requests are going to AWS and I could think of 3 different ways to inspect how the final HTTP request actually looks when sent to AWS:

Use a debugger and step through the execution in the deep corners of the boto3 code.
Use an HTTP proxy like Burp Suite to intercept all the outgoing traffic and have a chance to inspect what’s being sent to AWS.
Use the endpoint_url parameter to configure the boto3 client to send requests somewhere else.

I opted for option 3, just because I thought it would be the quickest and easiest for me. I quickly wrote a very simple HTTP debugging server in Node.js and let it listen for requests at locahost:8000.

At this point I could simply run the following Python code to see how boto3 would send a PutMetricData request to AWS:

import boto3
import gzip

endpoint_url = "http://localhost:8000/"

cw_client = boto3.client('cloudwatch', endpoint_url=endpoint_url, use_ssl=False)

cw_client.put_metric_data(
    MetricData = [
        {
            'MetricName': 'TEST_BOTO',
            'Dimensions': [
                {
                    'Name': 'APP_VERSION',
                    'Value': '1.0'
                },
            ],
            'Unit': 'None',
            'Value': 17
        },
    ],
    Namespace='BotoTest'
)

Note that I am also using use_ssl=False because my local endpoint supports only plain HTTP.

Ok at this point I used all my confidence in AWS and run this script… This is what I got in my debug web server:

As you can see from the picture above, boto3 does not use gzip compression by default.

Sad me… 😢 Time to roll up the sleeves and figure out how to gzip the request body manually!

The `boto3` events system

After some research, I discovered that boto3 has a built-in event system.

This event system allows you to intercept requests being sent to AWS and alter them before they are sent. You can also intercept responses coming from AWS.

This is not a very well known feature of boto3, but I can see it being useful in a couple of circumstances:

Log all outgoing requests and incoming responses (for debugging)
Alter HTTP requests or HTTP responses to and from AWS to implement custom behaviors (gzipping the payload of outgoing requests is a great example).

If you want to look at a quick example of how to use this feature, here is one: we want to add an x-trace-id header when we invoke a lambda using boto3:

import boto3

lambda_client = boto3.client('lambda')

# our event handler
def add_xtrace_header(request, **kwargs):
    request.headers.add_header('x-trace-id', 'trace-trace')

# get the event system for the lambda_client
event_system = lambda_client.meta.events

# attach an event handler to the client for
event_system.register('before-sign.lambda.Invoke', add_xtrace_header)

# invoke a lambda function
lambda_client.invoke(FunctionName='my-function')

The example should be pretty straightforward, but let’s zoom on some interesting details.

An event handler is just a function that receives a number of arguments that depend on the type of event.
Every boto3 client exposes the event system as client.meta.events.
We can register an event handler by using the register function on the event system of a given client and pass the event name and the handler to it.

The event name is interesting because it follows this specification:

<event-type>.<service-name>.<operation-name>

event-type is used to indicate the type of event. This is generally representing a phase of the lifecycle of a request. Some examples are provide-client-params, request-created, before-sign, before-send, and response-received.
service-name is the name of the service used. For instance: s3 or cloudwatch.
operation-name indicates the type of operation the client is trying to perform (the method). For instance: PutMetricData or Invoke.

Not all these parts are mandatory and you can create event listeners for multiple events. Let’s look at some examples:

* will listen to every event
after-call.lambda.* (or after-call.lambda) will listen to all after-call type events for lambda and intercept all methods.

The botocore documentation isn’t very clear on how these events work, but after some digging, I managed to find the slides of a very interesting talk by Kyle Knapp called Deep Dive on AWS SDK for Python (Boto3). There’s also a video available! This one was a real lifesaver for me to understand how all of this works. Thank you, Kyle!

A `boto3` event handler for gzipping requests

At this point, we know enough to understand how we can add the gzip behavior to our CloudWatch PutMetricData requests.

Let’s update our test example and add a custom event listener:

import boto3
import gzip

endpoint_url = "http://localhost:8000/"

cw_client = boto3.client('cloudwatch',  endpoint_url=endpoint_url, use_ssl=False)
event_system = cw_client.meta.events

# Gzip handler
def gzip_request_body(request, **kwargs):
    gzipped_body = gzip.compress(request.body)
    request.headers.add_header('Content-Encoding', 'gzip')
    request.data = gzipped_body

# registers the custom handler
event_system.register('before-sign.cloudwatch.PutMetricData', gzip_request_body)

response = cw_client.put_metric_data(
    MetricData = [
        {
            'MetricName': 'TEST_BOTO',
            'Dimensions': [
                {
                    'Name': 'APP_VERSION',
                    'Value': '1.0'
                },
            ],
            'Unit': 'None',
            'Value': 17
        },
    ],
    Namespace='BotoTest'
)

print(response)

Note that we are using the before-sign event type because we want to alter the request before the signature is applied to the request.

If we run this code and check our debug web server we will see something like this:

Now the requests are compressed! We are sending exactly the same request as before and now it’s sending only 151 bytes as opposed to 265 bytes from the original request!

That’s some pretty good compression!

Of course, your mileage might vary depending on the shape of your custom metrics. Figuring out how much data can you actually pack in 40 KB might take a little bit of research. Keep in mind you are still limited to 20 metrics per request and 10 dimensions per metric. Gzipping is only increasing your chances to stay within boundaries if you really have to use a lot of dimensions (maybe with long names) or if you have a lot of values per metric.

Now, if we remove the configuration for endpoint_url and send the request to AWS, we can see that everything still works and that the compressed request is processed correctly:

Success!

A few ideas for more defensive solutions

Compressing the payload for PutMetricData requests is not an absolute guarantee that we won’t be exceeding the 40 KB payload limit. Gzip certainly helps, but you might still bump into the limit and have failures in your application.

If you want to be failure-proof ™️, here are some ideas that we explored with our colleagues:

Optimistic solution: Try the request and catch potential exceptions. If you see the PutMetricData failed because the payload is too large, just split all the data into 2 requests.
Pessimistic solution: pre-compute the gzipped payload before sending the request and if that’s bigger than 40 KB, then split the request into 2 parts.

The pessimistic solution has the disadvantage that you will be gzipping the payload twice (before you create the request to make sure it’s within boundaries and then in your event handler), but it will avoid sending invalid requests upfront.

Right now, we haven’t implemented any of these strategies. In our case, these metrics are not business-critical and we can afford some sparse failure, so we preferred to keep the code as simple as possible.

If, later on, we will observe production errors, then it might be worth applying one of these strategies. I am more in favor of the optimistic approach because I expect it will be pretty rare to bump into this limit.

What do you think, which strategy would you apply? Do you have other ideas? let me know that in the comments box below! 😁

Extensibility in the AWS SDK

UPDATE: It turns out that also with other languages like PHP, Go and JavaScript, the AWS SDK offers some degree of extensibility that should allow you to accomplish something similar to what was discussed in this post.

If you are using any of these other languages you can consult the following resources:

Thanks a lot to the following (amazing) people at AWS for helping me to find out this information in the huge haystack that is the AWS documentation: Danilo Poccia, Heitor Lessa, and Trivikram. You rock! 🤘

Conclusion

The event system in boto3 is pretty powerful and it allows you to customize the default behavior of how clients will send HTTP requests to AWS.

In this article, we saw how that can be useful for inspecting how AWS APIs are actually used or to add additional functionality that isn’t supported out of the box (e.g. gzipping the HTTP payload of outgoing requests).

If you found this article useful consider following me on Twitter and feel more than welcome to leave a comment below. I’d be really curious to find out what was your use case and if this article helped you out.

A huge “thank you” goes to my colleague Martin for involving me in this piece of work (and indirectly for dragging me into this rabbit hole 🐇)! Also thanks to Eoin Shanaghy for kindly reviewing this article! Finally, thanks to Ben Bridts for reporting an error in one of the examples! 🙏

See you soon! 👋

About fourTheorem

If you want to know more about fourTheorem, we are a team of business-focused technologists that deliver. We have been helping a dozen companies to get the best out of AWS and we had a lot of fun while doing that. If you are curious to know more about what we do you can check out some of our case studies and our customer success stories.

If you would like to work with us, don’t be shy, we’d love to get in touch with you! 🙂

Fastify and Preact for quick web app prototyping

Luciano Mammino — Mon, 18 Feb 2019 20:22:58 GMT

In this article I will show my setup to build and share web apps prototypes using Fastify and Preact, two technologies that I love.

I will also use htm, a library that can be easily integrated with Preact to define DOM elements in a very expressive and react-like way (like JSX), without having to use transpilers like Babel. Finally we will see how to dockerize a sample app built with this setup. This will make the app easy to share with co-workers and friends.

I came up with this stack, few weeks ago when I had to build a very simple prototype web application to test some product assumptions with some co-workers.

My experiment was quite successful. I was able to build my prototype quickly and it was super easy for my colleagues to play with it and give me feedback, even if they didn’t have Node.js and NPM installed on their machines.

For these reasons, I think this approach is probably worth sharing and I look forward to receiving your feedback in case you decide to give this tech stack a try.

TLDR;

If you know already Fastify and Preact and you are just looking for a quick way to get started with a stack based on these 2 technologies… well you just have to run the following commands:

git clone https://github.com/lmammino/fastify-preact-htm-boilerplate.git my-new-project
cd my-new-project
rm -rf .git
npm install

Of course, you can replace my-new-project with the actual name of your project.

Now enjoy editing the sample code in src:

src/ui: is for your frontend (Preact + htm)
src/server: is for your backend (Fastify)

Then run your project with:

npm start

And finally test it on localhost:3000.

Oh, yeah… most importantly, don’t forget to give me a ⭐️ on GitHub (pleazeee 😇)!

Fastify

If you never heard about Fastify, it is a fast and low overhead web framework for Node.js.

Fastify was initially created by Tomas Della Vedova and Matteo Collina. Today, it counts a core team of 10 developers, more than 130 contributors and almost 10.000 stars on GitHub.

Fastify takes inspiration from Node.js frameworks that have been around for a while like Express or Hapi, but puts particular focus on performance, developer experience and composability. One of my favorite features is in fact the composable plugin system.

The official Fastify documentation is a great place to start from, if you want to find out more about this wonderful framework.

Full disclosure: I am a member of the core team and I mostly contribute with building and maintaining the Fastify website and its documentation.

Preact

If you never heard about Preact, it is a UI library for the web, initially created as a lighter and faster drop-in alternative to React by Jason Miller.

The project has been quite successful. It is now maintained by a team of contributors and has gained more than 20.000 GitHub stars.

One of the reasons why I like Preact, is that it has a pluggable view definition layer. Normally you would use it with JSX in combination with Babel for transpilation, but if you don’t want to setup Babel and have a build process, you can use Preact in combination with htm, which uses template literals and doesn’t require any transpilation on modern browsers.

We are going to use htm in this article, so hold your horses… you’ll see some examples soon!

Project overview

So, you decided to follow along and build this stack from scratch, very well!

Our goal will be to build a simple web app that displays the server time at startup.

Just to give you a more concrete idea, this is more or less how it is going to look when completed:

This is a Single Page Application (SPA) where Preact and htm are used to build the fronted App, while Fastify is used to build the server time API.

By the way, the meticoulus reader might have noticed that amazing favicon there. Stop squinting, here’s a zoomed-in version, just for your personal enjoyment!

Backend setup

Ok, let’s start by creating a new folder:

mkdir server-time
cd server-time

Now we can initialize it as an NPM project and get Fastify installed.

npm init -y
npm i --save fastify@next fastify-static@next fastify-cli

Notice that I am using @next for some dependencies to get Fastify v2, currently in release candidate stage, but that should become the main stable version very soon!

Note: you can also create a new Fastify project using the Fastify CLI:
Terminal window
npx fastify-cli generate server-time
At the time of writing this will generate a new project for Fastify 1.x, but it will be updated soon, once v2 will be released as stable.

Let’s analyze the installed packages one by one:

fastify is the core component of the framework
fastify-static is an optional plugin that allows you to easily serve static files from a Fastify server
fastify-cli is a command line integration that allows you to start your Fastify apps.

At this point we are ready to create our Fastify API, let’s place the server code in src/server/server.js:

const path = require('path')

module.exports = async function (fastify, opts) {
  // serves static assets from the `src/ui` folder
  fastify.register(require('fastify-static'), {
    root: path.join(__dirname, '..', 'ui'),
  })

  // Add your API endpoints here
  fastify.get('/api/time', async (request, reply) => {
    return { time: new Date().toISOString() }
  })
}

The code above should be quite self explanatory, but there are some interesting details to cover, especially if you are not familiar with Fastify.

The first thing you might notice is the usage of the async keyword. Fastify supports both the async/await style and a more traditional callback-based style, so you can pick your favorite flavor.

Another interesting detail is that we are defining a server as an exported module. This module (a plugin in Fastify lingo) is essentially a function that receives a Fastify instance (fastify) and a set of options (opts) as arguments. Inside the module definition we can use the fastify instance to register plugins, as we are doing here with the fastify-static, or add HTTP endpoints using methods such as fastify.get or fastify.post.

This module approach, while a bit unconventional, has its perks. First of all, it allows you to compose different servers together. Imagine you have created a server to manage a blog and one to manage a forum, you could easily embed them in an existing app and mount them over paths like /blog and /forum.

Moreover, this approach keeps your apps and sub-apps abstract from the actual server bootstrapping (socket binding and listening), which is left either to a root level app or to the fastify-cli.

Let’s see how to do that with the fastify Command Line Interface:

node_modules/.bin/fastify start --log-level info src/server/server.js

For simplicity we can add this command to our package.json scripts:

{
  "scripts": {
    "start": "fastify start --log-level info src/server/server.js"
  }
}

Before launching the server, we have to make sure our UI assets folder exists (fastify-static would crash otherwise), so let’s create it:

mkdir src/ui

Now we can run our app with npm start and point our browser to localhost:3000/api/time.

You should now see a response like this:

{ "time": "2019-02-17T19:32:03.354Z" }

At this point you are probably noticing another amazing feature of Fastify: JSON serialization is handled out of the box if a route returns an object.

Hooray, our server API is now implemented. 🥳

Let’s move on and let’s start to work on the frontend!

Frontend setup

All our frontend code will live in src/ui and it will be made of 5 files:

app.js: the code for our Preact app
bootstrap.min.css: the CSS code for styling our app (directly from the Bootstrap framework)
favicon.ico: our delicious favicon, because you are not building a serious app if you don’t have a serious favicon!
index.html: main HTML code for our SPA
preacthtm.js: the code for Preact + htm.

First of all let’s download the files for Bootstrap, Preact and our favicon:

curl "https://unpkg.com/htm@2.0.0/preact/standalone.js" > src/ui/preacthtm.js
curl "https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" > src/ui/bootstrap.min.css
curl "https://github.com/lmammino/fastify-preact-htm-boilerplate/blob/master/src/ui/favicon.ico?raw=true" > src/ui/favicon.ico

Now it’s time to create our src/ui/index.html:

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta
      name="viewport"
      content="width=device-width, initial-scale=1, shrink-to-fit=no"
    />

    <!-- Bootstrap CSS -->
    <link rel="stylesheet" href="/bootstrap.min.css" />

    <title>My awesome server time</title>
  </head>
  <body>
    <div id="app"></div>

    <!-- JavaScript -->
    <script src="/preacthtm.js"></script>
    <script src="/app.js"></script>
  </body>
</html>

This is a pretty standard HTML 5 page where we are loading all our resources (CSS and JS) and creating an empty div (id="app") where we will mount our frontend application at runtime.

Let’s now look at the code needed for our app in src/ui/app.js:

/* global htmPreact */
const { html, Component, render } = htmPreact

class App extends Component {
  componentDidMount() {
    this.setState({ loading: true, time: null })
    fetch('/api/time')
      .then((response) => response.json())
      .then((data) => this.setState({ loading: false, time: data.time }))
  }
  render(props, state) {
    return html`
      <div class="container mt-5">
        <div class="row justify-content-center">
          <div class="col">
            <h1>Hello from your new App</h1>
            <div>
              ${state.loading && html` <p>😴 Loading time from server...</p> `} ${state.time &&
              html` <p>⏱ Time from server: <i>${state.time}</i></p> `}
            </div>
            <hr />
            <div>
              👩‍💻 Have fun changing the code from this boilerplate:
              <ul>
                <li>UI code available at <code>/src/ui</code></li>
                <li>Server-side code available at <code>/src/server</code></li>
              </ul>
            </div>
          </div>
        </div>
      </div>
    `
  }
}

render(html` <${App} /> `, document.getElementById('app'))

In our frontend app we have only one stateful component called App.

The state for this component is defined by 2 variables:

loading: a boolean flag used to indicate whether an API request to fetch the current server time is in progress
time: a string that contains the last fetched time

If you have familiarity with React, the component should be pretty straightforward to understand.

By using Preact and htm, we can create a component by writing a class that extends from the built in Component class.

In this class we can define the component behavior using lifecycle hooks like componentDidMount and the look and feel using the render method.

In our case, once the component is attached to the the page (componentDidMount hook), we set the state as loading and we issue a request to our time API, using fetch.

Once the request is completed, we set the time and reset the loading state to false.

The render function is automatically invoked every time the component state or its props change. In this method we define the DOM for the component using htm.

htm allows us to define the DOM nodes using tagged template literals with the special tag html. Within our template literal, we can have dynamic expressions, like the ones we use to check the state and decide what to render in case of loading or not.

One last detail, aside from defining the behavior and the look and feel of our App component we have to create an instance and render it in our HTML page.

We do that by using the render function from the global htmPreact object.

That’s it.

Relaunch your server and visit localhost:3000 to see the app in action!

Feel free to play around, change and break things, until you are happy enough with the outcome and feel ready to share your creation with the world (or maybe with just some friends).

Dockerize all the things

In my opinion, the best way to share your new little project is to use Docker.

With Docker, whoever is trying to run your app doesn’t have to worry about having the right versions of Node.js and NPM and to run the right sequence of commands to install dependencies and run the server.

In order to “dockerize” our app we have to create a very simple Dockerfile in the root folder of our project:

FROM node:11-alpine

WORKDIR /app
COPY . /app
RUN npm install --production

EXPOSE 3000

CMD ["npm", "start"]

With this setup we are essentially doing the following:

Creating an image starting from the the latest version of Node.js 11, linux alpine.
Everything from our current folder is copied inside /app inside the container.
At this point we run npm install to make sure that all the dependencies are downloaded and installed. The --production flag will make so that only production dependencies are installed, speeding up the image creation in case you have many dev dependencies.
We also expose port 3000, which is where our web server will run by default.
Finally, we define our runtime command as npm start to start the application.

In order to build the image for this container, you can run the following command:

docker build -t server-time .

After few second the image should be ready and you should be able to run containers off of it:

docker run -it -p 3000:3000 server-time

The -p parameter allows you to map the port 3000 from within the container to the local port 3000, so that you can access the dockerized application from localhost:3000.

Now you are ready to share this application. Whoever is receiving it has to have Docker installed and run the two commands above!

Conclusion

In this article, I showed you how easy it is to bootstrap a quick web app development environment using Fastify and Preact. We also saw how to share the resulting app with Docker.

I said that this setup is ideal for building quick prototypes, so you are maybe wondering what’s missing in case you want to take the app to production.

Well, these are probably some of the concerns you will have when starting to think about making your code production ready:

Frontend assets compilation: how to create optimized (bundled) files, maybe by using Webpack, Babel or other similar tools.
Frontend routing
Server side rendering
Data persistence

I didn’t cover all these aspects with my setup yet, so I don’t want to consider this production ready, but I am quite sure you can expand it and cover the missing pieces and be able to build production ready apps with Fastify and Preact.

I hope this article was informative and that this stack will be useful to you for your next prototype application.

This article was possible only because of the support and the kind reviews of some developers I really admire, so thank you to them all (in alphabetical order) 😻:

I really look forward to hearing your feedback, so please don’t hesitate leaving a comment below!

Until next time, ciao 👋

A random emoji in your terminal prompt. How and Why!

Luciano Mammino — Mon, 17 Dec 2018 18:50:00 GMT

In this article, we will see how to customize our terminal prompt and add a random emoji in it.

This is my terminal prompt:

If I could have a penny every time somebody asked me how did I put a random emoji in there, well… I wouldn’t probably be a millionaire, but I would definitely be able to afford few more coffees every day and be less sleepy 😴!

In this article I’ll explain how I emojified my terminal and, since it is very simple stuff, I’ll also try to explore some extra details on how to customize your bash experience.

Bash config

Bash customizations are generally made in a file saved in the user home directory. This file is generally called .bashrc ~~(more common on Linux systems)~~ or .bash_profile ~~(more common on MacOS)~~.

UPDATE: .bash_profile is executed for login shells, and .bashrc is executed for non-login shells. (Thanks to @colonelpanic for the correction here).

This file is essentially a Bash script file that gets loaded every single time you open a terminal session.

Here’s an example of a possible .bashrc file:

# Source global definitions
if [ -f /etc/bashrc ]; then
  . /etc/bashrc
fi

# various command shortcuts
alias og='ls -ogrt'
alias ll='ls -al'
alias lc='ls -C'
alias dir='ls -lrt' # in case you are a MS-DOS lover :P
alias h='history'
alias p='pwd -P'  # shows the "real" path in bash, not the path via symlinks
alias et='emacs-tty'
alias python=python2.7

# environment
export PS1='[\u@\h \W]$ ' # Set the command prompt!
export HOST=`uname -n`
export EDITOR=vim
export PATH=$HOME/bin/:$PATH

As you can see, this script has the only goal of setting up your bash environment. For instance, you can load other configuration files (here we load a global configuration if it is available). You can define shortcuts and aliases for common commands. Finally, you can export environment variables, for instance to define your default text editor or to modify/extend your PATH (a list of directories where to look for executable commands).

The PS1 variable

We briefly touched on PATH and EDITOR. There are many other special variables in Bash that can be used to customize the look and feel and the behavior of your terminal.

One of those variables (the most important in this article) is called PS1. If I were a betting man, I would bet that the name stands for ~~Prompt Style 1~~ (Update 2020-04-18: Rohit Chakraborty actually clarifies in the comments that PS1 stands for Prompt String 1!). PS1 allows you to define the look and feel of the primary bash prompt format.

In the configuration example above we are defining the PS1 variable as follows:

export PS1='[\u@\h \W]$ '

Which, in my personal laptop, renders like this:

At this point, you are probably already grasping how this works. The PS1 variable acts as a template, where special escape sequences like \u and \h can be used to indicate “replace this with my actual username (and hostname)”.

There are several sets of escape sequences that might be supported on different operative systems and different types of console, but here’s a list of the most common and widely supported escape sequences:

\a: an ASCII bell character (07)
\d the date in “Weekday Month Date” format (e.g., “Tue May 26”)
\h the hostname up to the first `.’
\H the full hostname
\n newline
\s the name of the shell, the basename of $0 (the portion following the final slash)
\t the current time in 24-hour HH:MM:SS format
\T the current time in 12-hour HH:MM:SS format
\@ the current time in 12-hour am/pm format
\u the username of the current user
\w the current working directory
\$ if the effective UID is 0, a #, otherwise a $ (used to distinguish whether you are root or not)
\\ a backslash
\[ begin a sequence of non-printing characters, which could be used to embed a terminal con trol sequence into the prompt
\] end a sequence of non-printing characters

For more comprehensive lists checkout:

👌 Pro tip: There’s also a PS2 variable, which is the secondary prompt displayed when a command needs more input (e.g. a multi-line command). This is generally set as PS2='> '.
Also PS3 and PS4 exist. Funny enough this is not a pun to PlayStation players… If you are curious to know what they are used for, Archlinux has a great section on terminal prompts.

The RANDOM variable

For our purpose, another useful Bash special variable is RANDOM. Its goal is very simple, as the name suggests, it always evaluate to a different random value.

Try to run the following command multiple times to have a feeling for it:

echo $RANDOM

I just ran this command 5 times and this is the list of outputs I got:

31434
25704
32466
10374
29063

To be more accurate, when using $RANDOM, we are not referencing a constant value or a variable, but an internal Bash function that returns a pseudorandom integer in the range [0-32767] (both values included).

This primitive is generally good enough when you need to do something random, like extracting an arbitrary element from an array.

Be careful though because this technique is not strong enough for security sensitive situations like generating an encryption key.

There are slightly more advanced use cases, for instance you might need to extract a random number over a smaller positive range. In such case you can use the modulo operator and do something like this:

SMALLER_RANDOM=expr $RANDOM % 22
echo $SMALLER_RANDOM

This will print a number between 0 (included) and 22 (excluded).

For other more advanced use cases check out the dedicated $RANDOM section in the Advanced Bash-Scripting Guide.

Bash arrays

In Bash, an array is a variable containing multiple values. Any variable may be used as an array and there is no maximum limit to the size of an array.

You can define an array in several different ways. You can assign the elements of an array one by one over different assignment statements:

A[0]=1
A[1]=2
A[2]=3
# ...

In a more generic fashion, you can assign a value in an given array index with the syntax ARRAY[INDEX]=value, where INDEX is an expression that is evaluated to a non-negative integer.

Of course there is a shorter version that allows you to initialize an array with multiple elements with a single assignment instructions:

A=(1 2 3)

When you want to read (dereference) a value out of an array you can do it this way:

echo ${A[2]} # 3

This will print 3 as Bash arrays are zero-indexed.

If you want to reference all the elements in the array you can do it as follows:

echo ${A[*]} # 1 2 3

echo ${A[@]} # 1 2 3

The elements in an array can be also strings and you don’t need to wrap them in quotes (unless a string includes a space in it). For instance you can create the following array:

SPORTS=(Judo Jiu-jitsu 'Sicilian stick fighting')

To loop over the elements of an array:

for SPORT in "${SPORTS[@]}"
do
  echo $SPORT
done

# prints:
#
# Judo
# Jiu-jitsu
# Sicilian stick fighting

Finally, to get the length of an array you can use the following syntax:

echo ${#SPORTS[@]} # 3 (notice the "#" before the variable name)

There is a lot more you can do with arrays. This is enough for what we want to achieve in this article, but if you are curious you can checkout the array section in the Bash Beginners Guide.

Putting it all together

Ok, now that we know about Bash config, Bash PS1 variable, Bash RANDOM and Bash arrays, we have enough knowledge to be able to render a random emoji in our terminal prompt.

Let’s try to write a simple .bashrc file that can do that:

# .bashrc (or .bash_profile)

# declares an array with the emojis we want to support
EMOJIS=(😺 😸 😹 😻 😼 😽 🙀 😿 😾)

# selects a random element from the EMOJIS set
SELECTED_EMOJI=${EMOJIS[$RANDOM % ${#EMOJIS[@]}]};

# declare the terminal prompt format
export PS1='${SELECTED_EMOJI}  [\u@\h \W]$ '

This will look as follows in my machine:

Well… as you can see there is something wrong with this… The emoji is always the same!

The problem here is that SELECTED_EMOJI gets computed only once when we open a new terminal window. At that point it behaves like a constant values, so our prompt will always look the same after that.

With this approach we essentially end up with a random emoji per session, not per command!

We can fix this by introducing a Bash function:

# .bashrc (or .bash_profile)

# declares an array with the emojis we want to support
EMOJIS=(😺 😸 😹 😻 😼 😽 🙀 😿 😾)

# function that selects and return a random element from the EMOJIS set
RANDOM_EMOJI() {
  SELECTED_EMOJI=${EMOJIS[$RANDOM % ${#EMOJIS[@]}]};
  echo $SELECTED_EMOJI;
}

# declare the terminal prompt format
export PS1='$(RANDOM_EMOJI)  [\u@\h \W]$ '

Notice that we replaced ${SELECTED_EMOJI} with $(RANDOM_EMOJI) in our terminal prompt template. Previously we were just interpolating a variable in our terminal prompt, now we are actually invoking a function that extracts a random emoji every single time the prompt is rendered.

This is the new result:

Mission complete!

But wait, not so quick! Let’s go over all the code once again to make sure we understand what’s going on behind the scenes.

EMOJIS is just a simple array of strings, where every string is made up by just an emoji.

SELECTED_EMOJI inside our function RANDOM_EMOJI, is a dynamic value that gets initialized to a random emoji from the set of supported emojis (EMOJIS). This is the complicated part as it’s a one-liner made up by different expressions on the right-hand side of the assignment. Let’s break it down:

${#EMOJIS[@]} will be evaluated to the length of the EMOJIS array
$RANDOM % ${#EMOJIS[@]} will calculate a random integer and normalize it to the number of elements in the EMOJIS array using the modulo operator, so this is essentially extracting a random index for the EMOJIS array.
Finally we use the result of this expression to dereference a random element from the array and we assign it to SELECTED_EMOJI.

Pretty neat, right?

Bonus: bash formatting

You probably noticed that my prompt (as well as most people prompt) comes with some fancy colors and other formatting wonders like bold and underlined text.

Most terminals support different formatting options: bold, dim, underlined, blink, hidden, inverted and a number of different colors (for both background and foreground!).

This is done by prefixing the text with some special escape sequences.

To give you an example, let’s have a look at this code:

echo -e "My name is \033[32mLuciano\033[39m and my twitter handle is \033[1m@loige\033[21m"

This will render as follows:

The way this works is by using a special escape sequence, the \033[ prefix sequence. This sequence can be parameterized to indicate the start or the end of different formatting options. The generic syntax is \033[_FormatCode_m, where _FormatCode_ will be replaced with some value to indicate different formatting options.

In my previous example:

\033[32m stands for “start color green”
\033[39m stands for “start default color (color reset)”
\033[1m stands for “start emphasis (or bold)”
\033[21m stands for “reset emphasis (or end bold)”

Here is a table with different _FormatCode_ values and their effect.

`_FormatCode_`	Description
1	Bold
2	Dim
4	Underline
5	Blink
8	Hidden
0	Reset all
21	Reset bold
22	Reset dim
24	Reset underline
25	Reset blink
28	Reset hidden
39	Default color
30	Black
31	Red
32	Green
33	Yellow
34	Blue
35	Magenta
36	Cyan
97	White
49	Default bg color
40	Black bg
41	Red bg
42	Green bg
43	Yellow bg
44	Blue bg
45	Magenta bg
46	Cyan bg
107	White bg

For more information about additional formatting options checkout Bash tips: Colors and formatting.

Other ways to achieve similar formatting are tput and terminfo.

Bonus 2: my actual PS1!

I keep my .bash_profile public on GitHub (not because it’s cool, but because this way I can easily clone it into different machines).

If you are curious you can see how my bash-emojification is actually done or what else I put in my PS1 variable or even some aliases I use.

Feel free to tell me what you think or ask questions about it in the comment box in this article.

Bonus 3: A bash prompt generator

Of course there had to be a service online to help you with this stuff! It turns out there are many! If you want to build your fancy Bash prompt in a drag-and-dropppy or clicky-clicky way, checkout these links:

At this point, you should know enough to be able to build your own PS1 generator website! Maybe that’s an idea for you next hack-weekend!

Bonus 4: Use the return code

This is an amazing snippet from @bdesham on Lobste.rs.

function success_indicator() {
    if [ $? -eq 0 ] ; then
        echo "😎"
    else
        echo "💩"
    fi
}

export PS1='$(success_indicator) $ '

With this approach you won’t have a random emoji anymore but instead a very deterministic emoji given by the return code of the previous command. If the command exited successfully you will see a cool dude emoji, otherwise a surprised poop will show up on your prompt!

Why all of this?!

In the title of this article I said “how and why”, but if you have been careful you probably noticed I never really explained why you might want to go through all of this stuff!

Well, the real reason why you should care about this is because “why not?!”

Seriously, there isn’t a very strong reason why this stuff is important, it is definitely fun and interesting but you might be a fantastic developer even without having all this knowledge (and without emojis in your terminal)!

Anyway, if this answer is not convincing you, I can actually prove that emojis in your terminal will make you more productive. Try to put a toilet (🚽) or a poop (💩) emoji in your list of supported emojis. Next time you’ll mistype a command and you’ll randomly see a poop emoji, I can assure you that you’ll have a bit of fun, your morale will stay up and you’ll hopefully keep pounding your way on the keyboard enjoying your work 😎.

Whether you believe me or not on this, I hope I’ll see you soon in the next article 😜

Byez!

PS: A special thanks to all the people who contributed to this Lobste.rs thread. A lot of great insights! Also, thanks to @gschizas on Reddit for spotting some errors! 🙏

JavaScript iterator patterns

Luciano Mammino — Mon, 21 Jan 2019 03:25:58 GMT

In this article we will explore different ways to create iterators and iterable values in Javascript, specifically functions, iterators, iterables and generators.

JavaScript is a very flexible language and most often you can achieve the same goals in many different ways, iterators are no exception!

Wikipedia defines iterators as follows:

In computer programming, an iterator is an object that enables a programmer to traverse a container, particularly lists. Various types of iterators are often provided via a container’s interface.

We will extend this definition even further as we will not focus on building iterators for pre-computed values like lists, but we will see how to iterate over generative sequences like the Fibonacci sequence.

Most likely you won’t be using the Fibonacci sequence in your day to day programming (unless you are interviewing for some company who wants to validate your knowledge of recursion 😆), but the idea of generating a sequence of values on demand (lazy evaluation) translates well to a lot of real-life scenarios like:

traversing custom data structures
consuming paginated APIs
draining a queue
processing long files line by line
read all the records from a SQL table
etc.

The Fibonacci sequence

In case you have never seen the Fibonacci sequence before (or you don’t remember the exact definition), here is how it looks like:

1 1 2 3 5 8 13 21 ...

Essentially, a number in the sequence is given by the sum of the previous 2 numbers.

In more formal mathematical terms, you can define the sequence as:

F₁ = F₂ = 1
F_n = F_(n-1) + F_(n-2)

Few things to notice:

The sequence is infinite (it would be impossible to store it in a list without an upper limit).
It is made by positive integers.

So, how do we write some JavaScript code that allows us to iterate over this sequence and calculate an arbitrary number of elements?

Well, there are many ways…

Functions

In JavaScript, functions are first class citizens and most patterns can be modeled with the use of only plain functions. This will become natural once you master the concepts of function scope, anonymous functions and nested functions.

So, how can we build a fibonacci sequence by using only functions?

Here’s an example:

const genFib = (max = Number.MAX_SAFE_INTEGER) => {
  // initialize default values in the scope
  let n1 = 0
  let n2 = 0

  // returns an anonymous function that will return the next element
  // every time that it is called
  return () => {
    // calculates the next value
    const nextVal = n2 === 0 ? 1 : n1 + n2

    // redefines n1 and n2 to match new values
    const prevVal = n2
    n2 = nextVal
    n1 = prevVal

    // if we reached the upper bound return null (iteration completed)
    if (nextVal >= max) {
      return null
    }

    // return the new value
    return nextVal
  }
}

I added some comments to make the code easy to understand, but let’s go through it once more.

genFib is a function that accepts an optional parameter, which is the upper bound used to define when to stop computing elements in the sequence. JavaScript numbers starts to lose precision after Number.MAX_SAFE_INTEGER, so this is a sensible default.
The first thing that happens in the function is initializing the function scope. n1 and n2 are the only two values that we need to compute an element of the sequence. They represent the last 2 numbers computed. We set them to 0 by default.
At this point the function returns an anonymous function. This function can be invoked an arbitrary number of times and every time it will compute and return a new element in the sequence, making sure the internal state is updated accordingly.

Notice that genFib will initizalize a new isolated scope containing n1 and n2. These values will be accessible (and modifiable) only by the anonymous function returned by genFib. This means that you can generate multiple “iterators” and everyone of them will be independent from each other.

To understand this even better let’s see an example on how a user would use this code:

const f = genFib(6) // limit the sequence to numbers below 6
f() // 1
f() // 1
f() // 2
f() // 3
f() // 5
f() // null
f() // null
f() // null

// or with a loop

// prints all the numbers of the sequence below MAX_SAFE_INTEGER
const f2 = genFib()
let current
while ((current = f2()) !== null) {
  console.log(current)
}

The Iterator protocol

In the previous example we came up with our own way to define how to iterate through the elements (returned anonymous function) and how to understand whether the sequence was over (return of null).

ECMAScript 2015 provides a standard and interoperable way to define iterator objects. This is called the Iterator protocol.

In short, a JavaScript object is an iterator if it implements a next() method with the following semantic:

next() does not accept any argument.
next() has to return an object with 2 properties: done and value.
done is a boolean and it will be set to true if and only if there are no more elements in the sequence.
value will contain the actual value as computed in the last iteration (could be undefined when done is true).

Ok, now let’s rewrite our Fibonacci sequence to implement the Iterator protocol:

const genFibIterator = (max = Number.MAX_SAFE_INTEGER) => {
  let n1 = 0
  let n2 = 0

  // this time we return an iterator object (rather than a function)
  return {
    // the logic needed to compute the next element is inside the `next` method
    next: () => {
      // calculates the next value
      let nextVal = n2 === 0 ? 1 : n1 + n2

      // redefines n1 and n2 to match new values
      const prevVal = n2
      n2 = nextVal
      n1 = prevVal

      // if we reached the upper bound (iteration completed)
      // set done to true and nextVal to undefined
      let done = false
      if (nextVal >= max) {
        nextVal = undefined
        done = true
      }

      // return the iteration object as for the iteration protocol
      return { value: nextVal, done }
    }
  }
}

The comments in the code should help you to understand the new logic.

Let’s see how to use our new Fibonacci iterator implementation:

const it = genFibIterator(6) // { next: [Function: next] }
it.next() // { value: 1, done: false }
it.next() // { value: 1, done: false }
it.next() // { value: 2, done: false }
it.next() // { value: 3, done: false }
it.next() // { value: 5, done: false }
it.next() // { done: true }

// or

const it2 = genFibIterator(6)
let result = it2.next()
while (!result.done) {
  console.log(result.value)
  result = it2.next()
}
// 1
// 1
// 2
// 3
// 5

The Iterable protocol

In the previous section we saw how to define Iterator objects that conform the Iterator protocol. In reality, we might want to express the concept of “iterability” in a more generic fashion, so that, given any object, we can tell if such object is iterable or not.

For this reason, ECMAScript 2015 defines also the Iterable protocol.

An object is said to be iterable if it exposes a property called Symbol.iterator, which is a function that returns an iterator object.

You can introspectively check if an object is iterable with some code like this:

function isIterable(obj) {
  return Boolean(obj) && typeof obj[Symbol.iterator] === 'function'
}

ECMAScript 2015 also provides a new for construct (for...of) that allows to easily iterate over the elements of an iterable object:

for (let current of someIterable) {
  console.log(current)
}

Iterable objects can also be used in combination with the spread operator to eagerly load all the values and store them into an array:

const allValues = [...someIterable]

Ok, now let’s rewrite our Fibonacci sequence to implement the Iterable protocol:

const genFibIterable = (max = Number.MAX_SAFE_INTEGER) => {
  let n1 = 0
  let n2 = 0

  // returns an iterable object
  return {
    [Symbol.iterator] () {
      // returns an iterator
      return {
        next() {
          let nextVal = n2 === 0 ? 1 : n1 + n2

          const prevVal = n2
          n2 = nextVal
          n1 = prevVal

          let done = false
          if (nextVal >= max) {
            nextVal = undefined
            done = true
          }

          return { value: nextVal, done }
        }
      }
    }
  }
}

What we did here is to just move the implementation of the iterator seen in the previous section into the Symbol.iterator function.

Note that it is possible to come up with an implementation that can satisfy the Iterator and the Iterable protocols at the same time :

const genFib = (max = Number.MAX_SAFE_INTEGER) => {
  let n1 = 0
  let n2 = 0

  return {
    // this satisfies the Iterator protocol
    next: () => {
      let nextVal = n2 === 0 ? 1 : n1 + n2

      const prevVal = n2
      n2 = nextVal
      n1 = prevVal

      let done = false
      if (nextVal >= max) {
        nextVal = undefined
        done = true
      }

      return { value: nextVal, done }
    },

    // this satisfies the Iterable protocol
    [Symbol.iterator] () {
      // returns `this` because the object itself is an iterator
      return this
    }
  }
}

The comments in the code should help you to understand the logic in these 2 implementations.

With this new approaches you can generate numbers from the Fibonacci sequence as follows:

// prints all the numbers in the sequence until MAX_SAFE_INTEGER
const f = genFibIterable()
for (let n of f) {
  console.log(n)
}

// creates an array with all the Fibonacci numbers lower than 17
const f2 = genFibIterable(17)
const lowerThan17 = [...f2] // [ 1, 1, 2, 3, 5, 8, 13 ]

If at this point you are still struggling to see the logical difference between an iterator and an iterable object you can see it this way:

An iterable is an object on which you can iterate over.
An iterator is a cursor object that allows you to iterate over an iterable.

Generators

Another great addition coming from ECMAScript 2015 to JavaScript are Generators. More specifically, ECMAScript 2015 defines Generator functions and Generator objects.

A function* declaration (function keyword followed by an asterisk) defines a Generator function, which returns a Generator object.

Generators are functions which can be exited and later re-entered. Their context (variable bindings) will be saved across re-entrances.

To simplify this concept a bit, you can see generator functions as functions that can “return” (or “yield”) multiple times.

Let’s explore the generator syntax with a simple example:

// generator function
function* countTo3() {
  yield 1
  yield 2
  return 3
}

In this example we are defining a counter that generates numbers from 1 to 3. We can use it as follows:

// c is a generator object
const c = countTo3()
c.next() // { value: 1, done: false }
c.next() // { value: 2, done: false }
c.next() // { value: 3, done: true }
c.next() // { done: true }
c.next() // { done: true }
// ...

So, the way a generator works is the following:

When you invoke a generator function, a generator object is returned.
Generator objects have a next() method.
When you invoke the next() method of a generator object the code of the generator will be executed until the first yield (or return) is encountered.
If a yield was found, the code is stopped and the yielded value will be passed to the invoking context though an object with the following shape: { value: <yieldedValue>, done: false }.
The next time next() is invoked, the execution will be resumed from the point where it was initially suspended until a new yield or return is found.
If a return statement is found (or the function completes), the object returned will look like: { value: <returnedValue>, done: true } (notice the done now set to true).
Once the generator has completed, consecutive calls to next() will always produce { done: true }.

Of course, the reason why we are exploring this topic is because we can implement our Fibonacci sequence as a generator:

function* Fib (max = Number.MAX_SAFE_INTEGER) {
  // initialize the state variables
  let n1 = 0
  let n2 = 0
  // we can now pre-initialize nextVal to 1 as part of the state
  let nextVal = 1

  // loop until we exceed the max number
  while (nextVal <= max) {
    // yields the current value
    yield nextVal

    // shifts nextVal -> n2 and n2 -> n1
    const prevVal = n2
    n2 = nextVal
    n1 = prevVal

    // calculates the next value
    nextVal = n1 + n2
  }
}

The comments in the code should help you with understanding this implementation.

You can immediately notice that since we don’t have to deal with a nested function, the implementation seems easier to read, or at least it might feel easier to read the code and understand the actual execution flow.

For this reason, you might prefer to use generators over plain functions in this kind of scenarios.

We can use our new generator-based Fibonacci sequence as in this example:

const fib = Fib(6)
fib.next() // { value: 1, done: false }
fib.next() // { value: 1, done: false }
fib.next() // { value: 2, done: false }
fib.next() // { value: 3, done: false }
fib.next() // { value: 5, done: false }
fib.next() // { done: true }

// or

const fib2 = Fib(6)
let result = fib2.next()
while (!result.done) {
  console.log(result.value)
  result = fib2.next()
}
// 1
// 1
// 2
// 3
// 5

At this point you might be wandering:

Is a generator object an iterator or an iterable?

Well, it turns out that a generator object is both an iterator and iterable!

So you can also use our latest implementation with the for...of and the spread syntax:

const fib = Fib(6)
for (let current of fib) {
  console.log(current)
}
// 1
// 1
// 2
// 3
// 5

// or
const fib2 = Fib(6)
[...fib2] // [ 1 1 2 3 5 ]

Finally, since generators are iterators, you can use them as Symbol.iterator property of an iterable object. This could help you to define the iteration logic in a more elegant and concise way, taking advantage of the yield keyword.

To some extent, you can see generators as a syntactic sugar to define iterable objects.

Conclusion

In this article we learned about different ways to generate dynamic sequences using plain functions, iterators, iterables and generators.

Notice that these approaches are ideal when the operation needed to generate the next element is synchronous (it doesn’t require external resources to be loaded asynchronously).

When you have to iterate over values that become available asynchronously you have to rely on different patterns such as event emitters, streams, or async iterators.

Also, notice that generators have some interesting advanced features not covered in this article, like the opportunity to pass new values in the context every time .next() is called or to throw exceptions, so make sure you checkout the generators documentation.

If you liked this article and you are interested in similar content, be sure to checkout my book Node.js Design Patterns.

This book contains more than 500 pages, filled with more than 100 examples on Node.js (and JavaScript) design patterns. I am sure you won’t be disappointed!

Have fun! :)

Create resources conditionally with CDK

Luciano Mammino — Mon, 01 Nov 2021 09:15:00 GMT

Did you ever need to create a resource based on a condition in CDK? I recently needed to do that and finding a viable solution for this problem took me longer than I originally anticipated. In this article I will try to summarise what I learned and present my solution.

In short, we will learn about the CfnCondition construct and how it can be used to create CloudFormation conditions. Then we will see how to attach condition to low level construct. Throughout this article, we will discuss these concepts with a practical example: creating or importing an S3 bucket based on the value of an SSM parameter.

Create or import an S3 bucket based on a condition with CDK

Let’s start with a practical example: we want to define a stack using CDK and we need to be able to import or create an S3 bucket depending on a specific condition. Let’s also make a use case: our stack will be deployed to multiple environments (development, staging, production, etc.). In the production environment we will need to use a bucket that is already created, while in the other environments we want to create the bucket as part of the stack.

If our condition can be expressed statically (e.g. using an environment variable or a value in the CDK context) then things are easy and we should be able to do something like this:

import * as cdk from '@aws-cdk/core'
import * as s3 from '@aws-cdk/aws-s3'

export class ExampleStack extends cdk.Stack {
  constructor (scope: cdk.Construct, id: string, props: cdk.StackProps) {
    super(scope, id, props)

    const myBucket = process.env['CREATE_BUCKET'] === 'true'
      ? new s3.Bucket(this, 'MyBucket')
      : s3.Bucket.fromBucketAttributes(this, 'MyBucket', {
          bucketName: 'theNameOfTheBucketToImport'
        })
  }
}

But what if we try to replace our expression (process.env['CREATE_BUCKET'] === 'true') with something that depends on other resources on our AWS account? For example, what if we use the value from an SSM parameter?

Let’s see how that might look like:

// ⚠️ NOTE: This does not work!
import * as cdk from '@aws-cdk/core'
import * as s3 from '@aws-cdk/aws-s3'
import * as ssm from '@aws-cdk/aws-ssm'

export class ExampleStack extends cdk.Stack {
  constructor (scope: cdk.Construct, id: string, props: cdk.StackProps) {
    super(scope, id, props)

    const shouldCreateBucket = ssm.StringParameter.fromStringParameterAttributes(
      this,
      'ShouldCreateBucket',
      {
        parameterName: `/ExampleStack/Config/ShouldCreateBucket`
      }
    ).stringValue

    const myBucket = shouldCreateBucket === 'true'
      ? new s3.Bucket(this, 'MyBucket')
      : s3.Bucket.fromBucketAttributes(this, 'MyBucket', {
          bucketName: 'theNameOfTheBucketToImport'
        })
  }
}

If you read the comment at the top of the snippet, you know already this does not work as expected. But what is actually happening here?

Dynamic values like SSM Parameters are not known during the construct phase, which is the lifecycle phase in which our TypeScript code gets executed and CDK collects all the resources that we want to include in the stack. In this phase, CDK will use a Token to represent these values in the context of TypeScript.

So, in the example above, shouldCreateBucket will not contain the actual string value that is stored in the SSM parameter /ExampleStack/Config/ShouldCreateBucket. It will still be a sting value but it will contain something that will look like ${Token[TOKEN.55]}.

Because of this, our TypeScript expression is effectively the following:

'${Token[TOKEN.55]}' === 'true'

Which means that this expression will always evaluate to false. Therefore, we are always importing the bucket and never creating it, regardless of the actual value in our SSM parameter.

Of course, this is not what we want. But, how do we fix it?

Using `CfnCondition` with CDK

The way to solve this problem is to use the concept of condition in CloudFormation.

Conditions exists in CloudFormation to support use cases like ours. They allow to define the circumstances under which certain entities are created or configured for a given stack.

Since we are working with CDK, we can use the concept of condition with the low level CfnCondition construct.

Let’s see how we can create a condition based on the SSM parameter from the previous example:

import * as cdk from '@aws-cdk/core'
import * as s3 from '@aws-cdk/aws-s3'
import * as ssm from '@aws-cdk/aws-ssm'

export class ExampleStack extends cdk.Stack {
  constructor (scope: cdk.Construct, id: string, props: cdk.StackProps) {
    super(scope, id, props)

    const shouldCreateBucket = ssm.StringParameter.fromStringParameterAttributes(
      this,
      'ShouldCreateBucket',
      {
        parameterName: `/ExampleStack/Config/ShouldCreateBucket`
      }
    ).stringValue

    // here's the condition
    const shouldCreateBucketCondition = new cdk.CfnCondition(
      this,
      'ShouldCreateBucketCondition',
      {
        // a condition needs an expression
        expression: cdk.Fn.conditionEquals(shouldCreateBucket, 'true')
      }
    )

    // ...
  }
}

As you can see, a condition is created as any other resource by instantiating an object from the cdk.CfnCondition construct. Note that this is more of a logical resource as it does not create an actual resource on AWS. The interesting part is the expression attribute.

This attribute is used to define an expression that gets evaluated to determine the value of the condition which, at deployment time, needs to be either true or false.

Here we are using cdk.Fn.conditionEquals to indicate that the condition will be true if shouldCreateBucket matches the string 'true'. There are other functions you can use to create more complicated conditions including thing like and or or operators. Check out the documentation of the Fn class if you want to find out more.

This allow us to evaluate the expression at deployment time when the actual value of the SSM parameter will be available, therefore this condition will work as intended.

Now, this condition alone doesn’t really do much. We need to attach the condition to a resource to tell CDK (and CloudFormation) to actually create the given resource only if the condition holds true.

Attaching a condition to a CDK resource

This is where things get a little bit hairy and where I needed to spend a little bit of time to find a working solution.

I was expecting to be able to do something like this:

// ⚠️ NOTE: This does not work!

// ...
const myBucket = new s3.Bucket(this, 'MyBucket', {
  condition: shouldCreateBucketCondition // NOPE!
})

// or

myBucket.setCondition(shouldCreateBucketCondition) // NOPE!

// ...

Eventually I figured out that I can’t specify a condition on a high level construct such as s3.Bucket and that I need to fallback to the equivalent level 0 construct (s3.CfnBucket in this case).

Doing this is not really obvious and the final downcast looks like this:

const cfnMyBucket = myBucket.node.defaultChild as s3.CfnBucket

Once we have an instance of s3.CfnBucket, we can specify a condition with:

cfnMyBucket.cfnOptions.condition = someCfnCondition

So if we put what we have learned together, this is how our conditional creation of a bucket might look like:

import * as cdk from '@aws-cdk/core'
import * as s3 from '@aws-cdk/aws-s3'
import * as ssm from '@aws-cdk/aws-ssm'

export class ExampleStack extends cdk.Stack {
  constructor (scope: cdk.Construct, id: string, props: cdk.StackProps) {
    super(scope, id, props)

    const shouldCreateBucket = ssm.StringParameter.fromStringParameterAttributes(
      this,
      'ShouldCreateBucket',
      {
        parameterName: `/ExampleStack/Config/ShouldCreateBucket`
      }
    ).stringValue

    const shouldCreateBucketCondition = new cdk.CfnCondition(
      this,
      'ShouldCreateBucketCondition',
      {
        expression: cdk.Fn.conditionEquals(shouldCreateBucket, 'true')
      }
    )

    const myBucket = new s3.Bucket(this, 'MyBucket', {
      bucketName: 'my-special-bucket'
    })
    (myBucket.node.defaultChild as s3.CfnBucket).cfnOptions.condition = shouldCreateBucketCondition
  }
}

The final point to address is to figure out how to import the bucket if our condition does not hold.

After thinking about this for a while, I realised that we can always import the bucket. Based in our condition, one of two things can happen:

The bucket will be created (if the SSM parameter value is true)
The bucket is already there (otherwise)

In both cases, if we know the unique name of the bucket, we can import it using s3.Bucket.fromBucketAttributes:

const importedOrCreatedBucket = s3.Bucket.fromBucketAttributes(this, 'ImportedOrCreatedBucket', {
  bucketName: 'my-special-bucket'
})

The code above will give us a valid reference to the bucket in both cases. We can use this generic reference in our stack every time we want to do something with the bucket, for instance grant a permission to another resource:

importedOrCreatedBucket.grantReadWrite(someEc2Instance)

Conclusion

In summary, creating a resource conditionally with CDK requires us to do the following:

define a cdk.CfnCondition with a given expression
downcast the resource we want to create conditionally to it’s level 0 construct equivalent (e.g. from s3.Bucket to s3.CfnBucket)
attach the condition to the lower level construct using cfnResource.cfnOptions.condition = myCondition
finally, if we need to reference this resource in the rest of our stack, we can import the resource using some attribute that will be know regardless if we just created the resource or if we are importing it (e.g. s3.Bucket.fromBucketAttributes)

Our final snippet will look like this:

import * as cdk from '@aws-cdk/core'
import * as s3 from '@aws-cdk/aws-s3'
import * as ssm from '@aws-cdk/aws-ssm'

export class ExampleStack extends cdk.Stack {
  constructor (scope: cdk.Construct, id: string, props: cdk.StackProps) {
    super(scope, id, props)

    // read the SSM parameter
    const shouldCreateBucket = ssm.StringParameter.fromStringParameterAttributes(
      this,
      'ShouldCreateBucket',
      {
        parameterName: `/ExampleStack/Config/ShouldCreateBucket`
      }
    ).stringValue

    // define the condition comparing the value of the SSM parmater to 'true'
    const shouldCreateBucketCondition = new cdk.CfnCondition(
      this,
      'ShouldCreateBucketCondition',
      {
        expression: cdk.Fn.conditionEquals(shouldCreateBucket, 'true')
      }
    )

    // creates the bucket
    const myBucket = new s3.Bucket(this, 'MyBucket', {
      bucketName: 'my-special-bucket'
    })
    // attaches a condition to the creation of the bucket
    (myBucket.node.defaultChild as s3.CfnBucket).cfnOptions.condition = shouldCreateBucketCondition

    // import the bucket by name (regardless if it was just created or already existed)
    const importedOrCreatedBucket = s3.Bucket.fromBucketAttributes(this, 'ImportedOrCreatedBucket', {
      bucketName: 'my-special-bucket'
    })

    // from now on only use `importedOrCreatedBucket`
  }
}

And this is all that I have to share for today!

I hope you will find this useful and please let me know if you end up implementing something like this. I am still learning many of the CDK nuances, so I’d appreciate any feedback. Maybe there are other ways to achieve the same results.

Make sure to leave a comment below and to connect on Twitter so we can keep the conversation going.

Until then, see you on the next post 👋

Rust shenanigans: return type polymorphism

Luciano Mammino — Tue, 13 Apr 2021 09:15:00 GMT

In this article, I will describe Rust return type polymorphism (a.k.a. generic returns), a feature that I recently discovered and that I have been pretty intrigued about.

I am seeing this feature for the first time in a programming language and at first glance, it did seem like some sort of built-in compiler magic, available only in the standard library. In reality, it is a generalised feature that you can use in your own code every day.

Keep in mind that I am still quite a beginner with Rust, so my description might not be the most accurate but I will try to make a point on why I like this feature and how it works by using some examples. Hopefully, you will find this topic as interesting as I did!

Ok, enough chit chat, let’s get into it! 🧐

Return type polymorphism, what?!

Before trying to explain what return type polymorphism actually is, let me show you a couple of interesting examples that might look familiar to you if you have been playing with Rust already:

use std::collections::HashSet;

fn main() {
    let nums = [2, 17, 22, 48, 1997, 2, 22];
    let nums_square_no_dups: HashSet<i32> = nums.iter().map(|x| x * x).collect();
    println!("{:?}", nums_square_no_dups); // {2304, 484, 3988009, 4, 289}
}

Ok, did you see it? I mean, how in the world is that collect() figuring out that I want to “collect” values from an iterator into a HashSet of integers? Indeed, it is giving me exactly a HashSet of integers!

Not convinced yet? We can also change this example slightly and see what happens… Let’s try with Vec:

fn main() {
    let nums = [2, 17, 22, 48, 1997, 2, 22];
    let nums_square: Vec<i32> = nums.iter().map(|x| x * x).collect();
    println!("{:?}", nums_square); // [4, 289, 484, 2304, 3988009, 4, 484]
}

Ok that’s not removing duplicates anymore and it’s preserving the order of elements, but that’s not the point! The point is that collect() is still giving us what we want. This time we are asking for a Vec and indeed we get a Vec rather than a HashMap.

Note that this code is almost the same as the previous one. We didn’t change anything other than the type declaration (and a variable name, but that’s irrelevant)!

Also, keep in mind that, while Rust can ofter infer your types and, most often, you don’t have to provide explicit type definitions, when using collect() the type definition is necessary.

Let’s see what happens if we try to remove Vec<i32>:

fn main() {
    let nums = [2, 17, 22, 48, 1997, 2, 22];
    let nums_square = nums.iter().map(|x| x * x).collect();
}

This example won’t to compile because collect doesn’t know what is the return type, so we get this nice-looking error:

error[E0282]: type annotations needed
 --> src/main.rs:3:9
  |
  |     let nums_square = nums.iter().map(|x| x * x).collect();
  |         ^^^^^^^^^^^ consider giving `nums_square` a type

I hope you starting to get the point. Some functions like collect() can behave differently, based on the expected return type! The return type needs to be explicit, so the Rust compiler can pick the right behaviour for you.

Ok, this is one of the first things I learned in Rust when doing coding challenges and I have been thinking for a while “this is just some iterator magic and it probably works only for a few standard types”…

Then, more recently I saw some other piece of code that was doing something like this:

fn main () {
    let a: u8 = Default::default();
    let b: i64 = Default::default();
    let c: String = Default::default();
    let d: (u16, usize) = (Default::default(), Default::default());

    dbg!(a); // a = 0
    dbg!(b); // b = 0
    dbg!(c); // c = ""
    dbg!(d); // d = (0,0)
}

Ok, types have default values, so what?

Yeah, nothing impressive, I know… But yet again, we are always calling the same function, this time Default::default(), and it’s giving us the default value for the specific type we need!

In all honesty, I am almost glanced over this one thinking it was some other nice Rust compile-time magic for built-in types. Until I noticed a small note saying “by the way, you can have a default value for your custom types if you want to”.

That got my attention, and the example I saw was something like this:

#[derive(Debug)]
struct Point3D {
    x: f64,
    y: f64,
    z: f64,
}

// it's easy pal, just implement the `Default` trait
impl Default for Point3D {
    fn default() -> Self {
        Point3D {
            x: 0_f64,
            y: 0_f64,
            z: 0_f64,
        }
    }
}

fn main() {
    let origin: Point3D = Default::default();
    println!("{:?}", origin); // Point3D { x: 0.0, y: 0.0, z: 0.0 }
}

The Default trait allows you to define what’s the default value for your custom types. The trait forces you to implement a default() method which must return an instance of the given type (Self).

Again, nothing extremely exciting here… except that at this point something clicked in my head and I started to ask myself “so this is some sort of generalised feature that everyone can use…”.

My suspicion was that the Default::default() function can somehow infer the expected return type and, if that type implements the Default trait, then it simply calls the default() function for that type.

Of course, I immediately went and looked for the actual implementation of Default::default() to validate my guess.

Now, I am not going to claim I understood the implementation 100% (there are some lovely macros in there and my brain can’t compile those just yet), but I had a feeling that I was on the right path with my guess.

The interesting bit is the full signature of the Default::default() function:

pub fn default<T: Default>() -> T {
    // ...
}

I read this as:

Default::default() has a generic parameter T
T has a constraint: it can be any type as long as it implements the Default trait
finally (and this is the important point), T is the type that Default::default() must return

And this is where the magic is coming from. The implementation is actually generic over the return type. Also, by using the Default trait constraint, we can have an extensible definition: anyone can implement new types that will work with Default::default(). More on this later…

For now, this is enough theory to digest, let’s try to do something with it!

Let’s build something

Over the years, I learned that my brain can appreciate new programming concepts when I can build something using them. So, I rolled up my sleeves and started to think “OK, what can I possibly build with this?”

I suppose, a good question here is “When do I want to do different things based on different expected return type?”

I came up with a simple idea related to board games. Something like D&D where you have different kind of dice (different number of faces).

Can we use return type polymorphism to be able to roll different type of dice?

I think we can! This is more or less what I did with my first attempt:

use rand::{thread_rng, Rng};

trait Roll<T> {
    fn roll() -> T;
}

#[derive(Debug)]
struct D6(u8);
#[derive(Debug)]
struct D8(u8);

struct Die {}

impl Roll<D6> for Die {
    fn roll() -> D6 {
        D6 {
            0: thread_rng().gen_range(1..=6),
        }
    }
}

impl Roll<D8> for Die {
    fn roll() -> D8 {
        D8 {
            0: thread_rng().gen_range(1..=8),
        }
    }
}

fn main() {
    // let's roll a D6
    let roll: D6 = Die::roll();
    println!("{:?}", roll); // D6(3)

    // let's roll a D8
    let roll: D8 = Die::roll();
    println!("{:?}", roll); // D8(3)
}

For simplicity, I am showing only how I implemented D6 and D8. If you want the full version, check out my D&D dice Rust playground.

The interesting part of this implementation is that I have 2 different implementations for the function Die::roll(), one returns a D6 (6-faces die) roll and the other returns a D8 (8-faces die) roll.

Rust will automatically call the correct implementation based on what’s the expected return type for that invocation.

Well done to me for using return type polymorphism!

Let’s make it extensible

Ok, but my silly implementation has a problem: it is not extensible.

I mean, what if I turn this code into a library and someone needs to implement a custom die with 100 faces? Or what if they want a trick die that always rolls 100?

How can we support this use case?

Well, that’s pretty much what Default::default() does, so it should be possible, right?

Ok, here’s an idea:

We can define a public Rollable trait that makes dice, well … rollable
Our standard D6 and D8 will implement the Rollable trait to be rollable themselves
We provide one single implementation for our roll() function
We make that one implementation generic over a T where T must implement Rollable
T is also the return type for roll()
Now, this implementation of roll() just needs to use the Rollable trait on the current type for T

This idea isn’t really original. We are pretty much mimicking what Default::default() does with the trait Default, except that their naming choice is maybe slightly more confusing, since the module, the function and the trait are all called default 😰 …

Let’s see how it looks like in code:

use rand::{thread_rng, Rng};

/// This is the trait that every die needs to implement to be... well... "rollable", right?
pub trait Rollable {
    /// Roll the die
    fn roll() -> Self;
    /// Get the value from the latest roll
    fn val(&self) -> u8;
}

/// A generic function to roll a given die.
pub fn roll<T: Rollable>() -> T {
    Rollable::roll() // <- Note that here `Rollable` is the current type for a given call!
}

/// A D6 die (6 faces): a roll will give you a `u8` in the `1..=6` range.
#[derive(Debug)]
pub struct D6(u8);

impl Rollable for D6 {
    fn roll() -> D6 {
        D6 {
            0: thread_rng().gen_range(1..=6),
        }
    }
    fn val(&self) -> u8 {
        self.0
    }
}

/// A D8 die (8 faces): a roll will give you a `u8` in the `1..=8` range.
#[derive(Debug)]
pub struct D8(u8);

impl Rollable for D8 {
    fn roll() -> D8 {
        D8 {
            0: thread_rng().gen_range(1..=8),
        }
    }
    fn val(&self) -> u8 {
        self.0
    }
}

That’s pretty much it, now we can keep using our D6 and D8 as before:

fn main() {
    // let's roll a D6
    let r: D6 = roll();
    println!("{:?}", r); // D6(3)

    // let's roll a D8
    let r: D8 = roll();
    println!("{:?}", r); // D8(3)
}

But now, anyone using this library, can also implement their own custom dice:

#[derive(Debug)]
struct Fake100(u8);

impl Rollable for Fake100 {
    fn roll() -> Fake100 {
        Fake100 { 0: 100 } // <- forces it to roll 100
    }
    fn val(&self) -> u8 {
        self.0
    }
}

fn main() {
    println!("I bet I'll get a 100 this time!");
    let d: Fake100 = roll();
    println!("Look what I got: {}!", d.val()) // <- yeah this will always be 100
}

Pretty neat, isn’t it!?

I actually did end up publishing this silly example as a library. Check it out. Who knows, maybe you do really want to implement a board game of some sort!

The turbo-fish syntax

There is still one interesting detail to discuss before we can wrap this up.

We already saw that, when a function is using return type polymorphism, we need to explicitly declare the expected type.

What if you want to call a function implementing return type polymorphism but you don’t want to assign it? Let’s say we want to use the returned value immediately, maybe in a println!():

fn main() {
    println!("{:?}",  roll());
}

How can Rust understand what we want?

In fact, it doesn’t! If you try to compile that code you will get a beautiful error message:

error[E0282]: type annotations needed
  --> src/main.rs:47:23
   |
   |     println!("{:?}",  roll());
   |                       ^^^^ cannot infer type for type parameter `T` declared on the function `roll`

The compiler also suggests running rustc --explain E0282 to get a detailed guide on how to solve this problem. You gotta give it to the Rust team, they have done such a tremendous job in terms of providing great documentation!

Now, if you are patient enough to run that command and read the guide, you will find that we can solve this issue with the so-called turbo-fish syntax!

The turbo-fish syntax looks like… a fish: ::<> … yeah, with some degree of imagination!

So, this is how we actually use it to specify at call time the type for the generic parameter:

fn main() {
    println!("{:?}",  roll::<D6>());
}

OK… Maybe ::<D6>() looks a bit more like a fish…

Anyhow, with this syntax, the type for the parameter T is not ambiguous anymore: we are explicitly saying we want a D6!

In reality, the turbo fish is the extended syntax for functions with generic types, except that, when we are doing an assignment with an well defined type, the Rust compiler is smart enough to infer the type of generic parameters and make our life easier.

Update 2021-04-14: Note though that Rust can infer types from function arguments, so in most cases you can still rely on type inference. Let’s see a quick example:

fn try_dodge_attack(d6: D6, d8: D8) -> bool {
    d6.val() + d8.val() > 10
}

fn main() {
    let escaped = try_dodge_attack(roll(), roll());
    println!(
        "{}",
        match escaped {
            true => "You dogded!",
            false => "Ouch! The attack hit you!",
        }
    );
}

In this example, we are calling try_dodge_attack(roll(), roll()) and we don’t have to use the turbo-fish syntax. The Rust compiler looks at the type declaration of the function arguments and figures out that we want to roll a D6 and a D8.

Conclusion

This concludes our exploration of Rust return type polymorphism.

I do hope you found this topic (and this article!) as interesting as I did and feel more than welcome to let me know in the comments if you did know about this capability already. Are you already using it in production somewhere? I’d be really curious to know your use case, so please share it with me 😇

If you want a much more polished and detailed explanation of this idea check out this brilliant blog post by James Coglan called Generic returns in Rust.

That’s all from me now, see you in the next post! 👋

Thanks to @AlleviTommaso for reviewing this article!

Emerging JavaScript pattern: multiple return values

Luciano Mammino — Sun, 11 Nov 2018 17:00:00 GMT

In this article, I want to explore an interesting pattern that I am seeing more and more in JavaScript code which allows you to return multiple values from a function.

You probably know already that JavaScript does not support multiple return values natively, so this article will actually explore some ways to “simulate” this behavior.

One of the most famous usages of this pattern I have seen recently is within React Hooks, but before delving into that, let’s see what I mean with “multiple return values” by exploring this concept in other languages.

Multiple return values in other languages

Two languages that come to my mind which natively support multiple return values are Lua and Go. Let’s implement a simple integer division function that returns both the quotient and the remainder.

Lua

Let’s start with a simple implementation in Lua. It’s definitely worth mentioning that Lua’s official documentation defines multiple return values as “An unconventional, but quite convenient feature”:

function intDiv (dividend, divisor)
  local quotient = math.floor(dividend / divisor)
  local remainder = dividend % divisor
  return quotient, remainder
end

print(intDiv(10,3)) -- 3  1

Go

Here’s some equivalent code in Go:

package main

import "fmt"

func intDiv(dividend, divisor int) (int, int) {
  quotient := dividend / divisor
  remainder := dividend % divisor
  return quotient, remainder
}

func main() {
  fmt.Println(intDiv(10,3)) // 3 1
}

As you can see in these 2 code snippets, functions can return more than 1 value and this can be very convenient in cases where you logically have produce multiple outputs in a computation.

Note: a more realistic implementation in Go, would take into account errors (e.g. division by 0) and add an extra return value to propagate potential errors. We shouldn’t worry too much about this for the sake of this article, but it is definitely worth mentioning that multiple return values in Go shine when it comes to error propagation and error handling. We will touch a bit more on this later in this article to see how this idea can be applied to JavaScript as well, especially in the context of Async/Await.

Simulating multiple return values in JavaScript

So, as we said early on, JavaScript does not natively support a syntax to return more than one value from a function. We can workaround this limitation by using composite values like arrays or objects.

Multiple return values with arrays

Let’s implement our intDiv in JavaScript by using arrays as return types:

intDiv = (dividend, divisor) => {
  const quotient = Math.floor(dividend / divisor)
  const remainder = dividend % divisor
  return [quotient, remainder]
}

console.log(intDiv(10, 3)) // [ 3, 1 ]

Here we are just printing the result of a division, but let’s assume we want to handle the two return values individually, how do we reference those?

Well, the return value is an array so we can simply access the two elements in the array using the indices 0 and 1:

const result = intDiv(10, 3)
const quotient = result[0]
const remainder = result[1]
console.log(`Quotient = ${quotient}`) // Quotient = 3
console.log(`Remainder = ${remainder}`) // Remainder = 1

This syntax is arguably verbose and definitely not very elegant. Thankfully, ES2015 array destructuring assignment can help us here:

const [quotient, remainder] = intDiv(10, 3)
console.log(`Quotient = ${quotient}`) // Quotient = 3
console.log(`Remainder = ${remainder}`) // Remainder = 1

This is much nicer to read and we also trimmed away 2 out 3 lines of code, big win!

As nice as it is, this implementation has an important shortcoming: return values are positional, so you need to be careful and respect the order while destructuring.

Multiple return values with objects

An alternative implementation could use objects as return value, let’s see how:

intDiv = (dividend, divisor) => {
  const quotient = Math.floor(dividend / divisor)
  const remainder = dividend % divisor
  return { quotient, remainder }
}

Note that here we are using another syntactic sugar from ES2015 (Enhanced object literal syntax) that allows us to define objects very concisely. Prior to ES2015, we would have defined the return statement as {quotient: quotient, remainder: remainder}.

With this approach we will be able to use our intDiv function as follows:

const result = intDiv(10, 3)
const quotient = result.quotient
const remainder = result.remainder
console.log(`Quotient = ${quotient}`) // Quotient = 3
console.log(`Remainder = ${remainder}`) // Remainder = 1

Again, this is a bit too verbose and ES2015 has another fantastic syntactic sugar to make this nicer:

const { quotient, remainder } = intDiv(10, 3)
console.log(`Quotient = ${quotient}`) // Quotient = 3
console.log(`Remainder = ${remainder}`) // Remainder = 1

This syntactic sugar is called Object Destructuring Assignment. With this approach we are now independent from the position of return values (we can swap the position of quotient and remainder without side effects). This syntax also lets you rename the destructured variables, which can very useful to avoid name collisions with other local variables, or just to make variable names shorter or more descriptive as we please. Let’s see how this works:

const { remainder: r, quotient: q } = intDiv(10, 3)
console.log(`Quotient = ${q}`) // Quotient = 3
console.log(`Remainder = ${r}`) // Remainder = 1

Here we are not dependent by values position, but by their names in the returned object. If you are designing an API with multiple return values, it’s up to you to figure out which trade off will be the best to guarantee a proper developer experience.

Ok, now you should have a good idea on how to simulate multiple return values in JavaScript. In the next section we will see some more realistic examples that take advantage of this pattern.

Some more realistic use cases

As mentioned early on, this technique has been recently popularized by React hooks, so we are gonna explore this use case first. Later we will see other two cases related to Async/Await.

React Hooks

React hooks are a new feature proposal available from React v16.7.0-alpha that lets you use state and other React features without having to write a class.

The first and most famous React hook present is called State Hook.

Let’s see how it works with an example, let’s build a CSS color viewer component.

Here’s how our component is going to look like:

And here’s the code used to implement this:

import { useState } from 'react'

function CssColorViewer() {
  const [cssColor, setCssColor] = useState('Blue')
  //                          <-- multiple return values

  const onCssColorChange = e => {
    setCssColor(e.target.value)
  }

  return (
    <div>
      <input value={cssColor} onChange={onCssColorChange} />
      <div
        style={{
          width: 100,
          height: 100,
          background: cssColor,
        }}
      />
    </div>
  )
}

You can see this component in action and play with the code on CodeSandbox.

For the sake of this article, we are going to focus only on the useState call, but if you are curious to understand better how the hook itself works internally I really recommend you read the official State Hook documentation. I was personally curious to understand how multiple useState calls could maintain the relationship with the specific state attribute (since there’s no explicit labelling or reference). If you are curious about that too, well you should read the Hooks FAQ and Dan Abramov’s recent article about Hooks.

Back to the useState call in our example, now!

The useState hook acts like a factory: given a default value for the state property ('Blue' in our case), it will need to instantiate for you 2 things:

the current value for the specific state property (cssColor in our case)
a function that allows you to alter the specific property (setCssColor in our case)

React developers decided to handle this requirement by simulating multiple return values with an array.

Combining this with array destructuring and proper variable naming, the result is an API that is very nice to read and to use.

This React feature is still very experimental and subject to change at the time of writing, but it already sounds like a big deal for the React community to make the code more expressive and reduce the barrier to entry to start adopting React.

The point I want to make is that, in this specific case, the multiple return values pattern plays a big role towards this goal.

Converting callbacks API to Async/Await

Recently I found another great use case for the multiple return values pattern while trying to convert a callback oriented API into an equivalent Async/Await API.

To make this part clear, I am going to explain very quickly an approach I use to convert callback based APIs into functions that I can use with Async/Await.

Let’s take this generic example:

function doSomething(input, callback) {
  // ... do something asynchronously and
  //     compute response or error
  // When finished, invoke the callback:
  callback(error, response)
}

To convert this function into something that can be used with Async/Await we have to essentially promisify it. There are libraries to do it and, if you are using Node.js you can even use the builtin util.promisify, but that’s something we can do ourselves by just creating a wrapper function like the following one:

const doSomethingPromise = (input) => new Promise((resolve, reject) => {
  doSomething(input, (error response) => {
    if (error) {
      return reject(error)
    }

    return resolve(response)
  })
})

In short, our wrapper function doSomethingPromise is immediately returning a Promise. Inside the body of the promise we are invoking the original doSomething function with a callback that will be resolving or rejecting the promise based on whether there’s an error or not.

Now we can finally take advantage of Async/Await:

// inside an async function
const response = await doSomethingPromise(input)

Note: this will throw in case of error, so make sure you have it in a try/catch block to handle the error correctly.

If you are curious about promisifying callback-based functions, I have an entire article dedicated to this topic.

In my specific use case, I was using a twitter client library that follows this conventions:

// client is an instance of the twitter client
client.get('statuses/user_timeline', params, function callback(
  error,
  tweets,
  response
) {
  if (!error) {
    console.log(tweets)
  }
})

The important detail here is that the callback function is a bit unconventional because it receives 3 parameters: a possible error, a list of tweets and a response (which represents the raw HTTP response object). Conventional callback-style APIs will send only two parameters to the callback function: a potential error and some sort of result object.

The question here is: how to promisify this unconventional API?

Multiple return values to the rescue!

const getUserTimeline = (client, params) =>
  new Promise((resolve, reject) => {
    client.get('statuses/user_timeline', params, (error, tweets, response) => {
      if (error) {
        return reject(error)
      }

      return resolve([tweets, response]) // multiple return values
    })
  })

We can use this function with Async/Await as follows:

// inside an async function

// ... set `client` and `params`
const [tweets, response] = await getUserTimeline(client, params)

Note: don’t forget to have a try/catch to handle errors!

To recap, we can use the multiple return values pattern also with promises to allow them to be resolved to multiple values. This technique gives us a very nice interface especially when used in combination with Async/Await.

Async/Await with alternative error handling

Another closely related pattern I am seeing more and more in JavaScript is error propagation and handling à la Go.

In Go, when a function can generate an error, this error is not thrown but simply returned by the function. If the function has to return some output and can also generate errors, then the function will have multiple return values (output and error).

The caller code, should ideally verify if the returned error value is an actual error before proceeding, as in the following Go code example:

file, err := os.Open("file.go")
if err != nil {
  log.Fatal(err)
}

// ... do something with `file`

Some JavaScript libraries are starting to promote the same conventions as in Go to report errors, especially when it comes to Async/Await.

You can find some examples by just searching "error = await" on GitHub.

Let’s rewrite our getUserTimeline function from our previous example to follow this approach:

const getUserTimeline = (client, params) =>
  new Promise((resolve, reject) => {
    client.get('statuses/user_timeline', params, (error, tweets, response) => {
      return resolve([error, tweets, response]) // multiple return values
    })
  })

Notice that we are never telling the promise to reject, so when we will be using this function with Async/Await we cannot use try/catch to handle errors. This is what we should do instead:

// inside an async function

// ... set `client` and `params`
const [error, tweets, response] = await getUserTimeline(client, params)

if (error) {
  // handle error here
}

// ... do stuff with `tweets` and `response`

I am not sure if I would recommend this pattern or not in JavaScript land. I have a bit of mixed feelings about it. On one side I quite like it, because as happens in Go, it forces you to handle every single error individually, which makes you think a bit more carefully about the best way to handle the specifics of the error. On the other hand, this pattern it still feels a bit forced into JavaScript and people that never saw this pattern in other languages might find it annoying or even hard to understand. Moreover, if you don’t capture and handle an error this will not automatically escalate (as it happens with throw or rejected promises), so the error will totally be swallowed by the runtime, leading to potential inconsistencies in your app state. I’ll let you draw your own conclusions on this one! 😇

Array destructuring tricks

At this point, I would like to show you few “little tricks” that you can use with array destructuring that might come in handy when dealing with multiple return values.

Skipping elements

Let’s say, for instance, that you have a function doStuff that returns multiple values using an array and the values are in the array are an error, a rawResponse and result. Let’s say only for the sake of this example that you are not interested in using the rawResponse, you could easily skip that element while destructuring with the following syntax:

const [error, , result] = doStuff()

Notice the double comma there. That basically means that we are leaving an array index unassigned. You can bend this technique as you please, for example you might decide to destructure only the result:

const [, , result] = doStuff()

Well, you shouldn’t really skip errors though…

Aggregate remaining return values

Another interesting trick is that you can use the special ... syntax to accumulate the remaining return values under a single variable. Let’s make a dummy example to explore this idea.

Let’s say we have a function called listDogs which is implemented as follows:

function listDogs() {
  return [3, 'Bella', 'Lucy', 'Daisy']
}

The first element of the array is the number of dogs, while every other elements are actual dog names.

Since the number of returned elements here is variable it might be tricky to use destructuring directly. In these cases we can use this special syntax:

const [numDogs, ...dogNames] = listDogs()
console.log(numDogs) // 3
console.log(dogNames) // [ 'Bella', 'Lucy', 'Daisy' ]

The ... syntax basically allows you to destructure any remaining element of the array into another array. Of course you can only have a ... element in the left hand side of the destructuring expression and this has to be the last element in the list.

My example here is very dummy, but this pattern really shines in some real life use cases, for example when you want to unpack values from a regular expression match.

Performance implications

When React hooks were presented the community was super excited about this new feature, but some performance experts in the Google chromium team (@bmeurer, @_developit and @rossmcilroy) raised their eyebrows and wanted to dig deeper to see if this new approach could lead to serious performance issues in the web.

What came out from their research is an amazing paper that goes by the title “Array destructuring for multi-value returns (in light of React hooks)”.

I really encourage you to read it to get all the details, but here’s my attempt at giving you some sort of TLDR;

In React components the render() method is called often, so the code there should be optimized enough not to slow things down. That’s where you use React hooks.
Array destructuring is a very generic API, it doesn’t work only with arrays but with every type of iterable object, so the VM has a lot work to do to figure out how to traverse and destructure the specific object.
Object destructuring might be a more performant alternative, but it would probably offer a less pleasant developer experience.
If you use Babel in loose mode, array destructuring will be highly simplified (to direct element access) and it will result in highly optimized code.
There’s work going on to see if the different phases of the optimizing compiler in the Google’s V8 engine might be able to optimize destructuring even if you are not using Babel.

Recap

In this article we discussed this new emerging JavaScript pattern that is getting more and more traction, mostly because of its adoption within React.

Use cases are not limited to React though and I hope you will find this useful in your daily development life also outside the React land.

I am really curious to see what you will come up with, so please keep me in the loop using twitter or the comments in this article.

For the very curious ones I want to give you one last link, so that you can compare how multiple return values are implemented in many many other languages.

Until next time!

How to to_string in Rust

Luciano Mammino — Wed, 26 May 2021 18:50:00 GMT

In Rust, there are several ways to turn a value into a string. In this article, we will explore a few different ways and discuss what are the most idiomatic approaches depending on the context you are currently working on.

Personally, I have been quite confused for a while on what’s the best way to implement a “to string” functionality for a given Rust struct. The reason why this has been confusing to me is that there are indeed many ways to do that and they all have different purposes.

I finally decided to do a bit of research to try and demystify this topic a bit and, in this post, I want to share what I learned!

Are you ready? 🙂

Implementing our own `to_string()`

Coming from other languages and not having a deep knowledge of the most idiomatic Rust approaches, the first thing that I generally tend to do when facing a problem is “let’s just make this work for now”.

So, in the spirit of just making things work, the first thing that we can do is to just implement our own to_string() method on a given struct.

At the end of the day, what we want is just to be able to turn a given value into a String value, essentially something like:

someValue.to_string(); // returns a String value

For the sake of having a consistent example throughout the article, let’s pretend that we are working on a struct that allows us to manage API credentials. In this context credentials are made up of 2 separate values:

An api_key: effectively a unique id for the key.
A secret: a secret string associated with the key. Something that we could use to sign API requests.

We can store this data in a struct called Credentials:

pub struct Credentials {
    api_key: String,
    secret: String,
}

Ok, now let’s add a constructor and a to_string method to this struct:

impl Credentials {
    pub fn new(api_key: String, secret: String) -> Self {
        Credentials {
            api_key,
            secret,
        }
    }

    pub fn to_string(&self) -> String {
        // We don't want to disclose the secret
        format!("Credentials({})", &self.api_key)
    }
}

Quick note: the secret is a piece of sensitive information, so it makes sense not to print it out in our to_string() method.

Now, we can use our new struct:

fn main() {
    let creds = Credentials::new(String::from("SOME_API_KEY"), String::from("SOME_SECRET"));
    println!("{}", creds.to_string());
}

The snippet above is going to print:

Credentials(SOME_API_KEY)

Success! 🎉

OK, this is easy and it works! But, let’s face it, the implementation is very specific to our struct!

What I mean by that is that the rest of the codebase doesn’t really know that this type can be converted to a String. It is just a method like any other and there is no agreement or standard that says that this is how you signal that a given value can be converted to a string. Therefore, we cannot build abstractions on top of this…

In fact, note how we needed to explicitly call to_string() in our println!() call.

If we try to remove that and just pass the creds value we get an error:

error[E0277]: `Credentials` doesn't implement `std::fmt::Display`
  --> src/main.rs:22:20
   |
22 |     println!("{}", creds);
   |                    ^^^^^ `Credentials` cannot be formatted with the default formatter
   |
   = help: the trait `std::fmt::Display` is not implemented for `Credentials`
   = note: in format strings you may be able to use `{:?}` (or {:#?} for pretty-print) instead
   = note: required by `std::fmt::Display::fmt`
   = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error

For more information about this error, try `rustc --explain E0277`.

If we zoom in a little, the error message is clear:

"`Credentials` cannot be formatted with the default formatter"

As expected, the Rust compiler doesn’t seem to understand that we have defined a way to turn Credentials values into a string!

Wouldn’t it be nice if we could somehow tell the Rust compiler that our Credentials type can be stringified?

Rust traits

If we have a second, more in-depth, look at the error above, there’s an interesting hint there:

the trait `std::fmt::Display` is not implemented for `Credentials`

The Rust compiler is trying to be helpful and it’s telling us:

“You know, if you want to be able to automatically convert Credentials values to a string, you should look into implementing the std::fmt::Display trait”

In Rust, structs can expose certain common behavior by implementing specific traits.

In other languages, you can do the same by extending certain classes or implementing certain interfaces. Other languages do the same by convention (or by protocols): if you implement certain methods with very specific names, arguments, and return types then your type (or object) can exhibit a certain behavior.

Regarding the “to string behavior”, in Rust, there are several interesting traits that we should look into!

the std::fmt::Debug trait
the std::string::ToString trait
the std::fmt::Display trait (the one recommended by the previous error message)

They have very specific purposes, so in the rest of this article, we will be exploring all of them and discuss when you should be using them.

The `Debug` trait

Let’s start with the Debug trait.

The Debug documentation says:

“Debug should format the output in a programmer-facing, debugging context”.

This is a great way to provide details about a struct, providing a message that should be visible only to developers in a debugging context.

An interesting thing is that we can get Rust to auto-implement the Debug trait for us by using the Derive macro:

#[derive(Debug)]
struct SomeStruct{}

So, if we want to implement the Debug trait in our example, we could do it as follows:

#[derive(Debug)]
pub struct Credentials {
    api_key: String,
    secret: String,
}

impl Credentials {
    pub fn new(api_key: String, secret: String) -> Self {
        Credentials {
            api_key,
            secret,
        }
    }
}

fn main() {
    let creds = Credentials::new(String::from("SOME_API_KEY"), String::from("SOME_SECRET"));
    println!("{:?}", creds);
}

The code above will output:

Credentials { api_key: "SOME_API_KEY", secret: "SOME_SECRET" }

Did you notice that we used the {:?} placeholder in our format string? This is the placeholder that indicates you want to print the value in “debug mode”.

A small productivity tip here: you can also use the {:#?} placeholder (note the hash) if you want the output to be pretty-printed! If we do that in our with our Credentials struct from the previous example, it will be printed like this:

Credentials {
    api_key: "SOME_API_KEY",
    secret: "SOME_SECRET",
}

But what if we want to customize the string generated in “debug mode”?

For instance, we might not want to display the full secret but only the first 4 characters and obfuscate all the remaining ones with asterisks.

Well, in this particular case, we can implement the Debug trait manually:

use std::fmt;

pub struct Credentials {
    api_key: String,
    secret: String,
}

impl Credentials {
    pub fn new(api_key: String, secret: String) -> Self {
        Credentials { api_key, secret }
    }
}

impl fmt::Debug for Credentials {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        f.debug_struct("Credentials")
            .field("api_key", &self.api_key)
            .field(
                "secret",
                &self
                    .secret
                    .chars()
                    .enumerate()
                    .map(|(i, c)| if i < 4 { c } else { '*' })
                    .collect::<String>(),
            )
            .finish()
    }
}

fn main() {
    let creds = Credentials::new(String::from("SOME_API_KEY"), String::from("SOME_SECRET"));
    println!("{:#?}", creds);
}

The code above will output:

Credentials {
    api_key: "SOME_API_KEY",
    secret: "SOME*******",
}

Implementing the Debug trait manually is something that you rarely have to do manually. For the majority of use cases, the Derive macro will serve you well!

It is interesting to note that implementing the Debug trait manually requires you to use a Formatter.

From the documentation:

“A Formatter represents various options related to formatting. Users do not construct Formatters directly; a mutable reference to one is passed to the fmt method of all formatting traits, like Debug and Display.”

In short, a formatter is a utility that helps you to build the output string you want to generate. If you are curious to find out more, you can check out the official documentation page on the Formatter type.

The `ToString` trait

Let’s now talk about the std::string::ToString trait, which is defined as:

“A trait for converting a value to a String”

But the documentation also says that this trait shouldn’t be implemented directly. The Display trait should be implemented instead and by doing that you get the ToString implementation for free!

How is that possible? I mean, how is it possible that by implementing a trait, we get another one implemented automatically?

In Rust, we can implement a trait for any type that implements another trait. Implementations of a trait on any type that satisfies the trait bounds are called blanket implementations and are extensively used in the Rust standard library.

What this means in practice is that somewhere in the Rust core library there is some code like this:

impl<T: Display> ToString for T {
    fn to_string(&self) -> String {
        // blanket implementation here...
    }
}

This is basically telling the Rust compiler how to provide a default implementation of the ToString trait for any generic types T that implements the Display trait.

Implementing ToString for a type will force that type to have a to_string() method. But the more idiomatic way to tell Rust that a type can have a user-facing string representation is to implement the more generic Display trait.

UPDATE: It’s is interesting to know that the Display trait is implemented in the core module and does not use any memory allocator. String is heap-allocated, so it couldn’t have been used in Display’s definition. Rust designs traits carefully to keep heap allocating functions separated from the ones that don’t need a memory allocator (core).

Thanks to kornel from lobste.rs for this tip.

At this point, we can practically ignore the ToString trait and focus only on the Display trait!

The `Display` trait

From the official Display documentation:

“Display is similar to Debug, but Display is for user-facing output, and so cannot be derived”.

The documentation is essentially saying that Display allows us to provide a user-facing description of a type and that we can only implement the trait directly, no magic derive!

Let’s implement Display for our Credentials struct then:

use std::fmt;

pub struct Credentials {
    api_key: String,
    secret: String,
}

impl Credentials {
    pub fn new(api_key: String, secret: String) -> Self {
        Credentials { api_key, secret }
    }
}

impl fmt::Display for Credentials {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.write_str(self.api_key.as_ref())
    }
}

fn main() {
    let creds = Credentials::new(String::from("SOME_API_KEY"), String::from("SOME_SECRET"));
    println!("{}", creds); // "SOME_API_KEY"
}

Note that, again, we have to deal with a formatter.

A small productivity tip is that you can use the write! macro with formatters and convert:

f.write_str(self.api_key.as_ref())

into:

write!(f, "{}", self.api_key)

Nicer and more flexible, isn’t it?

UPDATE: the Formatter type is more flexible because it allows us to write data into an arbitrary buffer (rather than always allocating new Strings). For instance we could pre-allocate a buffer once (as a String) and write onto it multiple times as in the following example:

use std::fmt::Write;

fn main() -> fmt::Result {
    let foo = Credentials::new(String::from("foo"), String::from("foosecret"));
    let bar = Credentials::new(String::from("bar"), String::from("barsecret"));

    // pre-allocated buffer
    let mut output = String::with_capacity(200);

    write!(&mut output, "{}", foo)?;
    write!(&mut output, "{}", bar)?;
    println!("{}", output); // foobar

    Ok(())
}

Thanks to nicoburns from Reddit for this tip.

Also, remember that the placeholder to use the Display trait is just {} (as opposed to {:?} or {:#?} for the Debug trait).

Summary

So, just to summarise:

Debug allows you to generate a debug representation for a given type. It can be automatically derived.
Display is the equivalent but for user-facing information. It CANNOT be derived.
ToString… don’t implement it, just implement Display and you will get it for free thanks to the standard blanket implementation!

That’s all I know about stringifying things in Rust! 😊

I hope this article was insightful and I am curious to know if you learned something new or if all these things were already done and dusted in your Rust journey! Let me know that in the comments!

Did I miss or misunderstood something? Please, let me know that as well! ❤️

If you’d like to see more of my Rust learning journey check out my new twitch channel, where I stream every week (Monday 5PM GMT) with my friends Eugen and Roberto our attempts at cracking the Advent of code challenges using Rust!

And if you prefer to read rather than watching long random-ish streaming sessions, you could check the other Rust articles in this blog. There are already a good few! 😱

CIAO 🙃

Provision an Ubuntu-based EC2 instance with CDK

Luciano Mammino — Fri, 06 Aug 2021 13:05:00 GMT

You are using CDK and you need to provision an EC2 instance. What if you prefer to use Ubuntu over Amazon Linux? In this article, we will see exactly how to do that, and, hopefully, we will learn a bunch of interesting things in the process!

I have to be honest, I am more on the Ubuntu camp than I am in the Amazon Linux one… It’s a subjective preference. I find myself more comfortable with apt, snap, systemd and other Ubuntu nuances than I am with yum and other things in Amazon Linux. Plus, I find easier to find resources about how to do all sort of things with Ubuntu.

So, the story goes that I was playing with CDK and I was trying to deploy a simple Node.js application to an EC2 instance. All the CDK examples I could find were using Amazon Linux, but I thought it wouldn’t be too complicated to switch to Ubuntu instead. I already had a Systemd service definition written for my app and a script to install all dependencies using apt.

How hard could it be to use Ubuntu? It turns out there are some dark corners and the necessary documentation is scattered around the web. I will try to document my finding and provide a complete example in this article. We will also see how to install the aws CLI and other necessary AWS utilities in our Ubuntu-based virtual machine. Once you do all these things, you should have a Ubuntu image that is pretty much on par with the Amazon Linux one.

What is CDK

Ok, if you know CDK already, you can just skip this section. If you don’t, how on the world wide web did you end up on this page? 😅

CDK stands for Cloud Development Kit and it’s a relatively new tool from Amazon to write infrastructure as code (IaC).

CDK allows you to define all your cloud infrastructure programmatically so you can keep it versioned and you can have a reproducible deployment process. If you use CDK you won’t have to go in the AWS web console and click around to provision resources, you’ll write code and use command-line tools (or CI/CD pipelines) to do that for you.

If you have used tools like Terraform or CloudFormation already, CDK addresses the same type of problems. What’s different with CDK is that you don’t have to write a ton of JSON, Yaml or learn a new markup language such as HCL.

In fact, with CDK you can use your favorite programming language (TypeScript, Python, Java and C# are supported right now) and you can define the resources in your cloud infrastructure by importing classes and instantiating objects.

Just to give you an example, this is how you can create a new SSM parameter using CDK (TypeScript):

import * as ssm from '@aws-cdk/aws-ssm'

new ssm.StringParameter(stack, 'Parameter', {
  allowedPattern: '.*',
  description: 'The value Foo',
  parameterName: 'FooParameter',
  stringValue: 'Foo',
  tier: ssm.ParameterTier.ADVANCED,
})

You can see already that the main advantage of CDK is that you’ll be able to use your favorite language, your favorite code editor or IDE and get other nice things like syntax highligting, type checking, auto-completion, etc.

I have found that there’s much less guesswork or “writing configuration by trial and error” when using CDK as opposed to writing plain CloudFormation configuration or using Terraform.

It’s worth knowing that CDK is not a standalone tool, but it’s an abstraction built on top of CloudFormation. So when you synthesise (which is how you say compile in CDK lingo) a CDK stack, you actually get a CloudFormation stack. When you run cdk deploy you are actually deploying that stack with CloudFormation. This way, you still get all the benefits of CloudFormation, but you also get a nicer way to write down your infrastructure as code.

If you want to know how to get started with CDK check out the official guide Getting started with the AWS CDK.

Defining An EC2 resource with CDK

From here, I am assuming you have already installed CDK and generated a project with cdk init.

If you want to have a look at the final version of the code we will be discussing here, you can check out the project repository on GitHub.

To provision an EC2 instance with CDK using TypeScript we need to install the package @aws-cdk/aws-ec2:

npm i --save @aws-cdk/aws-ec2

Then we can define the instance in a Stack with the following code (intentionally incomplete):

import * as cdk from '@aws-cdk/core'
import * as ec2 from '@aws-cdk/aws-ec2'

export class MyAppStack extends cdk.Stack {
  defaultVpc = ec2.Vpc.fromLookup(this, 'VPC', {
    isDefault: true,
  })

  myVm = new ec2.Instance(this, 'myVm', {
    // the type of instance to deploy (e.g. a 't2.micro')
    instanceType: new ec2.InstanceType('t2.micro'),
    // the id of the image to use for the instance
    machineImage: someAmiId,
    // A reference to the object representing the VPC
    // you want to deploy the instance into
    vpc: defaultVpc,
    // ... more configuration
  })
}

As you can see that there are 3 mandatory pieces of information we need to provide:

The type of EC2 instance we want to run (i.e. t2.micro)
The id of the machine image (more on this in the next section)
The id of the VPC (Virtual Private Cloud) where you want to deploy your instance. For example, in the snippet above, we are using the default VPC in your AWS account.

You can also specify a ton of other optional configuration options (Security Groups, a role for IAM permissions, volumes, etc). For a full list of supported options you can check out the official documentation for the CDK EC2 Instance construct.

Ok, now how do we specify that we want to use an Ubuntu-based image? Let’s talk about AMIs!

Ubuntu AMIs and where to find them

When provisioning EC2 instances on AWS you have to provide a virtual machine image (an AMI in AWS lingo). You can build and manage your own custom private images or use one of the many public images for different Linux distributions.

These two approaches have their pros and cons:

If you use a public image, you will need to use an init script to provision the machine with all the needed software and configuration. This is a quick and easy way to experiment and iterate over different versions of your virtual machines. On the other hand, this approach might be slow, depending on how much stuff you need to do at every boot.
If you build your own custom images you’ll need to have a pipeline (or some other reproducible approach) to bootstrap a base image, install all your software and create a new image from there. This path adds a bit more complexity but then your images will have everything built in and the bootstrap will probably be much faster.

In this article we will go with a public image (Ubuntu) and we will provide the additional bits and pieces by using an init script.

If you are interested in the “build your own image” approach, I’d recommend you to check out EC2 Image Builder or Packer (a brilliant tool for building VM images from the same authors of Terraform).

So, back to our main question: how do we find a base image with Ubuntu installed?

There’s an entire website dedicated to finding public AMIs for just Ubuntu: Ubuntu Amazon EC2 AMI Locator.

Every image is built for a specific AWS region, processor architecture, and instance type. Every image is identified by a unique ID called AMI ID.

When provisioning a new EC2 instance you need to specify the wanted AMI ID and make sure to select the correct one for the region you are deploying your instance to.

There are a couple of different ways to do that.

Machine AMI map

The simplest way to specify AMI IDs for all the regions we want to support is to use the ec2.MachineImage.genericLinux() helper function.

This function accepts a map of AMI IDs where keys are regions and values are AMI IDs.

Here’s an example:

const machineImage = ec2.MachineImage.genericLinux({
  'eu-west-1': 'ami-0298c9e0d2c86b0ed',
  'us-east-1': 'ami-019212a8baeffb0fa',
  'ap-east-1': 'ami-04c4bc345657bf245',
  // ... add all the regions you need to support
})

Here I have copied some AMI IDs manually from the Ubuntu Amazon EC2 AMI Locator website.

This is a bit of tedious copy-paste ( 🍝 ) to do and you need to make sure you do it right. Furthermore, what if there’s an update? There will be new IDs and we will need to update them manually again… 🥲

Of course, there’s an easier and more maintainable way! 😋

Machine AMI from a public SSM parameter

The better way is to use the function ec2.MachineImage.fromSSMParameter().

As the name suggests, this function allows us to use take the AMI ID from an SSM parameter. At a first glance, this idea might not sound particularly appealing. In fact, using SSM adds another layer of complexity. We would need to store the wanted AMI ID in SSM first somehow, only to consume the value later from our CDK stack.

The good news is that there are some public SSM parameters that are already pre-populated with the latest ubuntu AMI IDs.

Let’s try to run this in our terminal (you’ll need to have the AWS CLI installed and configured):

aws ssm get-parameter --name "/aws/service/canonical/ubuntu/server/focal/stable/current/amd64/hvm/ebs-gp2/ami-id"

The command above will output something like this:

{
  "Parameter": {
    "Name": "/aws/service/canonical/ubuntu/server/focal/stable/current/amd64/hvm/ebs-gp2/ami-id",
    "Type": "String",
    "Value": "ami-0298c9e0d2c86b0ed",
    "Version": 221,
    "LastModifiedDate": "2021-07-21T14:31:50.717000+01:00",
    "ARN": "arn:aws:ssm:eu-west-1::parameter/aws/service/canonical/ubuntu/server/focal/stable/current/amd64/hvm/ebs-gp2/ami-id",
    "DataType": "aws:ec2:image"
  }
}

As the name of the SSM parameter suggests, these SSM parameters are managed directly by Canonical (the company behind Ubuntu). You can also see that there is a specific path structure to identify the AMI ID. If you are curious to find out more and to discover all the available images you can check out Finding Ubuntu Images with the AWS SSM Parameter Store.

One more thing to notice is that SSM parameters are region-specific. You can have the same parameter key in different regions and therefore have different values in different regions (given the same key).

I am running this command using eu-west-1 as the default region and you can see that the Value matches the AMI ID we have manually specified with the AMI map approach. Try to switch your region using the --region flag and you will get the correct AMI ID for the given region.

This is a rather clever usage of SSM if you ask me! 🤩

Enough chit chat, let’s see how to use this approach with CDK:

const machineImage = ec2.MachineImage.fromSSMParameter(
  '/aws/service/canonical/ubuntu/server/focal/stable/current/amd64/hvm/ebs-gp2/ami-id',
  ec2.OperatingSystemType.LINUX,
)

That’s it! Now we can pass the machineImage value to the settings of our EC2 instance!

A simple EC2 instance with Ubuntu

Ok, now we have all the building blocks to provision the most basic Ubuntu-based EC2 instance.

This is how the code for our stack (lib/*stack.ts) looks like so far:

import * as cdk from '@aws-cdk/core'
import * as ec2 from '@aws-cdk/aws-ec2'

export class CdkUbuntuEc2Stack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, {
      env: {
        account:
          process.env.CDK_DEPLOY_ACCOUNT || process.env.CDK_DEFAULT_ACCOUNT,
        region: process.env.CDK_DEPLOY_REGION || process.env.CDK_DEFAULT_REGION,
      },
      ...props,
    })

    const defaultVpc = ec2.Vpc.fromLookup(this, 'VPC', {
      isDefault: true,
    })

    const machineImage = ec2.MachineImage.fromSSMParameter(
      '/aws/service/canonical/ubuntu/server/focal/stable/current/amd64/hvm/ebs-gp2/ami-id',
      ec2.OperatingSystemType.LINUX,
    )

    const myVm = new ec2.Instance(this, 'myVm', {
      instanceType: new ec2.InstanceType('t2.micro'),
      machineImage: machineImage,
      vpc: defaultVpc,
    })
  }
}

If we try to run cdk deploy we should see something like this:

We can now say y to confirm the deployment and sit patiently with our fingers crossed… 🤞

Ok … ⏱ wait over!

If all went well, we should see something like this in the console:

 ✅  CdkUbuntuEc2Stack

Stack ARN:
arn:aws:cloudformation:eu-west-1:012345678901:stack/CdkUbuntuEc2Stack/a0d34481-a364-4261-864b-3cb16f57dc15

Our machine is deployed! 🎉

As a small aside, I have to say that I love the usage of some good emojis in the terminal, well-done AWS!

But let’s be fair, our virtual machine is not very useful… it doesn’t do anything right now!

Let’s try to update our machine and make it do something useful. For instance, we could install nginx and enable ingress on port 80 so that, once the machine is provisioned, we will be able to see the default nginx welcome page.

In a real-life scenario, you might do something a bit more sophisticated to deploy and run a real application on the machine.

Adding a security group to an EC2 with CDK

Let’s start by adding a security group to our EC2, this is something we can do by adding the following code to our stack:

const myVmSecurityGroup = new ec2.SecurityGroup(this, 'myVmSecurityGroup', {
  vpc: defaultVpc,
})
myVmSecurityGroup.addIngressRule(
  ec2.Peer.anyIpv4(),
  ec2.Port.tcp(80),
  'httpIpv4',
)
myVmSecurityGroup.addIngressRule(
  ec2.Peer.anyIpv6(),
  ec2.Port.tcp(80),
  'httpIpv6',
)

Then we need to associate this new security group to the instance by setting the securityGroup property:

const myVm = new ec2.Instance(this, 'myVm', {
  // ...
  securityGroup: myVmSecurityGroup,
  // ...
})

Ok, now our virtual machine will be able to accept ingress traffic on port 80 from all IPv4 and IPv6 addresses.

Installing software by using CloudFormation init with CDK

CloudFormation supports a way to execute init scripts once a machine boots.

CDK supports this feature as well and we can use it with the init configuration option in our VM definition:

const myVm = new ec2.Instance(this, 'myVm', {
  // ...
  init: ec2.CloudFormationInit.fromElements(
    ec2.InitCommand.shellCommand('sudo apt-get update -y'),
    ec2.InitCommand.shellCommand('sudo apt-get install -y nginx'),
  ),
  // ... more configuration
})

CDK signal problems with Ubuntu

At this point we are ready to do another deployment, so let’s run cdk deploy.

This time the deployment should take a long time (10-15mins) and then fail with an error that should look like the following:

CdkUbuntuEc2Stack: deploying...
CdkUbuntuEc2Stack: creating CloudFormation changeset...
17:28:01 | UPDATE_FAILED        | AWS::EC2::Instance        | myVm11CC711A
Failed to receive 1 resource signal(s) within the specified duration

It took me a while to figure out what was going wrong and how to solve this problem.

One thing that I eventually did during my investigation was to run cdk synth to see the CloudFormation stack that was being generated and deployed by CDK.

In there I noticed that as part of the user data script, the following command gets executed: /opt/aws/bin/cfn-signal.

Then, with a bit of timed effort, I managed to establish an SSH connection to an instance while it was being provisioned (and before it was destroyed by CDK) and I realized that the default Ubuntu distribution does not have cfn-signal installed.

This makes sense. This is a specific AWS utility that is pre-installed in Amazon Linux, but there is no reason why it should be installed in Ubuntu or other generic Linux distributions.

So now the problem is “How do we install cfn-signal?”

The answer to this question is “by using a custom user data script”!

Installing AWS Linux utilities on an Ubuntu Image using CDK

We can install cfn-signal and other AWS Linux utilities in our virtual machine by creating a custom user data script.

User data scripts run before the init step is executed, so by doing this we should be able to fix our current issue.

After some research and by putting together different resources I managed to get a working script to install the AWS Linux utilities on Ubuntu:

#/usr/bin/env bash
apt-get update -y;
apt-get install -y git awscli ec2-instance-connect;
until git clone https://github.com/aws-quickstart/quickstart-linux-utilities.git; do echo "Retrying"; done;
cd /quickstart-linux-utilities;
source quickstart-cfn-tools.source;
qs_update-os || qs_err;
qs_bootstrap_pip || qs_err;
qs_aws-cfn-bootstrap || qs_err;
mkdir -p /opt/aws/bin;
ln -s /usr/local/bin/cfn-* /opt/aws/bin/;

Note that this script also installs the AWS CLI and ec2-instance-connect. These are not strictly necessary for our use case, but they are nice to have.

In CDK, we can create a user data script with these commands as follows:

const userData = ec2.UserData.forLinux()
userData.addCommands(
  'apt-get update -y',
  'apt-get install -y git awscli ec2-instance-connect',
  'until git clone https://github.com/aws-quickstart/quickstart-linux-utilities.git; do echo "Retrying"; done',
  'cd /quickstart-linux-utilities',
  'source quickstart-cfn-tools.source',
  'qs_update-os || qs_err',
  'qs_bootstrap_pip || qs_err',
  'qs_aws-cfn-bootstrap || qs_err',
  'mkdir -p /opt/aws/bin',
  'ln -s /usr/local/bin/cfn-* /opt/aws/bin/',
)

Then, userData needs to be passed to our machineImage as the third argument:

const machineImage = ec2.MachineImage.fromSSMParameter(
  '/aws/service/canonical/ubuntu/server/focal/stable/current/amd64/hvm/ebs-gp2/ami-id',
  ec2.OperatingSystemType.LINUX,
  userData,
)

With these changes, we should be able to get everything working, but before doing that let’s do one last change: let’s add an output parameter so that, once our stack is deployed, we can see the URL to our web server running nginx.

Adding an output value in CDK

Adding an output value is quite simple with CDK, we just need to instantiate a new cdk.CfnOutput object:

const webVmUrl = new cdk.CfnOutput(this, 'webVmUrl', {
  value: `http://${myVm.instancePublicIp}/`,
  description: 'The URL of our instance',
  exportName: 'webVmUrl',
})

Now if we run cdk deploy, after a few minutes we should see an output like the following:

 ✅  CdkUbuntuEc2Stack

Outputs:
CdkUbuntuEc2Stack.webVmUrl = http://54.154.84.11/

Stack ARN:
arn:aws:cloudformation:eu-west-1:012345678901:stack/CdkUbuntuEc2Stack/22c0faa4-adb4-4724-a4c4-c8ea21b7ac07

If we visit the URL in the output parameter we should see the default nginx welcome page:

It’s working, we made it! 🥳

If you want to see how the full code looks like you can check out the “official” repository for this project.

If you want to clean up your account and destroy the stack created by this project, you can run:

cdk destroy

Conclusion

In this article we learned how to use CDK to provision an Ubuntu-based virtual machine and how to configure it to run the default installation of nginx.

In the process we also learned a bunch of related concepts such as:

what is CDK
how to use it to define an EC2 instance
how to reference an AMI
how to add security groups
how to use user data and init script
how to install the AWS Linux Utilities
…and finally how to create a CDK output argument.

WOW that was quite a journey, and there I thought this would be a short article! 😅

I hope I didn’t bore you to death and that this article was somewhat useful! Let me know what you think in the comments and feel free to connect with me on Twitter for more AWS and cloud chats.

If you are interested in working more with AWS and the cloud, you might also want to check out fourTheorem, a group of business focused technologists that has already helped many companies to get the best out of the cloud (yes, this is where I work 🙃).

See you in the next post!

Bye 👋

Thanks to Eoin Shanaghy for kindly reviewing this article!

Learning Rust through open source and live code reviews

Luciano Mammino — Sun, 11 Oct 2020 19:50:00 GMT

Hello from Luciano and Stefano! 👋

In the last few years we set ourselves onto the Rust learning path. We don’t work with Rust during our daily jobs and we come to be interested in the language for different reasons (more on this later), so we decided to take it easy and invest some of our free time reading books and tutorials, watching videos about Rust and doing occasional coding challenges. Finally after about one year of sporadic studying sessions we felt ready to attempt our first open source Rust project: jwtinfo.

We recently had the pleasure to have our crate reviewed by Tim McNamara of Rust in Action’s fame (one the best Rust books out there in our honest opinion!). The review was carried out live on Tim’s twitch channel and it gave us a lot of cues on things to improve. This article serves as a summary for our experience and memorandum of all the interesting things we learned. If you are learning Rust as well, we hope you will find some of these notes useful!

Our motivation

Before getting into the weeds of what we learned, it probably makes sense for us to give you a little intro on who we are and why we are interested in Rust. If you couldn’t care less about who is writing this, just skip to the next section! 😜

Luciano, considers himself a fullstack cloud developer. His career has been gravitating mostly around building web and cloud-focused products. He has been using mainly high level languages throughout his career (qBasic/VB, Php, JavaScript, Python, Go, Node.js). Node.js is his tool of choice and he is one of the co-authors of the book Node.js Design Patterns (Packt). He is fascinated by Rust because it unveils fundamental topics like memory management and safety which he doesn’t know much about, but he is also interested in using Rust on the web building Rust web servers and frontends with WebAssembly. If you are curious to know more about Luciano, check out the about section of this blog.

Stefano is a long time low-level C/C++ developer who got close to the world of web development in the last few years. Based on his previous experience, Stefano saw in Rust a great potential to write safe and performant code across all the stack.

The project and a quick JWT primer

Just to give you a little bit of context on the project, jwtinfo is a simple command line application (and a library) that allows you to validate and decode a JWTs (JSON Web Tokens) and visualize the content of the “body” (or “payload”) section of the token. If you are not familiar with what JWT is, you can read this 5-minutes article called what’s in a JWT.

jwtinfo is a command line utility that allows you to see what’s inside a JWT. For instance you can run it like this:

jwtinfo eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

And it will print:

{ "sub": "1234567890", "name": "John Doe", "iat": 1516239022 }

Which is essentially the body (or payload) of the token we pass as input to jwtinfo.

Of course, jwtinfo will also tell you if you try to pass a token that is not valid and can’t be parsed correctly, so, in a way, you could also call it a JWT validator, even though, at this time, it does not offer any functionality to validate the token signature against a secret or a public key.

The review

The idea of the code review came up when Tim asked on Twitter what people would be interested to see in a Rust live stream. Luciano took this opportunity to suggest to do some live code review of Rust beginners’ crates. Tim seemed to like the idea and of course at that point we had to volunteer our project 😇

After a couple of weeks Tim proceeded with the live review that you can now watch on YouTube.

Our ask was to get some generic advice around the following topics:

Project structure
Error management
Enums
Exposing a lib and a cli together
Testing
Build process

In hindsight, we reckon that this list might look a bit random and unclear, but nonetheless Tim managed to address most of these points and provided great insights around the concerns we had around these topics.

We are going now to showcase our notes, in no special order and try to provide as much context as possible based on what we learned during the review process and even after it when we decided to go in greater depth on some of the suggested improvements.

Simplified project structure and exposing a CLI and a library

When we started working on this project we just wanted to create a simple command line tool. As we were writing the code, we realized that all the JWT parsing and validation logic could be externalized to its own external module. We didn’t really want to split the project in two, mostly for simplicity, but also because there are many cool JWT crates out there and don’t want to compete or even try to provide feature parity with them. This is a learning project, and as such, we thought it might be nice to expose our JWT parsing layer as a library too. Tim provided 2 interesting pieces of advice. The first one is to simplify the folder structure. It turns out we were following a pattern that was required in Rust 2015 edition, which looks like the following:

src/
├── main.rs # our CLI app
└── jwt # our internal lib for JWT parsing
    ├── mod.rs
    └── test.rs

In Rust 2018 edition it is possible to avoid creating mod.rs files. But why would you want to avoid them in the first place? It turns out that those files tend to be quite anonymous in big projects. If you have many submodules, chances are that you will end up having many tabs open at a given point in time and they would all be saying mod.rs. It will probably take you a while to be able to navigate through them and find the file you were looking for. What we learned is that with Rust 2018 we could simply promote src/jwt/mod.rs to src/jwt.rs. No other code change required: all the imports still work as a charm! So now our code structure looks like this:

src/
├── main.rs
├── jwt.rs
└── jwt/
    └── test.rs

This seems much better, we like flatter structures and descriptive file names! If you are curious to find out more about this there is an official piece of documentation recommending no more mod.rs!

In order to expose the jwt.rs module as part of the public library we finally had to do a couple of minor changes in our cargo.toml and in our main.rs.

Our cargo.toml needed to have new sections indicating how to compile the project for the different targets (CLI application and library). We achieved that by adding the following lines:

[lib]
name = "jwtinfo"
path = "src/main.rs"

[[bin]]
name = "jwtinfo"
path = "src/main.rs"

If you are wondering why lib comes with only one pair of square brackets while bin comes with two, well we had the same question too! We found an answer in the official cargo documentation:

The double-bracket sections like [[bin]] are array-of-table of TOML, which means you can write more than one [[bin]] section to make several executables in your crate. You can only specify one library, so [lib] is a normal TOML table.

Once we applied these changes, everything seemed to work great. Except, when we published the changes on GitHub we started to our CI being polluted by the following warning:

“file found to be present in multiple build targets”

Once again, we had to roll up our sleeves and try to figure out what this actually meant. We found the answer in a Cargo issue where Alex Chricton suggested the following solution:

[…] we’d recommend either using a library or a mod foo pointing to the same location. If you use a shared library crate then you’ll get better compile times though!

So, finally we managed to resolve this issue by moving all our CLI logic into a dedicated cli.rs file and changed the content of the original main.rs into this:

pub mod jwt;

#[macro_use]
extern crate lazy_static;

Note how we also marked the jwt sub-module as public, so that it is actually accessible to whoever is using the library.

Finally our cargo.toml, now looks like the following:

[lib]
name = "jwtinfo"
path = "src/main.rs"

[[bin]]
name = "jwtinfo"
path = "src/cli.rs"

And our files structure looks like this:

src/
├── cli.rs
├── jwt.rs
├── main.rs
└── jwt/
    └── test.rs

And that annoying warning disappeared. Plus, we are probably getting slightly faster compile times now! ⚡️

Document all the things

One thing that we completely neglected so far was documentation. Rust has an amazing built-in utility to be able to document the API of crates and even to publish them online. There is even an official piece of documentation that gives guidance on how to write useful documentation comments.

If you just want to get started with writing docs for your crates, probably the easiest thing to do is to check out the “Rust by example” page about documentation.

As you start adding documentation comments to your crate, you can simply run cargo doc --open to compile the website with the documentation locally and open it with your browser. This is extremely convenient to make sure that, once you publish the crate, the documentation online will look as expected.

Few days after the review we rolled up our sleeves and started to document our library. If you are curious to see what it took to go from 0 to 100, this is our PR that added documentation to the crate: lmammino/jwtinfo/pull/22.

One of the most amazing thing was that, once we published a new version of the crate on crates.io, after few minutes we could navigate our crate doc online at docs.rs/jwtinfo:

Another thing that we loved about writing documentation in Rust is the ability to execute code examples to make sure they are actually correct. Just to show a real example, this is a piece of documentation in our crate:

//! To parse a given JWT as a string:
//!
//! ```rust
//! use jwtinfo::{jwt};
//!
//! let token_str = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c";
//! match jwt::parse(token_str) {
//!   Ok(token) => {
//!     // do something with token
//!     assert_eq!(token.header.to_string(), "{\"alg\":\"HS256\",\"typ\":\"JWT\"}");
//!     assert_eq!(token.body.to_string(), "{\"iat\":1516239022,\"name\":\"John Doe\",\"sub\":\"1234567890\"}");
//!   }
//!   Err(e) => panic!(e)
//! }
//! ```

When we run our test suite with cargo test, Cargo will also extrapolate this example here from the docs and execute it. This will help us make sure that our documentation is actually correct and that it doesn’t go stale over time.

If you have a big test suite and you just want to run your doc tests you can simply do that with cargo test --doc.

Finally, another small trick we discovered is that you can hide specific functions or types from the generated documentation (in case those are private details you don’t want to expose to the crate users). Hiding documentation members is as simple as adding the following annotation:

#[doc(hidden)]

For instance, in our crate we have:

#[doc(hidden)]
fn parse_base64_string(s: &str) -> Result<String, JWTParseError> {
    let s = base64::decode_config(s, *BASE64_CONFIG)?;
    let s = str::from_utf8(&s)?;
    Ok(s.to_string())
}

This is convenient because we don’t want any user of our crate to bother with this function which is just an implementation detail that we might decide to change in the future.

Convert a string to anything

Another great suggestion from Tim to improve the ergonomics of the JWT parsing library was to implement the str::FromStr trait for the Token type.

The FromStr trait is a special Rust trait that allows to convert a string into any type that implements the trait.

This trait is already implemented in some standard type like int32 and is what allows us to convert a string to an int32 as follows:

let value = "1987";
let result = value.parse::<i32>().unwrap();

By the way, if you find the syntax function::<type>() weird, don’t worry, we found it a bit “unfriendly” too. Once we discovered it is called “turbofish” we couldn’t help but to fall in love with it. You can learn more about it here: Where to put the turbofish.

Don’t worry, if you still don’t like the turbofish syntax, you can rewrite the code above using type hints as follows:

let value = "1987";
let result : i32 = value.parse().unwrap();

The official docs as a great example that shows how the str::FromStr trait can be very valuable to implement in your custom types. We are going to try to take this example and present it here in a slightly simplified fashion.

Imagine you have defined a Point type as follows:

#[derive(Debug, PartialEq)]
struct Point {
    x: i32,
    y: i32
}

Now imagine you are processing data from a text source, let’s say a file, and in there points are encoded in a string format such as 17,22. Wouldn’t it be nice to be able to easily turn this format of strings into Point values?

Well, let’s enable this using the str::FromStr trait:

use std::str::FromStr;
use std::num::ParseIntError;

#[derive(Debug)]
struct Point {
    x: i32,
    y: i32
}

impl FromStr for Point {
    type Err = ParseIntError;

    fn from_str(s: &str) -> Result<Self, Self::Err> {
        let mut parts = s.split(",");
        let x_str = parts.next().unwrap_or("0");
        let y_str = parts.next().unwrap_or("0");
        let x = x_str.parse::<i32>()?;
        let y = y_str.parse::<i32>()?;
        Ok(Point { x, y })
    }
}

fn main() {
    let value = "17,22";
    let point = value.parse::<Point>().unwrap();
    println!("{:?}", point); // Point { x: 17, y: 22 }
}

As you can see from the example above, the str::FromStr trait has the following spec:

impl FromStr for YourType {
    type Err = YourErrorType;

    fn from_str(s: &str) -> Result<Self, Self::Err> {
        // process the string `s` and return a Result
        // either a Err(YourErrorType) or
        Ok(YourType{})
    }
}

In our project it was rather trivial to implement this trait, because our parse function matches quite closely the signature of the from_str method defined in the str::FromStr trait:

impl FromStr for Token {
    type Err = JWTParsePartError;

    fn from_str(s: &str) -> Result<Self, Self::Err> {
        parse(s)
    }
}

This allows for the following syntax:

let token = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c".parse::<jwt::Token>().unwrap();

or, if you still hate the poor turbofish:

let token: jwt::Token = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c".parse().unwrap();

Accept any type that can be converted to string

For better or for worse, Rust is famous for having a number of different types to represent strings. The most famous are String and &str but from time to time we have seen even other types like CStr or OsStr.

During the review process, Tim recommended making our public API more flexible by supporting “every type that can be converted to a string”.

The way we can achieve that is by leveraging the std::convert::AsRef trait.

The official documentation starts with a “not very friendly” opening (at least for Rust noobs like ourselves), but then it provides a great example that makes quite obvious how to use this trait to support any type of string:

fn is_hello<T: AsRef<str>>(s: T) {
   assert_eq!("hello", s.as_ref());
}

let s = "hello";
is_hello(s);

let s = "hello".to_string();
is_hello(s);

Essentially, whenever you want to accept “anything that can be converted to a string” you can use the following recipe:

Make your function accept an argument x of generic type T
Define T so that it must implement the trait AsRef<str>
In your function body get the actual str value from x using x.as_ref()

As you can see from the example above, with this recipe, we can pass both &str and String values to the is_hello function.

Applying this pattern to our code was quite straightforward. Also, since our public surface is quite small, we only needed to change the jwt::parse function as follows:

pub fn parse<T: AsRef<str>>(token: T) -> Result<Token, JWTParsePartError> {
    let mut parts = token.as_ref().split('.');
    // ...
}

Remove code duplication

One of the things that amazed us when we started playing with Rust was to discover that Rust is primarily an expression language.

In practice, this means that most syntax elements, including if-else and match, once evaluated will produce a value that can be carried over to other expressions or assigned to a variable.

Coming from languages that don’t have this capability, we easily forget to take advantage of this characteristic of Rust and we end up writing code that contains duplication or that is more verbose than necessary.

Tim spotted a few places where we could get rid of annoying code duplication by simply using match and if-else in a different way. This made us go through the code base of our crate and find other places where we could improve readability and make our code more idiomatic.

Let’s show some examples:

if matches.is_present("header") {
    println!("{}", jwt_token.header.to_string());
} else {
    println!("{}", jwt_token.body.to_string());
}

This code has clearly some repetition. It is not terrible, but it seems unnecessary to have to repeat the println!() macro invocation and the to_string() call.

We ended up refactoring this as follows:

let part = if matches.is_present("header") {
    jwt_token.header
} else {
    jwt_token.body
};
println!("{}", part.to_string());

Once you know that if-else expressions will produce a value (depending on which branch will be selected at runtime), this alternative looks more readable. Also this refactoring makes quite clear that this code has essentially 2 parts: select a part, then print it on the standard output.

Another interesting example in our code base involved the match expression. This was the code before the refactoring:

impl fmt::Display for JWTParseError {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        match self {
            JWTParseError::MissingSection() => write!(f, "{}", "Missing token section".to_string()),
            JWTParseError::InvalidUtf8(e) => write!(f, "UTF8 error, {}", e),
            JWTParseError::InvalidBase64(e) => write!(f, "Base64 error, {}", e),
            JWTParseError::InvalidJSON(e) => write!(f, "JSON error, {}", e),
        }
    }
}

Applying the same principle as before we refactored the code above to this:

impl fmt::Display for JWTParseError {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        let message = match self {
            JWTParseError::MissingSection() => "Missing token section".to_string(),
            JWTParseError::InvalidUtf8(e) => format!("UTF8 error, {}", e),
            JWTParseError::InvalidBase64(e) => format!("Base64 error, {}", e),
            JWTParseError::InvalidJSON(e) => format!("JSON error, {}", e),
        };
        write!(f, "{}", message)
    }
}

Again, the trick here is to assign the result of the match expression to a variable (message in this particular case) and then we can re-use that variable in the next step.

Another cool thing that we discovered is that, by using clippy as a linter, it will help you out to spot some of these improvements.

For instance, the first time we tried to improve the readability of the if-else expression previously mentioned we ended up using a match expression unnecessarily:

let part = match matches.is_present("header") {
    true => jwt_token.header,
    false => jwt_token.body,
};
println!("{}", part.to_string());

When we run clippy on this code, it did trigger the match_bool rule and that it is suggested to refactor this with an if-else expression:

let part = if matches.is_present("header") {
    jwt_token.header
} else {
    jwt_token.body
};

Which is indeed more readable. Another lesson learned: trust linters!

In short, these types of change helped us to reduce the amount of inline composition and to make the code more sequential and easy to follow.

Improve tests for invalid strings

Another thing that Tim recommended us to do was to improve our coverage on testing different types of input. He tried to run our JWT parser against strings with non-utf8 characters and strings containing null bytes to see whether our code would handle that correctly (returning an error) or whether it would just panic.

The idea is to try to avoid panics at all costs and make sure all possible errors are handled and propagated correctly.

This pushed us to increase our test coverage adding a few more tests as you can see in this PR.

Another interesting recommendation was to use something like quickcheck to do fuzzy testing on our input. Although this seems like an extremely interesting thing to do, we haven’t got time yet to play with this idea.

Shell scripts deserve some love too

As a last point, Tim had a look at our cross-system installation script and immediately noticed our poor use of Bash scripting.

The main advice here was to use some shell linter like Shellcheck. Tim himself, ran the script against the linter by copy pasting the code on the web page and submitted a PR with some improvements.

Shellcheck can be used as a command line tool too, so it would be nice in feature release to integrate in our automated tests.

Conclusion

This concludes our recap article. We don’t have any special rule of thumb except that the Rust ride was quite fun for us so far. It has been a bumpy road but thanks to the amazing open source community and people like Tim, the road to learn Rust can become much smoother.

Our final word of advice is that, if you are learning Rust, or any other technology for what matters, don’t be afraid to start public open source projects and ask for help and reviews. Even better if you share this journey with some friends.

At the end of the day, all that matters is learning new things and having fun!

We take one more opportunity to thank Tim for the great support here and, if you like Rust, we do recommend you to follow him on Twitter, Twitch and Youtube.

Take care! 🙌

AWS Solution Architect Professional exam, my notes and tips

Luciano Mammino — Tue, 26 Apr 2022 18:20:00 GMT

Hello everyone, I am really happy to share with you that I recently got my AWS Solution Architect Professional certification (SAP-C01) 🎉

In this article I talk about my experience with the exam, how I prepared for it and share a few tips that can, hopefully, help you if you are preparing for the exam.

Who is the AWS Solution Architect Professional certification for

I’d recommend you embarking on this challenge only if you already have a few years of AWS experience and you already managed to explore a few different services and different types of architectures.

This certification is for those who want to have a comprehensive view of what’s possible with AWS and that are aiming to be able to advise about the best architectures for specific problems.

In my view, this certification is for you if you are working as a AWS cloud consultant (or you want to be one) or you are aiming to get an architect role at a company with a significant footprint on AWS.

If you have already taken other AWS certifications that would also be very beneficial for you. This is probably the hardest of AWS certification so having experienced the whole process with other AWS certifications will help.

In my case, I took the AWS Solution Architect associate certification 3 years ago. If you have less than 2 years of experience with AWS, I’d recommend taking this other certification first. You can read my notes and tips about that particular experience!

Exam Structure

The AWS Solution Architect is probably the thoughest exams I had to do to get an IT certification.

The exam is not easy, it requires quite a lot of focus.

There are tons of questions, every question is very verbose and getting the right answer while trying to manage your time, can be challenging. On top of that, you need a very broad knowledge about AWS services and different types of architectural concerns.

The exam is a multiple answer question. In it’s current incarnation (2022) there are 75 questions and you have 3 hours to complete it. This gives you about 2 minutes per question.

If you think that’s plenty of time, let’s have a look at one of the example questions. Start a time, NOW! ⏱

A company has two AWS accounts: one for production workloads and one for development workloads. Creating and managing these workloads are a development team and an operations team. The company needs a security strategy that meets the following requirements:

Developers need to create and delete development application infrastructure.

Operators need to create and delete both development and production application infrastructure.

Developers should have no access to production infrastructure.

All users should have a single set of AWS credentials.

What strategy meets these requirements?

A) In the development account:

Create a development IAM group with the ability to create and delete application infrastructure.

Create an IAM user for each operator and developer and assign them to the development group. In the production account:

Create an operations IAM group with the ability to create and delete application infrastructure.

Create an IAM user for each operator and assign them to the operations group.

B) In the development account:

Create a development IAM group with the ability to create and delete application infrastructure.

Create an IAM user for each developer and assign them to the development group.

Create an IAM user for each operator and assign them to the development group and the operations group in the production account. In the production account:

Create an operations IAM group with the ability to create and delete application infrastructure.

C) In the development account:

Create a shared IAM role with the ability to create and delete application infrastructure in the production account.

Create a development IAM group with the ability to create and delete application infrastructure.

Create an operations IAM group with the ability to assume the shared role.

Create an IAM user for each developer and assign them to the development group.

Create an IAM user for each operator and assign them to the development group and the operations group.

D) In the development account:

Create a development IAM group with the ability to create and delete application infrastructure.

Create an operations IAM group with the ability to assume the shared role in the production account.

Create an IAM user for each developer and assign them to the development group.

Create an IAM user for each operator and assign them to the development group and the operations group. In the production account:

Create a shared IAM role with the ability to create and delete application infrastructure.

Add the development account to the trust policy for the shared role.

OK, time off! ⏱

How long did it take you to read and make sense of this literal wall of text?! Do you already know what’s the right answer? 😅

I am probably slow at reading, but it took me about 2 minutes just to read this!

Some questions are even trickier because you’ll need to pick more than one answer!

The point is that you’ll need to be able to decide on your answer fast. And you will also need to be able to keep your focus for 3 hours straight!

🔥 First tip: if English is not your primary language you can request extra time when booking your exam, or do the exam in other languages (French, German, Italian, Japanese, Korean, Portuguese, Simplified Chinese, or Spanish).

I didn’t think I needed this and I went for the English version without extra time. I had to rush through the last 5 questions, so, in retrospective, it was a mistake not to take advantage of this!

I’ll share a few more tips to maximise your chances of succes later in this article!

Meanwhile, if you are curious to see more example questions, there is an official set of example questions for the Solution Architect Professional exam provided by AWS.

Exam topics and required knowledge

There are 5 domains of knowledge in which the various questions are grouped:

Domain 1: Design for Organizational Complexity (12.5% of the exam)
Domain 2: Design for New Solutions (31% of the exam)
Domain 3: Migration Planning (15% of the exam)
Domain 4: Cost Control (12.5% of the exam)
Domain 5: Continuous Improvement (29% of the exam)

In terms of overall knowledge, these are some of the topics that AWS recommends you to be comfortable with before taking the exam:

Familiarity with AWS CLI and APIs.
Be comfortable with CloudFormation.
Be able to understand Billing and to use the web console.
Know how to use a scripting language.
Know enough about running applications on Windows and Linux systems and how to keep them up to date.
Ability to provide guidance on the architectural design across multiple applications and projects of the enterprise as well as an ability to map business objectives to application/architecture requirements.
Ability to evaluate cloud application requirements and make architectural recommendations for implementation, deployment, and provisioning applications on AWS.
Ability to design a hybrid architecture using key AWS technologies (e.g., VPN, AWS Direct Connect) as well as a continuous integration and deployment process.
Be able to explain and apply the five pillars of the AWS Well-Architected Framework.
Architect a continuous integration/continuous delivery (CI/CD) process.

Yes, it’s quite a big surface of knowledge and this is probably not even a comprehensive list.

If you want a bit more detail on what kind of topics and services you need to be comfortable with you can Download the official Solution Architect Professional exam guide.

Preparation

Ok, if you are still up for this, let’s see how you can prepare yourself to smash this challenging exam! 💪

Well… unfortunately, I don’t think there is a bullet-proof method to pass this exam. This exam takes a good mixture of expertise, focus and (of course) a bit of luck. You should do your best to arrive prepared at the exam, but I also recommend you account for a good chance of failure. It’s not a shame to fail the exam and you can always try it again later.

So, how did I prepare myself?

I have been lucky to have been exposed quite a lot to various AWS projects in the last 5-6 years. This has given me a good breadth of knowledge and confidence. I have been working in a mixture of roles shifting from software engineering to architecture. I have architected, built and deployed serverless solutions, but also more traditional solutions based on EC2 instances and containers. I also had numerous chances to explore event driven architectures and use services like SQS, SNS, EventBridge and Kinesis.

With that being said, I don’t think this amount of practical experience was nearly enough to walk into the exam!

But how did I know that? Well, I didn’t… I was actually quite confident about my skills at first! Until I got knocked out by the first exam simulation…

Exam simulations

The first thing you should really do to asses where you stand is to take an exam simulation. An exam simulation will present you 75 questions similar to the ones you might find during the real exam and you’ll have 3 hours to answer them.

Those simulations are available on a number of platforms. I have tried 2 of them:

AWS Certified Solutions Architect Professional Practice Exam on Udemy, by Jon Bonso (Tutorials Dojo).
AWS Certified Solutions Architect - Professional 2020 on A Cloud Guru, by Scott Pletcher.

I really liked the Udemy ones for several reasons.

First of all, you can pause your exam. I know this is not really realistic, but it’s a great way to allow you to practice even when you don’t have a block of 3 full hours available to you.

Secondly, at the end of a simulation, you get an overview of all the questions. For every question you get a quite lengthy explaination about how you should have interpreted the question and a walkthrough over all the possible answer. This can teach you how to analyse questions and answers and it will help you train the right muscles to spot correct answers and recognize and avoid wrong answers (distractors).

Once you complete a simulation you can assess where you stand against the different domains of the exam. Where you weakness and strengths lies and where you need to study and practice more.

The Udemy course gives you four different variations of the exam, so you get a total of 300 questions to practice with!

Study material

Once you have identified your gaps, you need to study!

I found A Cloud Guru to be one of the best resources out there. The AWS Certified Solutions Architect Professional course by Scott Pletcher is very broad and informative. Scott keeps it quite entertaining with occasional stories and jokes and there are quite a few interactive labs for you to practice what you learn through the course.

Doing the full course could be quite time consuming (almost 30 hours of video content!), but if you are in a rush you can only pick the chapters that you need the most. That table of contents is really well structured and you can easily jump around.

One caveat is that what A Cloud Guru gives you is not necessarily enough. In some cases the videos and the labs only scratch the surface of certain topics and you might still struggle to answer some of the question.

In those cases, Scott is very upfront and gives you a bunch of additional study material published for free directly from AWS. Make sure not to ignore these papers because they can make a big difference in your understanding of AWS and in getting you ready for the exam!

Another great resource is AWS Skill Builders, which has an entire section on Exam Readiness: AWS Certified Solutions Architect – Professional.

I only skimmed through this one, so I can’t vouch 100% for this, but it seems like a great FREE alternative to what A Cloud Guru can offer.

Check out also the FAQ and the White Papers published on the official page for the AWS Certified Solutions Architect - Professional certification.

Recommended strategy

Here’s my very personal recommendation on how to prepare and approach the exam.

Before the exam

Do a couple of exam simulations on Udemy.
Don’t worry too much about your score (I failed almost all of them!), but focus on the questions you failed. Read the explanation and make sure to understand if you need to research more about the topic.
Compile a list of all the topics you need to research more about (your gaps).
Find study material (A Cloud Guru, YouTube, official AWS material, etc.) and try to fill the gaps.
Do more exam simulations.
Try to do a couple more of exam simulations without pausing the exam.
Don’t wait for your simulation score to be over 75% before you try the real exam, just make sure your score has improved and that you feel much more confident.

During the exam

Read the question carefully and try to understand what are the main keywords and what’s the context. After a few simulations you should know what to expect and you should be able to skim through the question quickly.
But if you feel you haven’t fully understood the question take the time to read it again.
Manage your time. Ideally you shouldn’t be spending more than 2 minutes per question, but sometimes there are question that require you more time. Don’t stress too much out if you are taking to long for a given question, there will be questions that you’ll be able to answer in less than 2 minutes and those will help you to balance things out. Make sure though not to spend too much time on a given question. If it’s taking you more than 4 minutes, you should probably skip it (or even better give a temporary answer) and come back to it later.
You can flag questions for review. This means that flagged questions will be highlighted and it will be easier for you to get back to them if you have extra time and want to review the answer.
A lot of questions don’t have absolutely objective answers. Actually most questions are arguable (in my opinion). In those cases try to go by exclusion and find reasons not to pick particular answers. Try also to wear the hat of AWS: ask yourself what would AWS itself most likely recommend?

After the exam

If you passed it… well, good job! Time to brag with your friends and co-workers and celebrate!
If you didn’t pass it… no worries! You have probably learned a lot anyway. Go back to more simulations and study material and try again in a few months! Next time will certainly go better!

My Study notes

Here’s a list of notes about topics that I didn’t really know about and I had to research a little bit. This is no way comprehensive but maybe it can help you to find gaps in your knowledge. I basically build this list out of answers I failed during the simulations, so these were basically part of my knowledge gaps!

Alexa for business

A service that allows you to provision Alexa Echo devices for office use.
Can be integrated with office resources (e.g. calendar, room booking systems, shared files, video conference tools, etc.).
Then Echo devices can be used to answer business questions, book meetings or other resources.

Amazon Connect

Cloud based contact center (system to allow communication between companies and the general public, e.g. support call centers)
Main features: Provides skill based routing, Automatic call distribution, Call recording, Reporting
Allows to design automated “Contact flows” that can direct the user to the right operator (“press X if you need to do Y”)
The contact flows can be integrated with lambda to customise the flow programmatically decisions for the workflow (e.g. fetch a reservation for the caller and decide the next step based on the state of the reservation)
Provides an IVR (Interactive Voice Response) and records transcripts for every conversation / flow

Amazon Kinesis Video Stream

Different service from Kinesis Data Stream (it goes into an entirely different category)
Cameras/microphones and other devices can use Kinesis video stream libraries (producer library) to publish data (real time or buffered)
Data can be consumed in real time or buffered (saved durably in S3) - Real time vs batch approach
Rekognition can be used to extract metadata from the stream

AWS Networking notes

Network ACL (NACL): firewall-like rules for entire vpc/subnets to enable/block traffic (stateless. Outgoing traffic for ephemeral ports needs to be specified explicitly )
Security Groups: per instance firewall-like rules to enable disable traffic (stateful. E.g. if you allow incoming port 80, outgoing port for traffic coming through port 80 will be allowed as well)
You can give a public IP to a Load Balancer either through a Network load balancer with Elastic IP or an Application Load Balancer with Gloabal Accelerator

AWS Shield

Service for DDoS attack protection.
There is an Shield Advanced option. In addition to the network and transport layer protections that come with Standard, AWS Shield Advanced provides additional detection and mitigation against large and sophisticated DDoS attacks, near real-time visibility into attacks, and integration with AWS WAF.

CloudFront

CloudFront signed cookies allow you to control who can access your content when you don’t want to change your current URLs or when you want to provide access to multiple restricted files, for example, all of the files in the subscribers’ area of a website. This topic explains the considerations when using signed cookies and describes how to set signed cookies using canned and custom policies.

Direct Connect

Direct Connect can be used to connect an Amazon VPC to an On premise network
Use Direct Connect link between the VPC and the on premise network
Use a network device in the on-premise network that supports BGP and MD5 Authentication (needed to establish a Direct Connect link from your data center to your VPC)
Direct Connect connections consist of a single connection between your network and AWS with no inherent redundancy.
Traffic coming from on-prem via a Direct Connect connect is restricted from internet access.
You must create a virtual interface (public, private or transit) in order to begin using your AWS Direct Connect connection.

IAM

Managed Policies

AWS managed policies don’t grant least privilege permissions. You must consider the security risk of granting your principals more permissions than they need to do their job.

You can attach AWS managed policies, including job functions, to any IAM identity. To switch to least privilege permissions, you can run AWS Identity and Access Management Access Analyzer to monitor principals with AWS managed policies. After learning which permissions they are using, then you can write a custom policy or generate a policy with only the required permissions for your team. This is less secure, but provides more flexibility as you learn how your team is using AWS.

AWS managed policies for job functions are designed to closely align to common job functions in the IT industry. You can use these policies to grant the permissions needed to carry out the tasks expected of someone in a specific job function. These policies consolidate permissions for many services into a single policy that’s easier to work with than having permissions scattered across many policies.

Most common managed policies

AdministratorAccess his user has full access and can delegate permissions to every service and resource in AWS.
Billing This user needs to view billing information, set up payments, and authorize payments. The user can monitor the costs accumulated for the entire AWS service.
DatabaseAdministrator This user sets up, configures, and maintains databases in the AWS Cloud.
DataScientist This user runs Hadoop jobs and queries. The user also accesses and analyzes information for data analytics and business intelligence.
PowerUserAccessThis user performs application development tasks and can create and configure resources and services that support AWS aware application development.
NetworkAdministrator This user is tasked with setting up and maintaining AWS network resources.
ReadOnlyAccess This user requires read-only access to every resource in an AWS account.
SecurityAudit This user monitors accounts for compliance with security requirements. This user can access logs and events to investigate potential security breaches or potential malicious activity.
SupportUser This user contacts AWS Support, creates support cases, and views the status of existing cases.
SystemAdministrator This user sets up and maintains resources for development operations.
ViewOnlyAccess This user can view a list of AWS resources and basic metadata in the account across all services. The user cannot read resource content or metadata that goes beyond the quota and list information for resources.

Lex

Conversation interface service
ASR (Automatic Speech Recognition) + NLU (Natural Language Understanding)
Can understand voice and text (voice automation + chatbots)
Based on intents (entities that define actions that the user might want to perform)
An intent has “sample utterances” (example phrases that can be used to express the intent)
An intent has “slots” (particular type of keywords that represent specific concepts in a phrase)
A “slot” can have a type (e.g. color, dates, day of weeks, city, country, etc.)
Slots can be used as a template variable in utterances (“Your trip is scheduled for {date}”)
The response is generated statically (predefined phrases) or dinamically (through a lambda)
Can be combined with comprehend for sentiment analysis
It understands audio (Amazon Transcribe: speech to text)
It can generate audio responses (Amazon Polly: text to speech)

Macie

Security tool that can be used to monitor events (Intelligent cloud protection)
It can scan data uploaded to S3 and recognize sensitive information (e.g. API Keys, but also passport numbers or address)
It can be used to detect anomalies in AWS API usage (e.g. it can detect if data is likely to have been stolen)
It can be integrated with CloudWatch and SNS to alarm on potential threats
It offers a dashboard that allows you to visualise the detected threats, but also all the AWS API calls (grouped by type but you can also drill down)
Supports Lucene-style queries to search through the data

RPO vs RTO

Metrics for establishing and measuring good practices in backup and recovery of data (disaster recovery)
RPO (Recovery Point Objective): Time between a good dataset (in the backup) and a disaster event.
RTO (Recovery Time Objective): Point in which the data is restored and operation continues.
If RTO is 30 minutes that means that recovering from an incident should not take more than 30 minutes. You can also see that as the maximum time that a company is willing to keep a service unavailable
RPO: before an incident occurs
RTO: after an event occurs
Reducing RPO and RTO have cost implications because it means more frequent backups and faster mechanism to recover from failure

S3

Objects that are stored in your bucket before you set the versioning state have a version ID of null.

S3 Select

Allows to query ONE s3 object and extract data from it without having to download the entire file locally
It uses a subset of standard SQL for queries
Supports CSV, JSON (also gzipped or bzipped) and parquet (non compressed)
Supports files encrypted at rest
Can query selectively part of a file (e.g. only the first 100Mb)
It can be cheaper to get data out of S3 this way (not transferring large files outside AWS)
It charges based on the amount of data scanned per query and the amount of data returned

Snowball

If copying files is slow: this is due to encryption overhead when copying files to the Snowball Edge device. Open multiple sessions to the Snowball Edge device and initiate parallel copy jobs to improve the overall copying throughput.

Storage Gateway

In the cached mode, your primary data is written to S3, while retaining your frequently accessed data locally in a cache for low-latency access.
In the stored mode, your primary data is stored locally and your entire dataset is available for low-latency access while asynchronously backed up to AWS.

VPC

NAT Gateway

NAT Gateway do not support IPv6
- same for NAT instances
- If you need IPv6 traffic use an egress-only internet Gateway
You’d generally prefer a NAT Gataway over a NAT instance because
- Gateway is a managed service
- Highly available
- Scales automatically
But you might prefer a NAT instance when:
- Need to save money for a small workload that doesn’t require High Availability
- You need to fine tune security groups
- You want the ability to detach your Elastic IP

Allow a VPC to resolve using on-premise DNS

Configure a DHCP Option Set to issue your on-prem DNS IP to VPC clients.

Egress-Only Internet Gateway

The purpose of an “Egress-Only Internet Gateway” is to allow IPv6 based traffic within a VPC to access the Internet, whilst denying any Internet based resources the possibility of initiating a connection back into the VPC.

Allows VPC based IPv6 traffic to communicate to the Internet
Prevents IPv6 based Internet resources initiating a connection into a VPC

Default DNS address for DHCP

For default settings the DHCP address of a subnet is always the base CIDR of the subnet plus 2.

E.g. if the base is 10.0.0.0/16 the DHCP address is 10.0.0.2

WAF

Web Application Firewall
Level 7 firewall that can be used to block web traffic
Can be used with ALB and CloudFront distributions

Tips

Here are some things that I picked up while doing the practice exams and that can help to analyse questions and identify the correct answer(s):

“Scalable and cost effective” storage is always a hint to pick DynamoDB, not RDS! Aurora could also be an option, but only if the question explicitly states “Relational Database”.
Everytime you encounter “Slow connection” and “migration from on-premise to the cloud” expect Snowball to be one of the answers.
Never pick reserved instances with workloads that are limited in time.
Oracle RAC is not supported by RDS: all questions mentioning RAC will probably require you to spin up Oracle RAC in a dedicated EC2 instance.
When TCO (Total Cost of Ownership) is mentioned, the answer is generally requiring you to use AWS Application Discovery Service.
“Dedicated” vs “Reserved” instances. Sometimes used in the wrong way in the answers. Make sure you understand the difference so you can easily spot choices that can be easily removed. Dedicated: instances that are not shared with other AWS customers; Reserved: instances that you reserved (paid for) in advance.
“Memcache” vs “Redis” (ElastiCache), always pick Redis, except when it’s explicitly requested that the caching layer needs to be multithreaded.

Conclusion

I want to give a huge shout out to fourTheorem for supporting me in this endeavor and giving me time to study and access to paid material! ❤️ Also, thanks a million to my colleague Marin for reviewing this article and suggesting a few improvements! 🙌

This concludes what I had to share. Please do let me know if you found this material interesting and useful.

Give me a nudge on Twitter (my DMs are ope) if you are preparing for the exam and there are topics that you want to discuss or if you have questions you want to ask. Chances are I won’t have great answers for you, but we can still have fun chatting about AWS and cloud topic! 🙃

I wish you the very best for your exam!

See you on the next post!

2021 - A year in review

Luciano Mammino — Wed, 02 Feb 2022 20:25:00 GMT

A post where I reflect on my professional life in 2021 as a look ahead for what to expect in 2022.

As usual, let me tell you that I write this kind of posts for myself and I don’t expect other people to be able to read this and be engaged. So, if you really feel like you want to venture forward and have a peek on my 2021, well, I am flattered… but make sure to grab a coffee, because this might get a little (just a little…) boring!

Yes, this GIF is probably the least boring element of the whole article! 😍

Joining fourTheorem

The biggest highlight of my 2021 is that in February I joined fourTheorem as a Senior Architect.

If you never heard about fourTheorem, here’s what you need to know. It’s a great service company founded by Peter Elger, Fiona McKenna and Eoin Shanaghy.

The company, while still relatively new and small, has a large amount of expertise in serverless, platform modernisation and AI as a service. Most importantly, fourTheorem has a huge amount of experience with AWS. We are AWS Consulting Partners and we have been helping small and big customers to transition to the cloud and get the best out of it!

I have to admit I was initially skeptical about being able to work with a company that is not focused on delivering one single product but that is mainly helping many other companies to do that. In the past I rarely had a great experience when being on the other side and dealing with other service companies.

But I was immediately reassured by the fact that Eoin and Peter are not just two amazingly good technical founders, but they also have an impressive track record working with companies of all sizes: from small startups to big corporations. Together with Fiona, the founding trio has a very inspiring vision for what they want the company to become.

I immediately had the feeling that they wanted to build something very different from the other service companies out there. This feeling became apparent pretty quickly just after the first week, when I saw how they managed to embed their team into the customer’s team and work together in all the phases of the product. FourTheorem can literally be an extension of a team, bringing not just people and technology skills, but also a very practical and agile approach to software delivery.

And after almost a year, I cannot hide the fact that I have never been happier in a technical position. I can work in an environment with fantastic colleagues, an amazing culture, in a very supportive environment where I am learning a ton. Most importantly, I feel like there is a lot of potential and that I can have an impact.

Finally I get a significant amount of “work time” to invest in content creation, open source and to study (new tech, certifications, etc.). This is something that I consider integral part of my professional career path and it’s great to see that it is valued and supported!

Mandatory disclaimer: we are always hiring, so if you are interested feel free to reach out to me on Twitter or LinkedIn.

Also… if you are looking for help for your cloud migration or to improve your AWS deployments, don’t be shy, let’s have a coffee-chat! 😉

Be not afraid of discomfort. If you can’t put yourself in a situation where you are uncomfortable, then you will never grow. You will never change. You’ll never learn.

— Jason Reynolds

Becoming an MVP

In 2021 I was awarded the title of Most Valuable Professional (MVP) for Developer Technologies by Microsoft:

The MVP program awards exceptional community leadership across the world of tech.

Apparently, to become an MVP, you need to do the following 4 things:

Demonstrate community leadership and influence
Be a technical expert
Be a great advocate for the community
Contribute to the success of our products

If nothing else, this makes me think that the community is getting some value from the work I am doing and this can only push me to do more!

I am really glad that the amazing Francesco Sciuti (who nominated me) thinks that I have done such a good job to deserve the award and thanks Microsoft for agreeing with him! ❤️

One funny thing is that, during the due diligence phase, Microsoft makes you fill a big form to list all the things you think had the most impact on the community. Having written posts like this in the last few years helped me a lot to come up with a consistent list. Maybe it wasn’t such a bad idea to write these posts after all…

If you are curious, you can also check out my profile on the MVP network.

It’s not like after winning an award, your job is done and you can relax. You still have to work hard.

— Diljit Dosanjh

Public speaking

In 2021 I had the pleasure (and the immense luck) to be able to do quite a lot of publick speaking.

It’s hard to pick a favourite but, being such a fan of Node.js, there’s one that I am particularly proud about: NodeConf Remote!

Here’s the full list of talks, workshops and interviews that I delievered in 2021:

An opinionated intro to Node.js - Devrupt Hospitality Hackaton (Slides)
From Node.js to Design Patterns - Codemotion DevCast (Slides)
Let’s play with Node.js streams (workshop) - CityJS (Slides)
Let’s play with Node.js streams (workshop) - Codemotion (Slides)
A podcast about Node.js with Vishwas Narayan (Slides)
My first Rust crate: jwtinfo (lightning talk) - Rust Dublin Meetup (Slides, Video)
Chats and career advice with Superheroes Valley
Live stream with Fabio Biondi - Node.js Streams (Video)
Finding a lost song with Node.js & async iterators - Node.js Dublin meetup (Slides)
Interview with Gitbar podcast (Podcast Episode)
Javascript: repetita iuvant - Codemotion DevCast (Slides)
Teach Kelvin Your Thing: JavaScript Async iterators (Video)
An intro to the JAMStack & Eleventy - Codemotion (Slides, Video)
Reiterating JavaScript and Node.js iterators - RomaJS (Slides, Video)
A chat with Luca Brunello (Video)
AWS Observability without the pain (Slides)
Finding a lost song with Node.js & async iterators - Sailsconf (Slides, Video)
How to send gzipped requests with boto3 (Slides)
Quality questions with the NoobDev podcast (Podcast Episode)
Finding a lost song with Node.js & async iterators - EnterJS (Slides)
How to Train a Computer Vision Model - Sparkd.ai on Twitch (Slides, Video)
Finding a lost song with Node.js & async iterators - NodeConf Remote (Slides, Video)
Semplificare l’observability per progetti Serverless su AWS - Cloud Day conference (Slides)
AWS Observability made simple - AWS Community Day Ireland (Slides)
Authors and Architects (Panel) - AWS Community Day Ireland
Serverless, Devops & Developer Tools (Panel) - AWS Community Day Ireland
Node.js, Rust & Cloud Architectures (interview) - Inference Podcast (Video)
A look inside the European Covid Green Pass - Codemotion (Slides, Video)
Node.js scalability tips - Azure Developer Community Meetup (Slides)

I am stopping counting them (more on this later), but just looking at the length of the list this year feels like a personal record from this the public speaking angle.

If you are curious to see all my talks, I keep collecting them in the speaking section of this blog.

After 2 years of remote speaking I have to admit that, even if I like to be able to speak to people all around the world with just a click, I do miss meeting people face to face. It feels like I haven’t been able to form the same level of relationship I had the fortune to form in the previous years where I was able to meet attendees, organisers and other speakers in person.

Hopefully this year it will be safe to do a few events in person again. 🤞

Sharing is good, and with digital technology, sharing is easy.

— Richard Stallman

AWS Bites

A new activity I started this year is AWS Bites podcast. AWS Bites was born from an idea of Eoin Shanaghy. It’s essentially a podcast series where, in every episode, we try to answer an interesting questions about AWS.

If you want to have a feeling for it, this one is probably one of my favourite episodes so far:

Some other episodes I am really proud about:

The goal for this podcast is to be able to share our AWS expertise in a simple and easy to digest format.

Even if we have recorded more than 20 episode it still feels like we are at the beginning of the journey and we are often trying to revisit and refine the format and the delivery. So, if you have any suggestion please let us know!

Also, if you like the format don’t forget to subscribe AWS Bites YouTube channel. 🙏

If you prefer the audio-only version, you can check out AWS Bites podcast on Anchor.fm, which also makes it available on Spotify, Apple Podcasts, Google Podcasts and others!

The art and science of asking questions is the source of all knowledge.

— Thomas Berger

Advent of Code in Rust (Twitch live streams)

On my personal quest to learn Rust, I decided to start a Twitch channel where I could live-stream my efforts. And the best thing ever is that I haven’t been alone in this effort because I have been joined by my good friends Roberto and Eugen!

Together we have been live-streaming our attempts at cracking Advent of Code using Rust. It has been a great challenge and a very good way to consistently have something to hack in Rust.

We are posting all the recordings on YouTube, so if you are really curious to see us stumbling against the Rust borrow checker or scratching our head with some interesting algorithm you can do that (even at 2x speed).

Last year we managed to complete all the exercises from year 2020 of Advent of Code. This year we are continuing our efforts at cracking the 2021 edition. We have already solved most of them, so we might decide soon to start streaming other types of Rust adventures… Maybe some serverless AWS with Rust or building a game with Bevy… who knows!

We live-stream every Monday at 6PM Dublin time.

Make sure to subscribe the YouTube channel or Twitch if you want to learn Rust with us!

Try to learn something about everything and everything about something.

— Thomas Huxley

Node.js Design Patterns

The main thing to report about Node.js Design Patterns is that the book kept getting some good amount of organic love.

I have been seeing many mentions online and even jokes!

Thanks Kevin for this brilliant mockup (Kevin has even created a PHP version)!

⚠️ For the records, this is a joke, we are not writing a book about Rust or PHP! 😅

What’s amazing is that Node.js Design Patterns has recently surpassed 100 reviews on Amazon and it’s still rocking positively at an average of 4.6 stars! This fills me with joy! ❤️

If you haven’t left a review for Node.js Design Patterns yet, please consider doing so. This is one of the most effective actions you could do to help us promoting the book and tell us what we can do to make it better! 🙏

A final note is that, throughout last year, we published 3 new blog articles on the blog section of the book website:

Node.js race conditions
Node.js development with Docker and Docker Compose (huge thanks to Giuseppe Morelli for authoring this one!)
JavaScript async iterators

Go check them out if you find them interesting!

You cannot open a book without learning something.

— Confucius

FullStack Bulletin

FullStack bulletin is a free weekly newsletter about full stack web development. I have been running this project with my dear friend Andrea Mangano for the last 4 years counting 258 issues today!

The project is totally open source and uses tons of automation to minimise the amount of manual work.

Although we had some plans for a redesign, we didn’t manage to find the time to ship any significant change In 2021, but the newsletter kept growing organically nonetheless.

One cool thing worth mentioning is that we started to get some interest in sponsorship campaign and we managed to get 5 amazing companies to sponsor several editions of FullStack Bulletin:

Sponsorships help us cover the costs associated with running the newsletter (Mailchimp, domains, hosting, etc.).

If you would like to support FullStack Bulletin and you have a product or an idea that you would like to expose to an audience of ~2.000 amazing full stack developers, consider sponsoring one of the next issues.

Finally, there are certainly many ways we can improve FullStack Bulletin, so if you have any suggestion, please do let us know!

More than kisses, letters mingle souls.

— John Donne

Middy

Middy is a Node.js middleware framework for AWS Lambda. I have been working on this project since the early days of Lambda (even though the first public commit was made the 3rd of August 2017).

With that being said, I can’t really claim anything for the success of the framework during the last couple of years. Insted all the credit goes to Will Farrell, who has been relentlessy maintaining the framework, and to the amazing community behind Middy!

The reason why I wanted to mention Middy here is because last year Middy started to get a lot more traction.

First of all Middy reached 2700 GitHub stars (not a crazy impressive number, but still something I can feel proud about):

Yeah, if you haven’t given your star to the project, what are you waiting for? 😛

But let’s be fair, stars are just a bit of a vanity metric! What is cooler is that Middy is currently trending at about 160.000 downloads/day:

If we look at the total number of downloads per year things seems even more impressive:

In this chart you can see that @middy/core + middy (legacy 0.x package) reached together about 6 million downloads in 2021!

And, just to put the cherry on top of the cake, what made me really happy is that Middy started to get some interesting coverage:

Lambda PowerTools for TypeScript by AWS works seamlessy with Middy and it the integration has even been showcased live by some AWS engineers
Middy is helping with delivering Tacos 🌮🌮. No kidding, there’s a re:Invent 2021 talk by Taco Bells mentioning Middy
Serverless Stack (framework) has an entire documentation page showing how to use Middy validator with Serverless Stack
Redwoodjs is considering integrating with Middy
Middy is also helping with testing formula-one style electric cars… Those must definitely be faster than a scooter!

While all this success fills me with happyness, I expect even greater results for this year as version 3 is shaping up really nicely and should come out soon!

PS: Last year, Will and I were interviewed about Middy by the folks at Cooper Press: A Chat with Will Farrell & Luciano Mammino of Middy.

I bought an electric scooter in sixth grade. Bankrupted me.

— Zac Efron

Pro-bono mentorship

In 2021, I invested some of my spare time to help engineers landing their first job. Together with my good friend Padraig O’Brien, we helped 3 engineers to fine-tune their CV, understand how to present themselves and get ready for some coding interview.

I mostly helped around preparing for the coding interview. Altough it was a lot of fun to meet eager engineer and do some pair programming with them, the most rewarding part was to actually see that they were actually able to land their first job and that this is having a very positive impact in their lives.

Out of the 3 engineers I had the pleasure to mentor, 2 of them got a permanent role and 1 is working off a few interesting contract roles. Let me know if you are thinking about hiring full-time an amazing React/JavaScript developer and I’ll put you in touch with them!

I’d love to keep investing 1 or 2 hours of my free time every month to continue this effort. It feels great to give something back to the community. I have learned from so many people during this year and it’s good to finally being able to carry the torch for a bit!

If you are looking to land your first job in tech and you feel like you can use some help for the coding interview, give me a shout!

If you are keen to getting your CV reviewed and understand how you can better present yourself, make sure to reach out to Podge.

PS: you can also check out this article I wrote last year: 10 tips to smash the code interview.

Don’t judge each day by the harvest you reap but by the seeds that you plant.

— Robert Louis Stevenson

SLIC Watch - Easy AWS o11y

Last year I had the fortune and the pleasure to use some of my fourTheorem time to contribute to a very cool open source project: SLIC Watch.

SLIC Watch is a Serverless Framework plugin that generates for you CloudWatch dashboards and alarms for:

AWS Lambda
API Gateway
DynamoDB
Kinesis Data Streams
SQS Queues
Step Functions

The idea behind this project is that very often we don’t invest in providing all these observability tools just because it’s hard and time consuming to get them right.

SLIC Watch tries to solve this problem by looking at all the resources in your stack and it automatically adds relevant dashboards and alarms based on common industry best practices.

This way you can get a good degree of “observability coverage” with very minimal effort (and time).

Of course, SLIC Watch cannot replace bespoke and fine tuned dashboards and alarms, but it’s still good in the sense that it can give you a good base that should cover 80% of your needs. This way you can focus on the 20% that matters more and that is very bespoke for your business.

Just to give you a more practical example, if you defined a Lambda function in your application stack, SLIC Watch will automatically provision the following alarms for you:

Errors
Throttles, as a percentage of the number of invocations
Duration, as a percentage of the function’s configured timeout
Invocations, disabled by default
IteratorAge, for function’s triggered by an Event Source Mapping

SLIC Watch also generates relevant dashboards for you:

If you find this interesting, make sure to check out the official repository (and while you are at it, give it a star 😛).

You can also check out these slides where we present the context for SLIC Watch in a more detailed way:

To acquire knowledge, one must study; but to acquire wisdom, one must observe.

— Marilyn vos Savant

Digital Green Certificate, Covid Library (Rust)

In June last year I got curious about the QRCodes used as part of the Digital Green Certificate program (a.k.a. DGC or DCC).

The specification allows to encapsulate certified proof of Covid vaccination, Covid test or Covid recovery within a QRCode.

I have been thinkering with the idea of using QRCodes with a digital signature in the retail world (tickets, certified coupon codes, etc.) so I was extremely curious to learn more about this project and see how they implemented the certification.

As part of this research I ended up creating a decoder library in Rust.

This library allows you to be able to read all the information encoded within the QRCode and to verify the autenticity of the certificate by checking its digital signature.

It was an extremely interesting project where I got to learn a bunch of new things:

…and I also reinforced a bit my understanding of cryptography and compression.

The best part is that this project somehow got the attention of the (very active) Rust Italia group, which helped me a lot to refine the library and to build new features.

This was amazing because it has been a great opportunity to learn from more experienced Rust developers and learn how to make a “production-ready” library. Thank you Rust Italia! ❤️

I had condensed my learning in a Codemotion talk: A look inside the European Covid Green Certificate:

If this topic is something that interests you, I will be delivering the same talk again at Rust Dublin Meetup, this month!

When solving problems, dig at the roots instead of just hacking at the leaves.

— Anthony J. D’Angelo

Articles

In addition to the articles already mentioned for the Node.js Design Patterns blog, I managed to write a few more articles during 2021.

Here’s the full (boring) list of the ones published in this blog:

In retrospective, it’s interesting to see a good mix of Rust and AWS articles. I think this reflects well where my learnings have been focused throughout the year.

I also ended up publishing a few articles outside this blog:

Let’s have a quick look at some stats for this blog:

It’s interesting to see a mix of new and old articles, But other than that the figures in terms of total number of page views are not to far off from last year (about 70k pageviews).

If we look at acquisition channels, the figure does not change too much from last year:

The majority (more than half) of the readers came from organic search. This is probably because I have a large chunk of articles in the long time still driving a lot of traffic and hopefully because of some SEO.

But, if I have to be honest, I am not particularly interested in doing an in-depth analysis and trying to figure out how to squeeze more traffic from search engines.

I should probably focus more on figuring out what topics are the most interesting for my audience and make sure I have a steady line of content to target those.

Whether you’re keeping a journal or writing as a meditation, it’s the same thing. What’s important is you’re having a relationship with your mind.

— Natalie Goldberg

Open Source

Every year I reported here a list with every single repo I contributed to. It’s honestly a waste of time (I am finally admitting that), so let me just summarise the ones that I think are more meaningful:

scrobbles is a Node.js library that allows you to fetch your “scrobbles” (songs you have been listening to) from Last.fm.
I have created a Node.js Streams workshop that you can avail for free: lmammino/streams-workshop.
I created the repo lmammino/async-iteration-http-requests, where i thinker with the idea of using async iterators to handle incoming requests in a Node.js web server. This is something that Deno suggests as standard way build web servers. This experiment sparked a bunch of interesting conversations and I ultimately think this might be a dangerous practice.
I created a repo with all my examples from the talk “Finding a lost song with Node.js and async iterators”: lmammino/node-async-iterators-talk-examples
I created the repo lmammino/javascript-iteration-protocols to show many different ways that you can use to do iteration in JavaScript and Node.js.
As part of my research for DGC, I ended playing with several Rust libraries for handling CBOR and eventually I contribured to my favourite one: enarx/ciborium. In this case I added a bunch of helper functions to simplify deserialization.
I contributed to udondan/cdk-ec2-key-pair, a CDK construct that allows you to generate key pairs for AWS EC2 instances. Small contribution to the docs, mostly redacting some examples that were a bit out of date.
I started gmaps-static, a Rust crate that allows you to build URLs for Google Maps static pictures. I even recorded a live video while coding this together with Tommaso Allevi. It is still not fully complete but it was a great experiment to learn a bunch of new interesting Rust concepts. Thank you, Tommaso! 🙌
I consider playing the best way to learn something. If you can make it fun, you won’t even notice you are learning. For this reason I started lmammino/awesome-learn-by-playing, a repository where I tried to collect all the resources that can teach you about coding and electronics by playing games. This project gathered a bunch of contributors and many stars (thanks everyone), so it seems many people like to learn by playing. 😊
allwords is another Rust crate that allows you to sequentially generate all the possible combinations of words over a set of characters. Could be useful for brute force attacks or fuzzy testing.
rollz is a very simple Rust crate that allows you to… roll dice! Nothing special except it was a great way to experiment with return type polymorphism and generics in Rust.
I have been a pedantic nerd by sending a PR to CircleCI where I renamed all their references to “NodeJS” to “Node.js”! 😅
Microfeat is a super-simple and incomplete feature flag server written in Rust. I wrote this during an hackaton with Roberto Gambuzzi and it was a lot of fun to learn how to deal with http servers and websockets in Rust.
I contributed to node-formidable/formidable a nice Node.js library that allows you to easily process multipart form data in streaming mode. Quite good to handle file uploads on a web server.

GitHub is something that is integral part of my daily routine. If I have an idea, 9 times out of 10, I start a repo on GitHub! If I use a library and I see something that I think can be improved I submit a PR.

So I think it will come natural for me to just keep doing more open source throughout the year!

If a person has a contribution to make, they must make it in public. If learning is not made public, it is a waste.

— Chaim Potok

Moar stuff

Let’s finish this post with some random stuff I did that didn’t deserve it’s own section.

I started using Polywork, a new interesting professional social media platform focused on achievements and content creation. If you want to connect, make sure to check out my PolyWork profile.
I started using Obsidian to take notes and organise my thoughts and learnings. I expect it might help me a lot for content creation, but I guess we’ll revisit this guess next year!
I have been attempting to learn how to type better using the touch type technique. This came from a realization that I am not comfortable with the Vim shortcuts because of the way I put my hands on the keyboard… I have been practing for a while with some online softwares, but I eventually gave up… I might try this again at some point this year!
Speaking of keyboards, I got a Keychron K3 and i am loving it. One moar reason to get better at typing!
I haven’t restarted practicing BJJ (damn Covid!), but I have been keeping a decent running routine! Last year I run a total of 415 km. Thanks to my friend Luca Marchesotti for holding me accountable and for often being a running partner!
While running, I have been listening to audio books. I listened to a bit of everthing between educational (tech/finance) and novels. Probably my favoutire e-book of 2021 has been To Sleep in a Sea of Stars by Christopher Paolini. Thanks to Rafael Del Nero for suggesting me to start listening to audio books while running. 🏃‍♂️💨

Expectations for 2022

In this section I generally used to list goals and compare with the previous years goals to see what I had achieved and what I failed at.

From this year I want to try something different. Rather than thinking in terms of goals I want to establish systems that allow me to have routine where I can do things that will eventually lead me to results.

I have already some routines with the Rust streams and AWS Bites podcast that help me to get better at some of the things I care about: learning Rust and getting better at AWS and cloud architectures.

One single goal that I want to achieve this year is to get a new AWS certification. I am proactively studying for that, so stay tuned because I will be sharing my notes!

Wrapping up

In conclusion, 2021 was a crazy busy year! I did not even realise how much stuff did happen until writing this post!

I feel really lucky and I can’t wait to see what I will be able to write at the end of this year… 😊

Whait, WHAT? Are you still with me?

That’s crazy, at this point even my mom would have closed the tab!

(Mom, if you are reading this, don’t worry, I love you anyway!)

What can I say? Thanks for reading all this stuff! You are my hero, so leave me a comment and let’s connect! 🥰

I would love to know what were your achievements last year and what are you planning to do this year!

Chat soon! 👋

The Definition of Senior: A Look at the expectations for Software Engineers

Luciano Mammino — Sun, 26 Mar 2023 16:30:00 GMT

In today’s fast-paced tech industry, the role of a senior software engineer has become increasingly important and something many engineers strive to be recognized as. But are you senior yet? And what’s expected from you a senior?

The truth is that answering these questions is a very subjective matter and different people or different companies might give very different answers.

In this article, I’ll try to give you my take, based on my opinion and personal experience.

So let’s get to it!

This article is based on a presentation I gave last year called The senior dev, an opinionated take.

Hopefully, this article can inspire engineers to grow in their careers, help managers to hire the right people, set expectations, and support their teams.

Senior? Yes, but how much? 🤌

The first remark I have to make is that I am not going to focus on different levels of seniority like what it means to be a Staff, a Principal engineer or even an Architect.

What I want to focus on is what it takes to be considered and valued as a Senior in the most general sense.

Ok Luciano, but what heck do you mean by being senior in the most general sense?

The core responsibilities of a senior engineer are to move projects and people forward.

Some people like to call that “being a force multiplier”, which, be aware, is very different from being a 10x engineer. In fact, a senior engineer is first of all a team player and not a lone hero, or a rockstar, a superstar, a magician, a grey-bearded wizard, a unicorn, or a whatchamaccalit!

Easier said than done! What kind of skills, mindset, and duties are we talking about? How can an engineer effectively help to move things forward (more than a junior or a mid-level engineer can do)?

Photo by Quinton Coetzee on Unsplash

Am I a senior yet?

The first myth to bust is that being senior is not necessarily a matter of how long one has been in the market!

More time ≠ more senior

Yes, it definitely helps to have been around for longer. The more you do something the higher the chances that you will have faced different challenges and learned something from them!

But let’s not say that you are a senior if you have 5+ years of experience or something like that. Unfortunately, this simple model won’t work well for many engineers!

Similarly, being a senior is not a matter of age. Growing older doesn’t necessarily make you better at your job… a senior is not someone with at least 40 years of age…

In my career, I had the fortune to truly appreciate this.

I had some very young but extremely passionate and skilled colleagues that I would definitely consider senior, and I also had older colleagues with many years in the industry on their CV but their contribution to the team and the project was nearly close to zero, if not even negative! I wouldn’t have called them senior, even though, most of the time, the company recognised them as such…

Photo by Elena Koycheva on Unsplash

What skills do I need?

If it’s not a matter of time in the industry, what is it then?

Remember, it’s all about moving your project and your team forward… and this takes some skills, so what kind of skills are we talking about?!

When it comes to technical skills, a senior software engineer is expected to have a solid foundation in programming languages, frameworks, databases, architectures, etc. However, technical skills are not the only thing that sets a senior software engineer apart. Soft skills, such as communication, leadership, and problem-solving, are equally important.

I would even argue that for a senior engineer, soft skills are even more important than technical skills, because it’s not just about what you know, but is about how you communicate that, how you influence other people to leverage your knowledge, how you share your expertise, unblock tricky situations, and make the team more self-sufficient and productive.

Communication skills are vital for senior software engineers, as they often work with cross-functional teams and communicate technical concepts to non-technical stakeholders. A senior engineer must be able to clearly articulate their ideas, listen actively, and provide constructive feedback.

Leadership skills are also essential for senior software engineers, as they are often responsible for mentoring junior team members, leading project teams, and making hard technical decisions. A senior engineer must be able to inspire and motivate their team, delegate tasks effectively, and provide guidance and support when needed.

Problem-solving skills are another critical skill set that senior software engineers must possess. They must be able to think critically, analyze complex problems, and find creative solutions. Senior engineers should be comfortable with ambiguity and be able to adapt quickly to changing requirements.

At the end of the day, software engineering is all about solving problems!

But in complex organisations, with many stakeholders coming from very different backgrounds, problems are not only technical. And problems can be badly defined and change often. So, being able to contribute to understanding what is the problem at hand, creating and sharing a vision, steering the team to take the right direction are all crucial skills that can have a massive impact.

The best technical solution is not always the absolute best solution for an organisation.

The best solution is one that accounts for how easy it is for more junior people to contribute to it, how easy it will be to evolve it and adapt it to changing requirements, how well will it scale if the business is successful, etc.

Sometimes you might have to compromise on purely technical qualities to come up with solutions that are optimal for the team and the business environment you are working on.

It takes a lot more than just technical skills to see all the facets of a project and be able to come up with solutions that can fulfill all the needs.

Photo by Kelly Sikkema on Unsplash

Am I technical enough?

Ok, ok… but I know we should really talk more about pure technical expertise!

After all, given the way the tech industry works today, this is the first thing people will recognize you for!

If you cannot demonstrate technical expertise you are not even going to have a chance at getting a spot for that job position you really like!

T-shaped profile

As a senior software engineer, possessing technical skills that span across different domains is crucial to thriving in the ever-evolving landscape of software development. To be an effective and efficient senior software engineer, a T-shaped profile that emphasizes mastery of one skill and proficiency in others is essential.

Photo by Jonathan Farber on Unsplash

A T-shaped profile for a software engineer refers to having expertise in a specific skill or domain (represented by the vertical stroke of the T), while also having a broad understanding of other areas (represented by the horizontal stroke). This allows individuals to contribute to a project in multiple ways while still being an expert in their primary area of focus. T-shaped profiles enable effective collaboration with team members who have different skill sets, leading to a more successful development process.

A good way to build a T-shaped profile is to start by developing your vertical stroke: what will eventually become your core strength!

Focus for a while on one area only. Yes, it’s better if you pick one that you really like.

When you feel you mastered that particular domain try to expand your knowledge around it and start developing your horizontal stroke. Step outside your comfort zone and explore related topics. Ideally, these related topics should compound your existing knowledge and extend your ability to build products end-to-end.

If you have become an expert in API development, you might want to explore databases or frontend development, because those will start to move you to a more “full-stack” profile. Similarly, you could also explore infrastructure-as-code and learn how to bring applications to production.

The trick is not to try to become an expert in every one of these additional areas, but just to learn enough to be able to understand the basic ideas and collaborate effectively with people who have these areas of knowledge as their core expertise.

Broad understanding

A broad understanding of software development is also necessary. This includes understanding the platform, architecture, code structure, testing, deployment processes, and scalability models.

Moreover, a senior software engineer should comprehend the tradeoffs between different technologies and paradigms such as Monolith vs Microservices, Memory vs CPU, Highly Scalable vs Low Latency, Reusable vs Bespoke, and Complex vs Simple. Understanding the short and long-term impact of these tradeoffs is also crucial.

Developing this kind of knowledge is non-trivial. It certainly requires spending a significant amount of time building products and doing so adopting different practices, programming languages, frameworks, methodologies, etc.

You can broaden your knowledge only if you appreciate that every technical choice comes with its own set of tradeoffs. There will always be positive aspects and negative ones.

There’s no silver bullet in technology!

There’s no framework that you can use to build any kind of application optimally!

If you have a comprehensive enough understanding of the tech landscape, you should be able to tell which set of technologies is the best to address the specific problem at hand.

And, of course, it’s ok to be proven wrong, as long as you keep an open mind and you leave yourself (and the team) room for making mistakes, learning from failures, and trying alternative approaches.

The path to success is rarely a straight line…

Photo by Sylvain Gllm on Unsplash

Gonna catch all bugs!

We know that bug-free software is a myth, so I am not going to say that a senior software engineer doesn’t create bugs! Of course, they do…

But it’s really important for a senior software engineer to know what can be done to reduce the number of bugs to the very minimum and to try to spot them as early as possible (possibly before the users do).

Understanding and refining user stories, writing different types of tests such as unit, integration, and end-to-end tests, finding and discussing edge cases, and keeping track of technical debt are all essential skills.

A senior software engineer should be able to make a big difference here and help the entire team to write better software by introducing and educating best practices about software testing.

I would argue that it’s not too hard to write tests, but writing good tests it’s almost an art!

To write effective tests, you need to write testable code. This means designing code that’s modular, loosely coupled, and has well-defined inputs and outputs. The better the code is designed, the easier it is to write tests for it.

Your tests should be easy to read, understand, and maintain. Use descriptive names for your tests, and write tests that cover one specific feature or behavior. Avoid testing multiple features in the same test, as it can make it harder to pinpoint the source of the error.

But how to learn to do all of this?!

Again, practice practice practice!

Start by accepting that untested code is bad code (or to the very least code that should keep you awake at night), then try to understand the specific business domain and what makes sense to be tested and how: what kind of inputs are the most representative, what kind of outputs are expected and what kind of side effects can happen.

Learn the different types of tests and different testing frameworks. Finally, learn which types of test will provide the most value for the business and optimise the team workflow to focus more on those.

Photo by Charlotte Descamps on Unsplash

Promote the right patterns

Having written a book on design patterns I am certainly biased on this one…

A senior software engineer should have developed some degree of flexibility in different programming languages and they should understand paradigms such as OOP vs Functional, Declarative vs Imperative, and Compiled vs Interpreted. This will allow a senior software engineer to be able to solve the same problem using different tools and techniques.

Similarly, a senior software engineer should understand various design patterns and best practices. This will allow them to suggest patterns that can have good long-term effects, and avoid patterns that will eventually lead to problems.

Code is something that will need to evolve as the business and customer needs change, so it is important to design with evolution in mind.

Patterns are not silver bullets, but just blueprints that you can decide to use to address common situations and pave the way for future improvements.

Don’t try to be dogmatic and apply specific design patterns just because you can, try to think about what’s the value that the specific pattern is bringing to the project and whether that value is actually important.

Sometimes, simple dummy code can be very effective and it can be understood and evolved easily. Some other times you do need the structure imposed by specific design patterns because that might able to make your code more testable, extendable, configurable, etc.

As a senior software engineer, this is an opportunity to shine and have a significant impact during pairing sessions and code reviews!

What kind of soft skills?

Now that we covered which technical skills will give you an edge in your career and allow you to be recognised as a senior engineer, let’s address the elephant in the room: soft skills.

You got that interview thanks to your hardcore technical skills, but now you have a chance to really make an impression by showcasing a range of soft skills.

Growth mindset

The mindset of a senior software engineer is just as crucial as their technical and soft skills.

A senior engineer must have a growth mindset, a willingness to learn and adapt, and a drive for continuous improvement.

They should be comfortable with failure, see it as an opportunity to learn, and be willing to take risks.

Always remember a senior engineer must also be a team player, not a hero. They should collaborate with their team, support their colleagues, and contribute to the team’s success.

Being an active lever

A senior software engineer should have a very proactive role in an organisation. They shouldn’t just isolate themselves in a room and keep smashing their fingers against the keyboard.

A senior should know when it’s time to ask hard questions and take leadership to find what they don’t know.

Being able to bring a strong technical perspective to a business conversation can be very impactful and inform the business on what’s the best strategy to move things forward.

To all effect, a senior should become a bridge between product and technology.

This sometimes means that a senior should also know when to say NO. No to quick and dirty solutions (that will eventually backfire hard) just to hit a deadline. No to plans that only account for building features on top of features without considering user experience and the long term stability of the product. No to arbitrary technical choices just because “we have always done it this way and it has been fine”.

Saying NO is easy, but also very hard. Everyone can just say NO, that’s the easy part! What’s difficult is to argue the why of that NO, propose alternative points of view, find compromises and defuse short or long term time-bombs.

Being effective at this requires a fine degree of soft skills. You should know how to communicate effectively and pick your battle wisely. You cannot always say NO to everything or you’ll become recognised as a grumpy naysayer.

It’s more about being able to say “NO, BUT” rather than just a dry “NO”. And you should also be capable of doing that in front of the right people and at the right time…

When done successfully, this can have a massive positive impact on the business and the team. It can lead to innovation and create new business capabilities or unique competitive advantages.

This is an area of growth for me. One of those things where, when I look back at my career, I feel like I have failed at it many many times. But hopefully, I have learned something and over time I’ll get better at it…

Photo by Michał Parzuchowski on Unsplash

Understanding the business

To be an active lever in your organisation, you need to be able to understand the business as deeply as possible.

Do you know what’s the purpose of the business? What’s the long-term vision? Is there a clear strategy? What are the unique strengths and what are the weaknesses?

If you can answer all these questions you are in a good position to use your technical skills to determine how technology can help the business to succeed.

Only with a solid understanding of the business you will be able to pick the right battles, focus on what matters and help design systems that can serve the business well today but also be adapted to future needs.

Communication

“Good communication is as stimulating as black coffee and just as hard to sleep after.” – Anne Morrow Lindbergh

As a software engineer, communication skills are crucial to your success. You must be able to communicate effectively with a variety of stakeholders, including other engineers, project managers, customers, and executives. This requires a wide range of communication skills, from speaking and listening to writing and presenting.

Since you will need to talk with all the stakeholders, you should learn how to communicate technical concepts to non-technical stakeholders in a way that is easy to understand. It also means being able to listen to feedback and incorporate it into your work. The ability to communicate effectively with stakeholders is essential for ensuring that everyone is on the same page and that projects are completed successfully.

Another important communication skill for a software engineer is the ability to explain uncertainties and propose ideas on how to address them. Software development is an inherently uncertain process, and there will always be unknowns and unforeseen challenges. As a software engineer, you need to be able to communicate these uncertainties to stakeholders in a way that is clear and concise. You also need to be able to propose ideas on how to address these uncertainties and move the project forward.

In addition, software engineers must be able to talk about failures and learnings. We already covered this, but it’s important to remark that failure is an inevitable part of software development, and it is important to be transparent about failures and the lessons learned from them. This requires effective communication skills, including the ability to take ownership of mistakes and communicate them honestly to stakeholders. It also requires the ability to communicate the lessons learned from failures and incorporate them into future projects.

Communication skills are also essential for writing documentation and delivering presentations. Software engineers must be able to write clear documentation that explains how their solutions work and how they should be used and operated in production. They must also be able to deliver presentations that effectively communicate technical concepts to a non-technical audience. This requires strong writing and presentation skills, as well as the ability to simplify complex topics and make them digestible for the audience.

Good communication skills can open many doors and unblock complex situations. So make sure to practice them as much as possible!

Supporting management

When you are a good communicator, you become someone who can be very effective at supporting various management activities, for instance, planning and driving ceremonies. You can help with keeping track of priorities and technical debt and split complex tasks into manageable parts.

You are in a unique position to understand and leverage team strengths and pull in the right people at the right time.

By working closely with management, you can ensure that your work aligns with the overall business objectives and that your team’s efforts are channeled in the right direction.

Planning is another crucial skill for a software engineer. This involves breaking down large projects into smaller, manageable tasks, estimating the time required for each task, and prioritizing them. Effective planning helps to ensure that projects are completed on time and within budget.

Driving ceremonies such as stand-up meetings, sprint reviews, and retrospectives can provide an opportunity to share progress, identify issues, and plan for the future. As a software engineer, you need to help drive these ceremonies and ensure that they are productive and efficient.

I am not suggesting that senior software engineers should replace the role of project or product managers, but instead, they should complement these roles and provide their unique technical perspective to keep objectives, plans and execution aligned with the expectations of the technical team.

Photo by Annie Spratt on Unsplash

Autonomy and focus on delivery

The last 2 soft skills I want to touch on are autonomy and a focus on delivery.

Sometimes senior software engineers are in a unique position to contribute significantly to projects that require in-depth research and grind. Heavily technical projects such as the ones related to performance optimization, significant refactoring, and re-architecture. These are often projects that are quite important for the business because they can unblock innovation and bootstrap new business capabilities.

In my experience, it’s not uncommon that in such projects, a single individual contributor might be well positioned to spend some amount of time focusing and achieving a significant amount of progress in a very short amount of time driving the first big chunk of the research forward.

But even in these situations, it is important to avoid silos at all costs.

If you are working on such a project, make sure to have regular check-ins with management and other senior colleagues to keep your progress in check, get useful feedback and share your learnings.

But the ability to deliver value by working autonomously is not only useful in research projects, but it’s also a good skill to master in general.

In the software industry, it is more common to work with things we don’t know than it is to work we things we know!

There are always new domains to explore, new challenges to face and new tools to deal with. Therefore, it’s important to be able to learn quickly and upskill fast when needed. You shouldn’t be waiting for other people to teach you and tell you how to do certain things, you should be able to do a certain amount of progress on your own, for instance by consuming and understanding documentation and existing code.

If you have broad skills already, it should almost be second nature to be able to add more to them when needed.

Of course, it is OK to ask for help if you feel stuck, but it’s also important to try to do everything you can to progress on your own so you can build as much context as possible and maximise the outcome of the time spent receiving help from others.

Ideally, a senior software engineer doesn’t need too much guidance. They can figure out what’s needed to move things forward by themselves and involve other people when needed.

Another facet of this involves negotiating expectations and understanding what it means to be successful in the current environment. It’s good to be upfront with management and the rest of the technical team and define what individual and team success looks like. If you don’t know what’s expected from you and the rest of the team, even if you feel you are doing the right thing, you might end up not delivering the value that is expected from you; similarly, you might not be able to help your team at being successful.

Building this attitude of bluntness and clarity generally keeps the mood more positive and helps with avoiding disappointments.

Of course, you can’t always avoid all disappointments or failure scenarios. So when the poop hits the fan, you might be tempted to blame someone else or the system, but it’s instead more mature to ask what could have been done better and what could be done in the future to avoid similar issues from raising again.

Photo by Immo Wegmann on Unsplash

How can I grow?

If you’re a software engineer looking to level up your skills and become a senior member of your team, you might be wondering what steps you can take to get there.

Well, fear not! Here are 4 ideas (plus some extras) that I believe will help you on your journey.

Go one level deeper

First off, don’t just scratch the surface of the technologies you’re working with. Dive deeper and learn about the underlying layers.

You have probably built a website or an API, but did you ever wonder how does the HTTP protocol work? Or even how does the TCP protocol works and what happens to establish a connection? Appreciating these details will give you a much richer understanding of the technologies you work with every day.

Yes, it’s an endless rabbit hole if you start to dig… so how we dive deeper without getting lost and get value from it?

My rule of thumb (which I have ~~stolen~~ learned from my friend Roberto), is to dig only 1 level down starting from the technologies you are already familiar with.

Have you spent some time with OAuth and OpenID Connect? Well, you’ll probably want to know what’s inside a JWT. Do you understand the OAuth authorization code flow and what happens with all the redirects? Actually, do you know how signing algorithms such as RSA and HMAC work?

These are all topics that you can explore to consolidate your knowledge of this particular domain. They will allow you to explore more general concepts that you might be able to re-use somewhere else.

All good steps towards broadening your knowledge and becoming a more well-rounded engineer.

I am often biased toward building stuff. I feel like I don’t truly understand something if I can’t build a small prototype for it. If you are like me you could try to do that as well. Can you decode a JWT token without using a library? Can you take it a step further and even implement the signature verification algorithm?

Building prototypes is a great way to memorise certain concepts and truly put your understanding to the test. Of course, it takes more effort to build stuff, so choose your prototype exercises wisely: you can’t possibly re-implement 60+ years of software engineering just for the sake of learning!

One funny video that I really enjoyed lately is The Computer Science Iceberg. Why do I like it so much? Because it truly illustrates this idea of descending levels of abstractions and exploring more fundamental and generic pieces of knowledge.

Photo by Sergey Pesterev on Unsplash

Have fun

This is one of my core beliefs: software engineering is a very challenging profession. Not because it’s harder than other professions but because things move so quickly that you can never stop learning and you’ll always have to feel behind.

The one thing that could make the profession a little bit easier is a genuine passion for the subject. If you are passionate it’s going to be easier to motivate yourself to learn new things over and over!

But what if I don’t know if I am passionate enough?

Fair, especially if you are at the beginning of your career. There is so much in front of you that it might be scary and even discouraging.

Again, my recommendation is to be biased toward building stuff. Building something will give you tangible feedback that what you are learning can actually be useful. You can build side projects to put new knowledge into practice.

Also, you shouldn’t be shy to show what you built (and what you learned) to your peers and even your friends outside of work! Chances are you’ll get feedback and come up with new ideas and new things you’d like to learn and try.

Even better if you realise you can apply some of these learnings at work. Maybe you can develop a new path in the company you are working for, maybe you can help with something that is currently being neglected because no one else has the time or the expertise, maybe all of this can demonstrate more value and you’ll get a promotion!

When you apply this mindset of continuous learning and sharing knowledge at work, this might generate cross-pollination and you might end up with a team that has fun learning and building together.

Another idea to make people gel with each other and to generate cross-pollination of ideas is to organise company hackatons or free-study days. These are great ways to help teams to become more passionate about what they do, explore new ideas, learn new things, and ultimately put people in a position to deliver more value in the short and the long term.

Pair programming

Pair programming is another great tool that you can leverage to give a boost to your expertise.

pairing with as many people as possible within your organization is key. Even if someone is more junior than you, they can still have insights and perspectives that you may not have considered before. Pair programming gives you a platform to share ideas and learn from one another.

Similarly, don’t discount the value of teaching others. Even the most senior members of your team can benefit from your knowledge and expertise. By pairing with others, you can share your skills and help everyone grow and develop.

I picked up so many tricks by pairing with other people. Even small things such as how they configured their editor or their terminal.

Now, I get it - not everyone loves pair programming. If that’s the case, don’t worry! There are other approaches to follow that can still provide you with valuable feedback and help you learn from your peers. Interactive code reviews and show-and-tell sessions are great alternatives that allow you to share your work and receive feedback without having to work together in real time.

In the end, the most important thing is to remain open to new ideas and approaches. By continuing to learn and grow, you’ll be well on your way to becoming a senior software developer. So why not give pair programming a try and see what you can learn?

Photo by Janko Ferlič on Unsplash

Content creation

First off, creating content can take many forms: articles, talks, videos, Twitter threads, you name it! And guess what? You don’t need to be an expert to share something new you learned. Even if you’re just starting on a given subject, your fresh perspective can bring value to others and help you establish yourself as a thought leader. So do create content on what it felt like to try a new programming language or a framework. What did you like? Was there something confusing? What did you miss from your previous experience?

But here’s the thing: you gotta make it a habit. Commit to a regular schedule, and you’ll develop discipline and consistency in your content creation. Plus, the more you create, the more you’ll hone your communication skills. And I’ll say it one more time, communication is a critical skill for any senior engineer.

And speaking of communication, that’s where content creation shines. As engineers, we love getting lost in technical details, but explaining complex concepts to non-technical stakeholders can be a challenge. By creating content, you’ll learn to communicate your ideas in a way that’s accessible and engaging to a broader audience. Plus, you’ll have an opportunity to get feedback on your content and, if you listen to feedback, you can improve your communication skills even further.

Have heard of the concept of “atomic essays”? That just means breaking down your ideas into shorter, more focused pieces of content. This approach can make your content more engaging and easier to consume - especially on platforms like Twitter, where brevity is key. If you don’t know what kind of content to create, this could be a good format to start with.

If you don’t know what to create content about, here’s my tip: every day, at the end of your work day write down 1 new thing that you think learned during that day. At the end of the work week, review your 5 points. I am sure that at least one of these will be worth creating some content about.

Creating content can help you become a more senior engineer by establishing yourself as a thought leader, developing your communication skills, and contributing to the community. So go ahead and share your knowledge with the world. You will be surprised at how much impact it will have on yourself and others!

Other ideas

Here are some other random-ish tips that you can add to the ones above.

First off, always try to keep a positive attitude. When things get tough, it’s easy to get discouraged and lose sight of the bigger picture. But remember, with enough time and resources, teams can overcome any challenge. When that luxury is missing, I am sure there are decent compromises that can meet everyone’s needs.

Another key to upskilling is to avoid being too picky about technology or programming styles. Make an effort to expose yourself to different ways of solving problems. It’s an exercise for your problem-solving skills.

It’s important to support our colleagues’ ideas, even if we would have done things differently. This kind of collaborative attitude helps build trust and fosters a culture of innovation and teamwork. Also, next time you will propose something, I am sure that people will be willing to discuss and support your ideas.

When it comes to taking on tough jobs, don’t be afraid to volunteer. What about that refactoring that nobody wants to do? Step up and take it on. Those messy, complicated parts of the code that everyone avoids? Dive in and try to make sense of them. By taking on these challenging tasks, you can develop a reputation as a problem solver and become the go-to person for the toughest jobs.

Finally, as a general rule, try to make things just a little bit better wherever you have the chance. Whether it’s by documenting a process, improving an application’s user interface, or streamlining a workflow, every small improvement counts. Over time, these small wins can add up and help us become more valuable team members.

How do I sell myself as a senior?

If I think I am a senior engineer what can I do to be recognised as one?

First, start by taking on more responsibility. Look for opportunities to lead projects or mentor junior engineers. Share your knowledge and experience with others and provide guidance.

Secondly, try to make an impact beyond your immediate team or project. Participate in cross-functional initiatives, contribute to open-source projects, or speak at industry events. These types of activities can help you establish yourself as a thought leader in your field and demonstrate your expertise to a wider audience.

Thirdly, focus on developing your soft skills. As a senior engineer, you will be expected to communicate effectively with both technical and non-technical stakeholders, lead meetings, and negotiate effectively. So work on improving your communication, leadership, and conflict resolution skills to become a well-rounded and respected member of your team.

Another way to demonstrate your expertise is by sharing your knowledge with others. Write technical blog posts, record tutorial videos, or speak at internal or external events to showcase your expertise and demonstrate your willingness to help others learn.

Lastly, seek feedback from your colleagues, mentors, and managers. Ask for constructive criticism and be open to suggestions for improvement. By being receptive to feedback, you can continuously improve your skills and demonstrate your commitment to growing and learning as a senior engineer.

Photo by Miguel Bruna on Unsplash

Conclusion

The definition of a senior software engineer is complex and multifaceted, but it generally involves possessing a deep technical expertise, a broad understanding of the industry, and a range of soft skills that enable effective communication, leadership, and problem-solving.

To become a senior software engineer, it’s important to continually develop and refine your skills, focusing not just on technical abilities but also on communication, leadership, and business acumen.

This might involve pair programming, creating content, volunteering for challenging projects, and seeking out new learning opportunities.

Ultimately, the path to seniority is not a linear one, and it requires dedication, perseverance, and a growth mindset to continue improving and evolving as a software engineer. So keep pushing yourself to go one level deeper, to have fun and find joy in your work, and to always strive to make a positive impact on your team and organization.

But if you have been wondering all the time whether I have been describing the perfect senior software engineer, let me only tell you that I am not expecting people to nail every single aspect mentioned in this article. There will be things you will be great at and others where you’ll be barely sufficient. That’s just human nature and it’s okay!

We cannot excel at everything but we should know our strengths and weaknesses work with our team to amplify strengths and compensate for weaknesses …and strive to get better every day!

It’s a journey, not a destination! Touché…

Why you should consider Rust for your Lambdas

Luciano Mammino — Wed, 16 Aug 2023 09:58:00 GMT

Rust is such a cool language, even though it might take a bit to get the hang of it. But trust me, the payoff is huge. As someone who’s all into serverless stuff and loves playing around with AWS, I’ve been digging into how to write AWS Lambda functions in Rust and whether it’s worth the sweat.

MEGA-Spoiler: Yep, you guessed it from the title - it’s totally worth it. And in this article, you’re gonna find out why, so stick around!

Okay, so there’s this pic in the article of Morpheus holding a Lambda in one hand and Ferry, the Rust mascot, in the other. But guess what? You don’t have to pick just one. It’s not a do-or-die choice. You can have both cakes and eat them too!

What is serverless and why do I like it

Let’s start with a bit of a generic intro. If you are already familiar with serverless, feel free to skip this section.

How do we learn anything these days? We ask ChatGPT, of course! So let’s see what it thinks about serverless.

Dear ChatGPT, what the hell is serverless?

Serverless is a cloud computing model where the cloud provider manages the infrastructure and automatically allocates computing resources to execute code in response to events or requests.

In traditional server-based architecture, the server is a dedicated machine that runs constantly and waits for requests. However, in serverless architecture, there are no servers to manage or provision. Instead, the cloud provider takes care of the underlying infrastructure, automatically scaling the necessary computing resources up and down based on the amount of incoming traffic.

With serverless computing, developers only pay for the actual usage of their application, not for the infrastructure. This makes serverless a cost-effective option for running and scaling applications. Serverless computing also reduces operational complexity, as the cloud provider takes care of infrastructure maintenance, security, and availability.

Examples of serverless services include AWS Lambda, Azure Functions, and Google Cloud Functions, which allow developers to execute code without managing servers or infrastructure.

Pretty good definition for generative AI, but let me give you my (shorter) summary:

Serverless, is a way of running applications in the cloud.
Of course, there are servers somewhere, we just don’t have to manage them.
We pay (only) for what we use.
We code in small units of compute (functions), which are triggered by events.

Now, why is all of this very cool?

In my experience, serverless helps with focusing a lot more on business logic and less on other concerns such as infrastructure, scalability, etc. Of course, there’s still a learning curve and there are tradeoffs. We are not going to get into the details in this article, so, for now, take this at face value: serverless gives us more time to focus on what matters for the business: providing value to the customers.

Serverless can also increase team agility. By virtue of forcing us to think in terms of small, event-driven functions, we are forced to think about keeping the code-base modular. This can help the team in many ways. For instance, it can be easier to distribute and parallelise the work. Also, if at some point we realise we want to rewrite or re-engineer part of the software since the various units are generally more decoupled, it should be easier to do that. You might have seen this one coming, but if we want to rewrite an entire piece of functionality in Rust, we can do that without having to rewrite the entire code base. We can change things incrementally, one function at a time! I am actually in the process of rewriting a serverless project in Rust and I am doing it live on Twitch (recordings here), just in case you are curious to see some examples…

Serverless gives us some degree of automatic scalability. Lambdas will be spawned up and down depending on the number of events happening. If we have a sudden surge of user activity, the system is generally able to provide the necessary amount of computing power to handle that. This is not an absolute. In reality, it is important to understand how cloud providers achieve this level of auto-scalability. The way they do it is not always effective for all use cases, but in practice, it’s something that can help a lot and that works well for the most common cases.

So yes, serverless is great, but it’s important to say that it is not a silver bullet. Certain scenarios are still better implemented with bare metal virtual machines, where you have to do all the hard work of managing servers… Or with long-running containers deployed on some container orchestration platform, where you have less infrastructure to manage, but still significantly more than what you get with serverless functions.

AWS Lambda

Let’s add another bit of context here. Feel free to skip it if you are already familiar with what AWS Lambda is.

AWS Lambda is the FaaS (Function as a Service) offering from AWS.

Lambda allows us to write your business logic as functions that are automatically triggered when certain events happen.

Just to give you some examples, such events could be:

An HTTP request is made against an API Gateway
A new file was created in an S3 bucket
A new job was published in a Job queue
A scheduled event
A manual invocation
An invocation orchestrated by a workflow (e.g. a Step Function)

And to make things a bit more concrete, here are some realistic examples of what we can do with this idea:

Build an HTTP API that implements a payment gateway using API Gateway and Lambda.
Automatically generate thumbnails for new videos uploaded to an S3 bucket.
Synchronise data from an FTP drive on a schedule
Scrape data from a website
Send welcome emails when a new user signs up (using an SQS queue or an Event Bridge event)
Analyse system and application logs and generate alarms for suspicious activities
Rotate secrets and restart the necessary applications

And so much more…

Lambda is pretty cool (you know I love it!), but it comes with some important limitations.

As of today, a Lambda execution cannot last more than 15 minutes. Also, the payload size is limited (both request and response payloads are constrained). And finally, you cannot have a lambda with a GPU.

There are other subtleties, but these limitations alone should make you think about some use cases where Lambda wouldn’t be a great fit. Therefore, don’t try to do everything with Lambda. It’s not another silver bullet (in tech, nothing is, really)!

If everything you have is a ~~Lambda~~ hammer…😏

AWS Lambda pricing model

One of the reasons why I think Rust is a promising solution for writing Lambda functions is because of potential pricing implications, therefore it is important to understand Lambda’s pricing model.

The pricing model for AWS Lambda is (relatively) simple and it’s based on the idea that we should pay only for what we use. The actual formula is:

Cost = Allocated Memory ✖️ Execution Time

To better understand what this means in practice, let’s discuss an example.

Let’s say we create a new Lambda function and assign 512 MB of RAM to it. At the time of writing in the eu-west-1 region (Ireland) the cost per millisecond for this configuration is $ 0.000_000_008_300. How many zeroes is that?! Anyway, let’s say we run this function for the maximum time available: 15 minutes, ergo 900_000 ms, this is what we end up with:

$ 0.000_000_008_300 * 900_000 ms = 0.007 $

Yes, 007 … I know this was a bad joke, but let’s pretend you liked it and let’s move on… 😅

The real question is: is .007 $ a lot? Or is it very cheap? Well, I guess it depends (of course it does!).

If we are running this Lambda function once a week, it’s basically for free… If we are running it a thousand times per minute, we end up paying about 300.00 $ per month, which isn’t my definition of cheap! And imagine what happens if we have thousands of requests per second!

But this should give you an idea of the pricing model and how it scales.

In general, for unpredictable usage patterns (sporadic or very spiky invocations), Lambda tends to be a cheap solution, but when you have a consistently high rate of invocations, it might get quite expensive and there could be significantly cheaper alternatives.

When it comes to serverless, it’s also important to consider the TCO (Total Cost of Ownership) which should also include the cost of maintaining the infrastructure (provisioning, patching, security, etc.). This tends to be very low with serverless, while it gets more expensive with more traditional approaches.

So, yeah, cost is always a complex matter, but if you want to deep dive into better understanding how the price equation might influence your choice of computing technology, I’d recommend reading this excellent article: “Why AWS Lambda Pricing Has to Change for the Enterprise ” by Eoin Shanaghy.

Lambda functions and CPU allocation

Now, you might be wondering, how come CPU doesn’t come into play in the pricing model? What if we need to run a very CPU-intensive task and we would like to have multiple cores to distribute the load and speed up the computation?

Well, the catch is that we don’t explicitly get to configure the number of vCPUs, we automatically get a certain amount of vCPUs proportionally to how much memory we are configuring for our Lambda.

The number of vCPUs that we will get can be seen in the following table:

Memory	vCPUs
128 - 3008 MB	2
3009 - 5307 MB	3
5308 - 7076 MB	4
7077 - 8845 MB	5
8846+ MB	6

Note that this is not an official table but the result of an independent study by SentiaTech.

The takeaway here is that, if we need more CPU, we are forced to provision more memory, even if we don’t need all of that memory… and yes, increasing the memory will increase the price per millisecond of execution. But that doesn’t necessarily mean that the executions will be more expensive because with more CPU we might be able to finish the task much faster, therefore we are billed for a smaller amount of milliseconds. I know, it’s tricky to find the sweet spot between memory, CPU and price. If you are looking for a tool to help you out, check out the excellent Lambda Power Tuning.

Anyway, even though it sucks not to be able to specify exactly how much CPU we need, this model keeps things simple for most use cases… Having to think about only 2 dimensions (time and memory) keeps pricing and configuration rather straightforward.

… and most likely it also helps the Lambda team to allocate our Lambdas efficiently in their compute clusters.

The execution model of a Lambda function

We have seen that the whole promise of Lambda, being a serverless service, is that we only pay for what we use.

For this pricing model to work, the Lambda team has to be very savvy with the allocated resources, ideally provisioning them only when they are needed and de-provisioning them as soon as they are not needed anymore.

How does this happen? Without getting too technical, this is a high-level overview:

When we deploy a Lambda function, we are shipping its code into an S3 bucket. So, in a way, we are just saving files in the cloud.
Lambda functions are event-based. So this means that when an event trigger happens, AWS should be able to execute the code we provisioned.
To execute the code, AWS needs to allocate some computing resources. In the case of Lambda, this means spawning a Firecracker micro-VM somewhere and initializing it with the code from our S3 bucket.
This operation might take some time. After all, we need to think that behind the scenes, AWS is downloading the code from S3 and spinning up a VM. They surely have very efficient ways of doing that, but it can still result in noticeable delays in the order of hundreds of milliseconds. When this happens it is called a cold start.
Once the VM is up and running, the Lambda execution environment can invoke our lambda and pass the event data.
Once the execution is completed, the runtime doesn’t immediately destroy the instance. If the same type of event happens in a short amount of time, the instance is re-used, which means that the new event will be passed into it. This saves us from having to endure another cold start. This might happen multiple times, especially if we have events being generated at a consistent rate.
But let’s assume that at some point we stop receiving events for a few minutes. At this point, the Lambda runtime might decide to reclaim the resources allocated for our Lambda instance and then it shuts it down and de-provisions it.
If a new event will come in at some point in the future, the whole lifecycle will start again.

If you want to go more in detail, I recommend reading about the Lambda execution environment.

Why Rust + Lambda = ❤️?

Ok, we are finally here! Now that we have covered enough background knowledge, we should be able to make the point that Rust is an ideal language to write Lambda functions with.

Cost saving

The first reason why Rust is so cool for Lambda functions is that Rust is a typed and compiled language which produces very efficient binaries. If well written, Rust code can be on par in terms of speed with equivalent C or C++ code. Other than being very performant, these binaries are generally very memory efficient and very small in size.

All these characteristics are perfectly aligned with all the billing dimensions we have with Lambda, so Lambda function written in Rust can be much cheaper than most alternative languages.

To summarise this point:

The performance will likely reduce the number of ms needed to complete the execution.
The memory efficiency might allow us to keep the allocated memory low.
The small binary size might reduce the duration of cold starts (less time needed to fetch the code from S3).

Regarding cold starts, there is a fantastic benchmark by Maxime David, and guess what? Rust is the fastest runtime in terms of cold start! And by far!

If you have a very hot Lambda that you are running thousands of times per day and you used Java, Python or Node.js, and you are looking for opportunities to reduce your AWS billing, consider rewriting this one Lambda in Rust.

After all, that’s the beauty of Lambda-based architectures: Lambda functions are generally very small and decoupled, so it should be possible to rewrite one of them without having to touch the rest of your infrastructure.

Thank you, dear Rust, for helping us to save some sweet money. Cheers to you!

Now, I know what you are thinking…

“But having to learn Rust, will cost me sweet money and time too!”

Tru dat! 🤷‍♂️

Of course, you need to evaluate whether the juice is worth the squeeze case by case, but what I would say in response to that is “Well, yes. But consider it as an investment!”

Memory safety

This is one of the main propositions of Rust. Rust was created with the intent of eliminating an entire class of memory and threading bugs. These kinds of bugs have been one of the main sources of security issues for a few decades! Even Microsoft believes that if they had written Windows in a language with the guarantees of Rust they would have avoided 70% of their security issues!

But, in practice, what does this mean for Lambda?

The way I see it is that most developers (or maybe it’s just me) are scared about writing multi-threaded code and therefore we end up trying to avoid doing that. Which means we might be losing precious opportunities to optimize our algorithms.

In the context of Lambda, this means potentially longer execution times, which again might be another reason for an unnecessarily large bill.

Since Rust can give you the confidence that your multi-threaded code is safe (if it compiles it is safe), this might encourage scared developers like me to try to leverage more opportunities to write multi-threaded code.

Isn’t that a nice proposition?

Fewer bugs

Ok, this is probably the most opinionated point I have about Rust. But having written enough Rust in the last 4 years, I came to the realisation that when I write code in Rust, it generally just works. Or to put it more sincerely, it’s generally less buggy than it would be compared to using other languages such as JavaScript, Python, or Java.

This is not because Rust is magic or because you become some kind of super programmer with it. I honestly think that it’s just due to some clever language design decisions that force you to think more about the cases when things can go wrong.

In particular, I am referring to the fact the the language has no null types. If something may or may not exist, the language forces you to handle that explicitly using the Option type.

Also, the language doesn’t have a concept of exceptions. You can’t just throw stuff wherever you want. When something can fail, you also need to explicitly use the Result type.

I have been loving using these constructs and, in retrospect, I wish that languages like Python and JavaScript had something like that. They just make it so much more obvious when something can go wrong in your business logic and force you to think about a way to handle these edge cases.

Now, just to be clear, I am not promising Rust will magically solve all your bugs 🐞. You’ll still write buggy code (that’s probably inevitable), but I am sure your code will be much more well thought out and that you’ll be automatically handling many more error cases compared to using other languages where you can just assign stuff to null or throw random exceptions here and there without thinking too much about the possible consequences of these decisions…

Where do we start?

Ok, did I convince you to entertain the idea of writing Lambda functions in Rust?

Yes? Great!

So, now you are wondering where to start…

This is probably a topic for a series of more hands-on articles, so stay tuned for more.

But if you really can’t wait, I will be hosting a talk titled “Rust, Serverless and AWS ” at the next Rust Dublin Meetup on Tuesday, August 22nd, 2023.

In this talk, I will show many practical code examples and use cases and some of the tooling you can use today to bootstrap, test and deploy Lambda functions written in Rust.

If you are not in Dublin, you can join the event anyway. There will be a live stream as well 🙂

Update: here’s the recording of the talk, I hope you’ll enjoy it

Finally, if you think your organisation needs help with Serverless or with reducing your AWS bills, consider reaching out to fourTheorem. We are a consulting company specialised in serverless and all things AWS, and I like to think we are lovely to work with. 😊

Conclusion

Rust is a really cool programming language and it’s something we should consider using for writing Lambda functions.

Due to its characteristics and design choices, Rust might help us to save money on our lambda function and to write more correct (read “less buggy”) Lambda code.

So, here’s a choice. Do you pick Lambda or do you pick Rust? Why don’t you pick both?

👋

JavaScript, low-level or AI?

Luciano Mammino — Fri, 10 Nov 2023 14:37:00 GMT

The software industry is going to be fun to see in the coming 5-10 years.

I see an interesting tension happening right now…

The generative AI side

On one side we have generative AI capturing almost every bit of the software development lifecycle and effectively becoming a new high-level abstraction, possibly the highest level we have ever seen in our industry.

There seems to be no stopping this phenomenon… just looking at the latest announcements from GitHub Universe, it’s clear that we will have to adopt AI in one way or another and this is not necessarily a bad thing if it truly makes us all more productive and focused on generating business value.

If you don’t know what I am talking about you should check out this video:

And if that’s not enough you should watch the full GitHub Universe 2023 keynote.

GitHub Universe 2023, Day 1. Thomas Dohmke on stage with a slide in the background saying “One more thing”, mimicking the Steve Job’s launch of the iPad

The part that impressed me the most is that now Copilot can also help you lay out the structure of a project, breaking down requirements into potential tasks. Once you are happy with the result, it can start to work on the individual tasks and submit PRs. This is called Copilot workspace, if you want to have a look.

It seems too good to be true and it’s probably going to be far from perfect for a while, but there’s great potential for efficiency here, and I am sure GitHub (and other competitors) will keep investing in this kind of products and maybe in a few years, we’ll be mostly reviewing and merge AI-generated PRs for the most common use cases.

If you think that ChatGPT was launched slightly less than 1 year ago, what will we be seeing in 5 or 10 years from now?

The low-level side

On the other side, we have a wave of new low-level languages such as Go, Rust, Zig (and Carbon, and Nim, and Odin, and VLang, and Pony, and Hare, and Crystal, and Julia, and Mojo, and.. I could keep going here… 🤷‍♀️).

OK, I really wanted to put the lovely Hare language mascot here. Hare is a systems programming language designed to be simple, stable, and robust. Hare uses a static type system, manual memory management, and a minimal runtime. It is well-suited to writing operating systems, system tools, compilers, networking software, and other low-level, high-performance tasks.

All these languages take slightly different trade-offs, but, at the end of the day, they are built on the premise that we need to go lower level and have more fine-grained control over how we use memory, CPU, GPU and all the other resources available on the hardware. This is perceived as an important step to achieve better performance, lower production costs, and reach the dream of “greener” computing.

If you are curious to know why we should be caring about green computing, let’s just have a quick look at this report: Data Centres Metered Electricity Consumption 2022 (Republic of Ireland).

The report findings state that in 2022, in Ireland alone, data centers’ energy consumption increased by 31%. This increase amounts to an additional 4,016 Gigawatt/hours. To put that in perspective we are talking about the equivalent of an additional 401.600.000.000 (402 billion!) LED light bulbs being lit every single hour. If you divide that by the population of the Republic of Ireland this is like every individual is powering ~80.000 additional LED light bulbs in their home, all day and night! And this is just the increase from 2021 to 2022… How friggin’ crazy is that?! 🤯

Photo by D A V I D S O N L U N A on Unsplash

Ok, now one could argue that we had low-level languages pretty much since the software industry was invented. So why isn’t that the default and why do we bother wasting energy with higher-level programming languages?

That’s actually quite simple: because coding in low-level programming languages such as C and C++ is hard! Like really really hard! And it’s also time-consuming and therefore expensive for companies! And I am not even going to mention the risk of security issues that come with these languages.

So why should this new wave of low-level programming language change things?

Well, my answer is that they are trying to make low-level programming more accessible and safe. They are trying to create paradigms that could be friendly enough to be used for general computing problems (not just low-level), which could potentially bring the benefits of performance and efficiency even in areas where historically we have been using higher-level languages and made the hard tradeoff of fast development times vs sub-optimal performance.

Take for example Rust. It was historically born to solve some of the hard problems that Mozilla had to face while building Firefox. But now it’s being used in many other areas, including embedded systems, game development, and even web development. Not just on the backend, but even on the frontend using WebAssembly!

I am not going to claim that writing stuff in Rust is easier than doing the same in JavaScript or Python, but it’s definitely easier than doing the same in C or C++.

So there might be many cases where we will be able to use these new languages to achieve better performance and efficiency without having to pay a massive development price for using a low-level language.

And I would go as far as saying that these use cases exist in the industry today and there’s a staggering lack of talent in these areas.

Why the tension?

So, is there really a tension here between generative AI-driven development and using low-level languages or are these just two very disjoint things?

I would personally say yes, there’s a tension.

Again, generative AI is pushing us to care less about the details. We trade our time and attention for the ability to focus on the business value and let the AI do the rest. This is a trend that has been going on for a while now and it’s not going to stop anytime soon.

Investing in using a low-level language goes in the opposite direction. It’s a bet that we can achieve better performance and efficiency by going lower level and deciding to be explicit about the minutia of how we want to use the hardware at best.

But, wait… Am I saying that AI is not going to be able to write efficient and hyper-optimised low-level code? 🤔

Maybe! Or, at least my belief is that, as with any abstraction, there’s always a price to pay. And the price of using AI is that we are going to be less explicit about the details and therefore we are going to be less efficient.

But I also expect this equation to change with time. As AI improves, it might be able to generate more efficient code. Possibly even better than code we would write manually, even with tons of expertise on our side.

What can we do as software developers

Where does that leave us?

As individual software engineers, we can’t expect to be able to change these trends. We can only try to understand them and adapt.

Investing in learning a new language is a multi-year effort, and although it might be fun (if you are a language nerd like me), it is time that you might be taking away from other activities that might be more rewarding in the long term or just more valuable to you. For instance, you could be learning more about generative AI, right? 🤓

My personal bet is to invest in both! I am currently learning Rust and I am also trying to keep up with the latest developments in the AI space.

For instance, Eoin and I just released a new episode of AWS Bites where we explore Bedrock, AWS generative AI service… Check it out if you are curious to find out what we built with it!

I am not sure how much I will be able to keep up with both, but I am going to try my best.

I tend to be a generalist and it’s only natural for me to try to explore a wide space of possibilities rather than going super deep on one specific topic.

But I am also aware that this is not the best strategy for everyone. So, if you are a specialist, you might want to focus on one of these two areas and try to become an expert in that. It might come with a risk, but it might also come with a great reward.

I am also of the belief that the more we learn the more we are capable of learning. So regardless if you decide to go wide or if you put all your eggs in one basket, the important thing is to always keep learning and keep an open mind.

If the future takes an unprecedented turn and we all end up writing code in a new language that is generated by AI, I am sure that the skills we have acquired in the past will still be valuable and will help us to adapt to the new paradigm.

Only the future will tell… And maybe, even after all this fuss, we’ll still be writing tons of JavaScript in 10 years from now! 😜

What do you think?

So what’s your opinion and what’s your strategy for the future? I’d love for you to strongly disagree with me… or not?! Either way, let me know what you think here in the comments or on X, formerly Twitter.

See you around and happy coding! 🤓

2022 - A year in Review

Luciano Mammino — Tue, 20 Dec 2022 08:50:00 GMT

Yup, 2022 flew and it’s that time of the year again. I want to write that boring post that I am the only one to look forward to writing and I don’t expect anyone to read.

Well, if you are brave enough to go ahead and read it, let me say thank you and then, to follow the tradition, let me greet you with a boring GIF:

Becoming an AWS Serverless Hero

Let’s start by addressing the elephant in the room, the biggest (and most unexpected) achievement of the year: being awarded the title of AWS Serverless Hero by AWS! 🦸‍♂️

Now, why is it a big deal to me? Let me quote AWS:

The AWS Heroes program recognizes a vibrant, worldwide group of AWS experts whose enthusiasm for knowledge-sharing has a real impact within the community. Heroes go above and beyond to share knowledge in a variety of ways including online via social media, blog posts, open source projects, videos, and forums; or in person at conferences, workshops, and user group events.

I have surely done a good amount of work in this sense and I am really happy to get the recognition, but this one is very hard to get and I did not expect it at all.

In addition to that, it’s not a program you can simply apply to. AWS has obscure ways to discover and select potential heroes every year, and there aren’t that many worldwide.

Heroes are divided into areas of expertise (Community, Container, Data, DevTools, Machine Learning, Serverless). I was awarded the Serverless hero title, which makes a lot of sense, since most of my AWS efforts have been around serverless in the last ~5 years.

What I believe has been the most important thing that I have done in the realm of serverless is middy, a Node.js middleware engine for AWS Lambda which allows you to organize your Lambda code, remove code duplication and focus more on business logic.

And this brings me to the part where I have to say thank you to other people because without them I probably wouldn’t have gotten this recognition:

Will Farrell, maintainer of Middy. Will has done incredible work in the last 3 years taking Middy above and beyond. I wouldn’t have been able to keep the project going without Will’s hard work, so I am incredibly grateful for continuing from where I mostly left off. Thanks, Will!
The amazing team at fourTheorem, because I feel like I am learning and growing a lot by working with such a talented and enthusiastic team of professionals.
Eoin Shanaghy for sharing his endless knowledge with me and for being such a fantastic co-host for AWS Bites Podcast.
Tons of folks who have been inspirational in my AWS serverless journey: Yan Cui, Heitor Lessa, Sara Gerion, Matt Meckes, Andrea Amorosi, Alex DeBrie, and many more that I am sure I am forgetting (sorry).
All the other other heroes, for being such an incredible source of inspiration.
All the AWS folks who decided to pick my name for the award!

What happens next? Well, for starting I got some cool swag. Thanks, AWS much appreciated.

Yeah, my box probably got a few kicks on the way…

Other than that, you’ll probably see me posting more AWS and serverless-related content and most likely I will attend the re:Invent in 2023!

Also, I want to take this opportunity to give a big thank you to the AWS Heroes team: Farrah, Taylor, and Albert.

“With great power comes great responsibility”

— Uncle Ben

Confirmed as MVP

In July I got a nice azure postcard from Microsoft telling me that I got re-confirmed as MVP (Most Valuable Professional) for Developer Technologies for the years 2022 and 2023!

The Microsoft MVP program is somewhat similar to the AWS Hero program, meaning that it aims to reward people who contribute heavily to the tech industry by sharing their knowledge, creating content and practicing public speaking. The main difference from the Hero program is that the MVP program is much more generic, it does not necessarily focus on Microsoft technologies.

It makes me incredibly proud to be recognized as such by Microsoft, especially considering that I am not a big user (nor promoter) of Microsoft products and languages.

One cool thing about the MVP program is that they give you a little totem the first year and every year you get a new token that can be added to the stack!

I should put it on a shelf at some point!

You can also check out my MVP profile to make sure I am not lying to you. 😜

“Sharing is good, and with digital technology, sharing is easy”

— Richard Stallman

Codemotion Ambassador

In 2022 I got nominated Codemotion Ambassador!

Codemotion is the biggest tech community in Italy and one of the biggest in Europe. They also organize some of the best tech conferences that I had the pleasure to attend.

Codemotion ambassador program aims to create a structured Codemotion supporters network. Ambassadors are supported to share their software development knowledge, grow their professional skills, and provide value to the global tech community.

Since I have been eagerly contributing to Codemotion during the last 8 years, it makes me really happy to be part of this group of people who can evangelize this community and make a bigger impact together on the overall tech community.

I am particularly proud of one of the many initiatives I have been involved with as an ambassador: the Wannabe speaker program, in which, every ambassador got the chance to help a first-time speaker to deliver a kick-ass presentation.

I had the pleasure to work with Danilo Spinella who presented a talk called Beautiful CLI Applications in Rust. You can watch the recordings if you are curious (and you should).

To be fair, Danilo did not need a lot of support and he was able to deliver an excellent and enjoyable presentation like a true veteran speaker!

Also, I want to take this opportunity to thank Francesco Sciuti, Mara Marzocchi, and the rest of the Codemotion team for adding me to this fantastic group of professionals. Ok, we are not always that professionals but we definitely know how to have some fun!

“You cannot teach a man anything. You can only help him discover it within himself”

— Galileo Galilei

Public Speaking

I did not stop doing public speaking in 2022, so let’s have a look at a picture and then some stats.

This picture is from my talk at Codemotion Milan in October where I presented a talk called: “Serverless for High Performance Computing”.

In 2022, I spoke at 25 different events and surpassed more than 100 speaking engagements in total! Yeah, vanity metrics feel good sometimes! 😛

Here’s the full list for 2022:

Migrate the monolith to the cloud on Fabio Biondi’s Twitch Channel (in Italian) (Slides)
A look inside the European Covid Green Pass at Rust Dublin Meetup (Slides, Video)
Dal monolite al cloud: no stress! at Codemotion DevCast (in Italian) (Slides, Video)
The senior dev at Private event (Slides)
Happy days with Node.js streams workshop at CityJS London (Repository)
Finding a lost song with Node.js & async iterators at private event (Slides)
AMA: Node.js design patterns, AWS e altre cose fantastiche live interview with Emanuele Bartolesi for UGIdotNET (in Italian) (Video)
Teach Kelvin Your Thing: Node.js streams with Kelvin Omereshone (Video)
Finding a lost song with Node.js & async iterators at Node.js Global Summit (Slides, Video)
Javascript iteration protocols workshop at Codemotion Workshop Fest (Slides)
Let’s build a Serverless e-commerce from scratch workshop at CityJS Athens (Slides)
Finding a lost song with Node.js & async iterators at CityJS Athens (Slides)
Serverless for HPC at AWS Dublin Meetup (Slides)
Async JavaScript and Node.js Design Patterns at Sailsconf (Slides, Video)
Interview with Reconfigured podcast (Video)
Building an invite-only microsite with Next.js & Airtable at React Dublin Meetup (Slides)
JavaScript Iteration Protocols workshop at NodeConfEU (Slides, Repository)
Serverless for High Performance Computing at Codemotion Milan (Slides, Video)
Serverless for High Performance Computing at Cloud Day (Slides)
Come sviluppare una carriera in tech inteview with NonCompila on Twitch (in Italian) (Video)
Full Stack Developer Creating Content interview with Zak Aghbal for the Content Creator Life Podcast (Video)
Let’s build a 0-cost invite-only website with Next.js and Airtable at Conf42 JavaScript (Slides, Video)
Everything I know about S3 pre-signed URLs at AWS User Group Dublin Meetup (Slides)
From Node.js To Design Patterns at BuildPiper Meetup (Slides, Video)
Building an invite-only microsite with Next.js & Airtable at React Milano (in Italian) (Slides, Video)

If you are curious to see all my other speaking engagements, I keep the complete list in the speaking section.

If you are brave enough to withstand my accent, you can invite me to speak at one of your events.

“All life is problem solving”

— Karl Popper

AWS Bites

In 2022, Eoin Shanaghy and I produced 45 new episodes of AWS Bites, a weekly podcast where we discuss interesting AWS-related topics.

“What can kitties teach us about AWS” is probably my favorite episode so far:

I don’t really know if it’s the best episode to date, but certainly, it is the one with the best thumbnail! 😼

One cool thing we did for AWS Bites is that this year we tried to build an actual product from scratch in a series of live streams. We built a minimal but functional clone of Dropbox transfer or WeTransfer and we did that by using serverless technologies on AWS.

The playlist with the recording of the live streams is available on YouTube: Live coding a serverless file transfer app on AWS.

Finally, it is worth mentioning that we surpassed 1000 subscribers on Youtube and that we also got some awesome stats from our Podcast on Spotify.

The episode count somehow does not match exactly what we have done, but all the other stats are more impressive than we expected. Let’s assume these stats are somewhat accurate and let’s celebrate… I guess. 🍻

“Success is not final, failure is not fatal: It is the courage to continue that counts”

– Winston Churchill

Live coding on Twitch

Throughout 2022 I kept streaming on Twitch quite consistently. As I did last year, most of it was about solving Advent of Code challenges and learning Rust with my friends Eugen and Roberto (thanks a lot for being there, having fun, and learning with me).

We streamed 36 times in total (almost once every week). Here are some more nerdy stats:

I am not at the levels of ThePrimeagen, but hey we had a lot of fun and I certainly learned a lot!

Towards the end of the year, we also spent a bit of time learning Bevy, a game engine written in Rust and built some game prototypes with it.

If you want to watch the recordings of these live experiments, here’s a YouTube playlist: Rust gamedev with Bevy.

A big shout-out goes to Tommaso Allevi, for helping us kickstart this journey into Rust gamedev. I have to say I really like the ECS (Entity Component System) model of Bevy so far, so I look forward to playing more with it!

Finally, I have been experimenting a bit with OBS and did a solo stream where I played a bit with Solid.js a frontend JavaScript framework that I am liking a lot and that I’d like to use more.

All the recordings of my live streams are available on YouTube, so make sure to give me a follow there!

“Play is the work of the child”

— Maria Montessori

Node.js Design Patterns

Node.js Design Patterns, the book I co-authored with Mario Casciaro, is still trending quite well after 2.5 years since the third edition was published.

The thing that makes me really happy is that the book keeps receiving very positive reviews. We doubled them since last year while keeping an incredibly high average rating of 4.6/5.

At the time of writing the book is trending at:

#110,173 in the global list of Amazon books
#11 in the Web Services category
#22 in the JavaScript category

We also keep receiving spontaneous comments and seeing random Reddit comments about how the book has been useful to readers.

It seems like the book is even being used as teaching material in some universities… and someone even contacted us to say that they discovered this book in the library of a tech investment fund in San Francisco. 🤯

Node.js Design Patterns keeps feeling like one of my biggest achievements professionally and I am always grateful to Mario for involving me in this awesome project.

The book still feels quite relevant and up-to-date (updated to Node.js 14 and using ESM throughout), so I don’t feel like it’s time for a new edition just yet.

If you disagree, do let me know. I’d love to hear your opinion!

Finally a HUGE shout out to my dear friend Kelvin Omereshone for gifting copies of the book to quite a few people! ❤️

“A book is a garden, an orchard, a storehouse, a party, a company by the way, a counselor, a multitude of counselors”

— Charles Baudelaire

FullStack Bulletin

FullStack bulletin is a free weekly newsletter about full stack web development. I have been running this project with my dear friend Andrea Mangano for the last 5 years.

The achievement of 2022 is that we surpassed 300 issues and 2500 subscribers!

FullStack bulletin is always looking for sponsors, so if you like this format and you would like to reach a vetted audience of full stack engineers please consider sponsoring us!

“The art of art, the glory of expression and the sunshine of the light of letters, is simplicity”

— Walt Whitman

Middy

Middy is a Node.js middleware framework for AWS Lambda. I have been working on this project since the early days of Lambda (even though the first public commit was only made on the 3rd of August 2017).

With that being said, I can’t really claim anything for the success of the framework during the last three years. Insted all the credit goes to Will Farrell, who has been relentlessly maintaining the framework, and to the amazing community behind Middy!

The main news is that Middy v4 was published this year bringing significant innovation including:

Support for Node.js 18
Adoption of the new modular AWS SDK v3
Better support for ESM
Various performance improvements and bug fixes

Middy has been growing even more in 2022, here is a shiny graph to prove it:

This graph shows the number of weekly downloads and it clearly highlights that the amount is more than doubled from the previous year!

Note that @middy/core represents newer versions (from v1 to v4), while middy is the legacy 0.x package (surprisingly still being used a lot).

If we look at the total number of downloads throughout the year we surpassed more than 8 million downloads!

I look forward to checking these stats again next year!

Meanwhile, if you use Middy, make sure to support Will Farrell for his awesome open-source work (yes, you should consider donating on GitHub)!

“If you want to heal your soul, ride a Vespa instead of driving a car”

— anonymous

Articles

In 2022, I wasn’t really a prolific author on this blog. I published only 2 articles:

On the other end, I think I was quite prolific in other media. Below there’s a full list, but the most important ones in terms of authority are definitely AWS and InfoQ! Both feel like massive achievements on their own!

“Every secret of a writer’s soul, every experience of his life, every quality of his mind, is written large in his works”

— Virginia Woolf

Open Source

In 2022 I continued steadily to do some degree of open source work by contributing to existing repositories and starting new ones.

Here’s a list of the most meaningful contributions:

lmammino/tall - a Promise-based, No-dependency URL unshortner (expander) module for Node.js. I have been working with Karl Horky to make the library extensible and to add some advanced plugins.
lmammino/rust-advent - the repository where we publish all our Rust solutions for Advent of Code.
I contributed to aws-samples/serverless-patterns with a small fix that leverages the streaming capabilities of sharp.
I created a repository to benchmark the performance of phf, a Rust library using perfect hash functions to generate build-time hash tables, against a mega-match expression.
I delivered a talk called “The Senior Dev” and since I decided to use slidev to generate my slides, I hosted the source code for the slides on GitHub.
I created fourTheorem/api-gateway-sls-example, an example of how to build a simple API with API Gateway, Lambda and the serverless framework.
Thanks to my open source work on lmammino/iteration-protocols-workshop, I had the pleasure to be invited to review a PR that adds Async generator documentation to MDN.
I developed a repository called lmammino/secret-pizza-party, a sample Next.js website to showcase how you can make invite-only SPAs with Next.js, AirTable & Vercel.
I worked with the awesome Theodore Vorillas and submitted a couple of PRs to improve proximahq/clickhouse, a simple, plain, efficient Clickhouse client.
I worked with Eoin Shanaghy on awsbites/weshare.click, an example personal file transfer app on AWS.
During NodeConf EU I attended a fantastic workshop by Patty O’Callaghan about tensorflow.js and, as a result of what I learned, I developed a sample application that you can find in lmammino/crappy-object-recognition-web-app (screenshot below).
During NodeConf EU I also collected all my notes in a repository.
I created a simple script that helps me with tracking my expenses for tax reasons. I am not a tax expert, but you can take a look and adapt it to your particular use cases (while making sure to consult your accountant)!
I have created several repositories with my Rust Bevy experiments: lmammino/bevy-wrapping-square, lmammino/stranded-in-space, and lmammino/bevy-fuel-tank.
I also created a Rust repository with my experiments with Nannou, an open-source creative-coding toolkit for Rust.
I started solving Advent of Vue challenges but using Solid.js instead and I am keeping my code in this repository: lmammino/advent-of-solidjs.
I created a simple repository that can be used as a library and an API to get facts about Mount Etna (the volcano close to where I am from in Sicily).

Above a screenshot of my crappy object recognition application built with Tensorflow.js and Solid.js. I am also about 70% sure that I am a real person! 😆

A final mention is my open PR together with Simone Sanfratello where we are trying to bring the awesome anti-trojan-source detector by Liran Tal directly into nodesecurity/eslint-plugin-security. I hope this will be merged before the end of 2022!

“Perseverance is not a long race; it is many short races one after the other”

— Walter Elliot

Random stuff

And as we get ready to wrap things up, here are a few more random things:

Last year, I became a Certified AWS Solution Architect Professional. If you are curious to know what was my experience and get some of my study notes, make sure to check out my article AWS Solution Architect Professional exam, my notes and tips.
I got married to the most wonderful person in the world (and this alone makes 2022 the best year ever)! 👰‍♀️❤️🤵‍♂️
I Restarted BJJ training (and after almost 3 years it feels like I have to re-learn everything from scratch)!
I kept running, running a total of 340 Km… 🏃‍♂️ And once again big thanks to my friend Luca Marchesotti for motivating me.
I got covid… twice! 🤧

Expectations for next year

OMG, are you still here?! I am seriously impressed. 🥸

At this point, you should go the extra mile and make sure to leave me a comment!

Ok, before we wrap things up, let me tell you what’s my plan for 2023.

Well… there is no big plan, really!

My overall goal is to keep growing my expertise in AWS and Serverless and work on interesting projects. Luckily there is no shortage of opportunities when working with fourTheorem, so I feel this goal will be pretty much achievable organically by just keep doing what I do in my daily work.

I’d also love to have an opportunity to put all the Rust learnings of the last 4 years into something a bit more practical, or production-ready if you will! If a work project does not manifest itself I might try to get a bit more serious with my side projects and maybe try to build something more involved in Rust. I will most likely continue some game dev sessions in my streams, but I might also try to do something that combines Rust with AWS, possibly including Serverless. I have been experimenting with Rust lambdas and I have been blown away by the performance, so there’s an opportunity there.

Finally, I have been quite excited by Solid.js, so I’ll be seeking new opportunities to play with it!

Of course, I’ll try my best to keep pushing things that I have been investing in the last few years: AWS Bites, FullStack bulletin and live coding on Twitch. I will try to find other opportunities for content creation, most likely publishing articles and delivering talks at conferences.

If you have any suggestions or comments on this obviously perfect plan, I am all ears!

I’d also love to know what was your biggest achievement of 2022 and what you plan to accomplish in 2023, so don’t be shy and leave a comment!

This is me, signing out and getting ready for 2023!

Until next time, peace! 👋

I am co-authoring a book about Rust and Lambda

Luciano Mammino — Mon, 16 Dec 2024 09:30:00 GMT

I’m beyond excited to share some big news with you: I’m co-authoring a new book called Crafting Lambda Functions in Rust together with the amazing James Eastham! If you’re as passionate about serverless and Rust as I am, you’ll want to check this out. We even have an official website for it, check out the book Crafting Lambda Functions in Rust.

This book is your hands-on guide to building efficient, scalable, and cost-effective serverless applications with AWS Lambda functions and using the power of the Rust programming language.

How it all started

The idea for this book didn’t happen overnight. It all began when I kept crossing paths with James at conferences over the years. Every time we met, we’d end up deep in conversation about two of our favorite topics: serverless computing and Rust.

This year, at Rust Nation in London, the stars seemed to align. We realized that we had both been toying with the idea of writing a book to share what we’d learned about building AWS Lambda functions in Rust. We talked about how Rust is not only incredibly powerful for Lambda but also surprisingly easy (perhaps unexpectedly) to get started with. Rust has a reputation of being hard to grasp, so it might come as a surprise to many to discover that building Lambda functions with this language it’s a very accessible endeavor. While we were debating different approaches we had played with and discussing the current tooling ecosystem, it became clear that a little evangelism could go a long way in bringing this tech combo to a broader audience.

At that point, joining forces felt like the most natural thing to do. We agreed to tackle this project together, and what started as an idea is now becoming a reality. And while we’re still knee-deep in writing, we hope that someday we’ll look back and say, “And the rest is history!” For now, though, there’s a huge pile of work ahead, and we’re ready to take it on.

By the way, before you move on to find out more details about our motivations and the book itself, I’d like to make it very clear that the book is already available for purchase in early access. This means that you can start reading it right now and help us shape the final version with your feedback. If you’re interested, head over to Crafting Lambda Functions in Rust’s website to grab your copy. And yes, at the time of writing it’s heavily discounted, so I’d like to think you get yourself a bit of a good deal…

Oh yeah, let me show you the book cover as well, so you can tell that this is real!

Yes, our mascot is a crab (a cRustacean, get it?) with a Lambda hat… If you have been wandering around this blog long enough you’d probably get the video game reference as well!

Why Rust and Lambda?

I have actually written an entire blog post on this last year: Why you should consider Rust for your Lambdas. But don’t worry, if you are in a rush, I’ll give you a quick summary here anyway.

The combination of Rust and AWS Lambda is a game changer. Over the past few years, I’ve been using Rust to build serverless applications, and I’ve seen firsthand how well they complement each other. Here’s why I think Rust is an awesome choice for writing Lambda functions.

Cost Savings and Sustainability: As a compiled language, Rust produces highly efficient binaries that are compact and lightning-fast. This means your Lambda functions run quicker, consume less memory, and ultimately cost less to execute. Remember that with Lambda functions the execution cost is given by the formula memoryCost * executionTime where memoryCost is a price unit that depends on how much memory you allocated for the Lambda and executionTime is how long that function has been executing in milliseconds. With Rust, in most circumstances, you can lower both dimensions, compared to interpreted languages such as JavaScript and Python. But it’s not just about saving money: Rust’s efficiency reduces resource usage, which translates to a lower carbon footprint. Your apps can be both cost-effective and environmentally friendly.
Reliability and Correctness: Rust’s strict compiler and type system are like having an extra pair of eyes on your code. I am a big fan of the idea of not having a null type in the language and that you have to express the potential absence of data using a dedicated wrapper type: the Option type. The same goes for the Result type which lets you deal with errors in a very explicit way. These features make it easier to spot edge-cases as early as possible and you have to explicitly decide how to handle them. Once you get used to these abstractions and appreciated their benefits, you’ll find yourself being much more confident that your code is correct and reliable. When you’re working in a serverless environment where reliability is often non-negotiable, this is a huge win.
Fast Starts, Smooth Runs: With Rust, you get tiny package sizes and no garbage collection. This often translates to faster cold starts. How fast you might ask… What I have seen is something on the range of 10-60 ms (depending on the amount of libraries and the amount of init work to be performed). This is in the order of 10-20x of what I have generally observed with JavaScript and Python. Now think about using Lambdas as your HTTP backend or for any other task that has a user waiting for a response. This means that, even in the occasional presence of a cold start, your users will barely notice any additional delay which translates to a more seamless experience, even during peak loads.
A Streamlined Developer Experience: Last but definitely not least, we should talk about DX. When I first talk to people about using Rust for Lambda functions, most of them are initially skeptical. And I get it: although fast-growing, Rust is still a relatively niche language, and it’s certainly not common to see it being used with Lambda. Plus Rust has gotten a reputation of being somewhat difficult to learn, so all of this creates an expectation that, although there might be strong benefits, it must be real hard to write a Lambda in Rust and bring it all the way to production. Now, we can argue about Rust being difficult or not. I acknowledge there might be some initial challenges when you come from languages that manage memory for you, but it’s also fair to say that it isn’t that hard to start writing Rust and get productive with it. You don’t need to become an all around expert of the language to write decent code that be shipped to production. But this isn’t the only point, because the real super power for the combo Rust-Lambda is a tool called Cargo Lambda. This tool makes it super easy to scaffold a Lambda, test it locally, compile it and ship it to production. It even integrates with most of IaC tools out there such as SAM, Terraform and even CDK. This let’s you focus on writing great code while letting the tooling handle the heavy lifting. I seriously wish a tool like this existed also for Node.js and Python!

Early Access: Read and Shape the Book!

If you got this far, hopefully, you are now convinced that using Rust for your Lambda functions is not that crazy of an idea, so the next question is how do you get started?

Well, that’s why we are writing this book, so you are in the right place!

I have a bad and a good news though!

The bad news is that the book isn’t finished (unless you are reading this a bit ahead in the future). But, the good news is that everything we have produced so far is already available and there’s a significant amount of work already!

You can get your copy of the book right now in early access. This means you can get started today! By purchasing early access, you’ll:

Receive chapters as they’re written: You’ll get new content and updates regularly.
Contribute to the book: Share your feedback, help shape the final version, and, if you like, get your name on the book!
Enjoy a discounted price: Early access comes at a (significantly) reduced cost from what we are expecting the final cover price to be!

This approach lets us work collaboratively with readers to ensure the book is as practical and comprehensive as possible.

Is it worth to get the book right now? I don’t know, you tell us! 😅

I’d like to think it is a good deal if you are looking to get started ASAP.

Worth mentioning that what we have published so far includes 6 chapters that will guide you from building a serious serverless application (a URL shortener service) from scratch and ship it to production. This accounts for more than 100 pages and tons of code examples and exercises.

What’s in (and what will be in) the Book?

Here’s a list of the chapters of the book, giving you a clear roadmap of what’s available and what’s coming:

Chapter 1: Rust & Lambda (available): We’ll explore why using Rust and AWS Lambda together is a great idea. We’ll discuss the benefits of this combination compared to other alternatives.
Chapter 2: A ‘Hello, Serverless’ API (available): You’ll learn how to write your first Rust-powered Lambda function. You’ll create a simple “Hello, Serverless” API that demonstrates the basics of AWS Lambda and Rust.
Chapter 3: Infrastructure as Code with SAM (available): We’ll show you how to deploy and manage your Rust Lambda functions using the Serverless Application Model (SAM). You’ll learn how to create, update, and delete your functions.
Chapter 4: Integrating with External Systems (available): We’ll teach you how to make arbitrary HTTP requests or interact with other AWS services from your Lambda functions. We’ll cover various techniques for integrating with external systems.
Chapter 5: Code Organization (available): This chapter will focus on best practices for organizing the code for projects with multiple Lambda functions. You’ll learn how to structure your code for maintainability and readability.
Chapter 6: Making the Code Testable (available): We’ll show you how to structure your code in a way that allows you to write good tests for your Lambda functions. You’ll learn about testing fundamentals and how they apply to Rust Lambda functions.
Chapter 7: Handling Different Kinds of Events (in progress): In this chapter, we’ll discuss how to handle AWS-specific events, custom events, and arbitrary JSON in a strongly-typed way. You’ll learn about event handling best practices for your Rust Lambda functions.
Chapter 8: Different Ways of Writing Lambda Handlers (in progress): We’ll explore different ways to write your Lambda function handlers. We’ll cover simple async functions, custom structs, and implementing the Tokio Tower service trait.
Chapter 9: Configuration Management and Handling Secrets (planned): This chapter will focus on best practices for injecting configuration and secrets into your Rust Lambda functions. You’ll learn how to securely manage sensitive data in your serverless applications.
Chapter 10: Observing All the Things (planned): We’ll show you how to make your Rust Lambda functions observable using CloudWatch or OpenTelemetry. You’ll learn about monitoring best practices and how they apply to your serverless functions.
Chapter 11: Middlewares (planned): In this chapter, we’ll introduce the middleware pattern and show you how to handle cross-cutting concerns in an elegant and reusable way. You’ll learn about middleware best practices for your Rust Lambda functions.
Chapter 12: Hosting Existing HTTP Services (planned): We’ll teach you how to wrap existing HTTP services in a Lambda function to make them easily available as serverless functions. You’ll learn about the benefits and trade-offs of using this approach.
Chapter 13: Lambda Extensions (planned): We’ll dive into writing Lambda extensions using Rust.
Chapter 14: Integrating with GitHub Actions (planned): Learn how to configure the integration between AWS and GitHub Actions to test, build, and deploy your Lambda functions in an automated fashion.
Appendix A: IaC Alternatives (planned): Discover how to use CDK or Terraform as an alternative to SAM for managing your serverless infrastructure.

This list isn’t set in stone! If you think there’s a big topic missing or something you’d love to see covered, please let us know. We want to write the book you’d like to read about using Rust with AWS Lambda!

Ready to Start?

As we wrap up, it’s worth mentioning that this journey isn’t just about writing AWS Lambda functions in Rust, but it’s about diving deep into the serverless ecosystem. Along the way, we’ll explore DynamoDB, S3, EventBridge, and other building blocks of serverless architecture, all while building a real, production-ready application.

And of course, this journey isn’t one you have to take alone. There’s a thriving community waiting to support you, including our Discord server, where you can connect with like-minded developers, ask questions, and share your experiences. Whether you’re here for the Rust, the serverless insights, or the hands-on learning, I’m glad to have you along for the ride.

So, what are you waiting for? Dive in, experiment, and share your feedback. Together, let’s redefine what’s possible with serverless and Rust. Head over to rust-lambda.com and start your journey today.

Let’s build something amazing together!

2023 - A year in Review

Luciano Mammino — Wed, 27 Dec 2023 15:00:00 GMT

2023: End of another year!

Another year is gone and it’s time for another review of all the things I have accomplished professionally (and not) in 2023.

I feel the usual disclaimer is mandatory here: I generally write these posts for myself and I don’t expect other people to read them. So this is going to be a long boring post and I hope that if you are willing to read through it you are doing it from the comfort of your bed and that it’s going to be a good read for you before falling asleep.

First time at re:Invent

It’s hard to pick what was my favourite thing of the year, but I want to start by sharing that 2023 marked my attendance of re:Invent. re:Invent is the biggest conference organized by AWS and it takes place every year in Las Vegas. It’s a huge event with more than 60k attendees and it’s a great opportunity to learn about the latest news from AWS and to meet a lot of people from the community.

I admit I did not attend many talks but I managed to attend a few interesting ones and participate in different community events. This allowed me to focus on networking and meet so many amazing people from the community. I also had the opportunity to meet in person a lot of people I have been engaging with remotely for years and that was a great experience.

These are only a few of the many people I met! From the upper left: Andres Moreno, Franchesco Romero, Matthew Bonig, AJ Stuyvenberg, Alex DeBrie, Allen Helton, Jacopo Nardiello, Luca Bianchi, Paolo Latella, Monica Colangelo, Mario & Pacman, Allen Helton (again!), Luca Mezzalira, Jon Myer, Andrea Amorosi.

These are really only a few of the many awesome folks you can meet in an event like re:Invent and I wish I had taken more pictures!

I had the pleasure to be able to interview a few more people for an episode of AWS Bites Podcast where I asked them questions such as “How to get started with AWS and Serverless”, “What is your favourite and least favourite thing about AWS”, “What’s your bold prediction for the future of serverless”, “Multi-cloud, yes or no?”, and, of course “What about AI? How is it going to change things?”.

I also want to give a shout-out to a few folks who delivered some amazing talks that I’d recommend checking out:

In case you are curious, all the other talks are available on the official AWS Events YouTube channel. There’s also a dedicated playlist for all the breakout session talks.

A big thank you is mandatory to the AWS Hero program team: Taylor Jacobsen, Farrah Campbell, and Albert Zhao. They did an amazing job at organizing the AWS Heroes Lounge and the AWS Heroes reception. They also did a great job at making sure that all the AWS Heroes had a great time at re:Invent and that we had the opportunity to meet and network with each other. I am really grateful for the opportunity to be part of this amazing program and I am looking forward to the next re:Invent!

One last thing I want to mention is that re:Invent is a fantastic place for collaborating on content creation. I had the pleasure to collaborate with Chriss Williams doing a quick interview for his vBrownBag channel. I also spoke with Julian Wood at their ServerlessLand booth (yet to be published) and with Sam Williams from Complete Coding (yet to be published as well).

Life is beautiful not because of the things we see or do. Life is beautiful because of the people we meet.

— Simon Sinek

Middy

Now, let’s talk a bit about Middy!

Middy is a Node.js middleware framework specifically designed for AWS Lambda! I’ve been part of this project since the early days of Lambda (even though the very first public commit only happened on August 3, 2017), and it’s been quite a journey.

However, I must admit, I can’t take credit for the success of the framework over the last four years. All the kudos go to the dedicated efforts of Will Farrell, who has been tirelessly maintaining Middy, and of course, to the fantastic community that surrounds it!

The biggest news for Middy this year are the following:

Secured a substantial sponsorship from fourTheorem
Secured a significant sponsorship from AWS itself! This is massive also because AWS is now officially endorsing Middy as a framework for building Lambda functions as part of their AWS PowerTools suite.
Released a new major version: Middy v5.0!

I had the pleasure to contribute a bit to this new Major release by helping with improving the internal typings and by making some minor improvements to the website. I admit the TypeScript stuff was wild and I had to learn tons of tricks to make it work, but I am happy with the result and I am looking forward to contributing more to Middy in the future. A big thanks to all the people who helped through this massive PR. The Middy community is awesome and this is a testament to how much you can learn and grow by contributing to open source projects.

If you are using Middy and you want to know how to upgrade to V5, you can check out this dedicated Migration guide from Middy v4 to Middy v5.

But let’s talk numbers! Middy is constantly growing in its adoption, but how much?

In this graph we can see that, in 2023, Middy became the number one middleware framework for Lambda, surpassing @vendia/serverless-express. If I have to guess this is because Middy is purely focused on Lambda and it doesn’t just try to adapt the HTTP-first Express paradigm to Lambda. The fact that @vendia/serverless-express has been now moved to @codegenie/serverless-express might have been another contributing factor since the switch is fragmenting the data and most likely causing migration friction…

If we zoom-in on the download numbers for Middy, the numbers, IMHO, look quite rewarding:

Total downloads of the year: almost 18mln downloads (core only) + almost 78mln downloads including all the other 35 supporting packages (data)
A peak of almost 600k downloads/week for Middy core

If we look at some vanity metrics (GitHub stars count), there was some nice steady growth as well:

We are so close to 3600 stars, so if you use Middy and find value in it, consider giving it a star on GitHub and help us reach this vanity milestone! 😋

One final call to action: if you like Middy and care about its future, it would be only fair to consider sponsoring Will Farrell on GitHub. He is the one who is putting most of the effort into maintaining the project and he is doing an amazing job at it. 🙏

Side projects create the greatest companies because they begin as freedom machines, labors of love and truth vehicles. Precious time is traded to pursue something pure.

— Brian Norgard

Almost 3 years at FourTheorem

February next year will mark my third year as a Senior Architect at fourTheorem. It’s crazy to think that, because I am still as happy as I was in the first few months which I admit is something fairly unusual for me since I have been a bit of a job-hopper before changing jobs regularly every ~2 years. Now I am not just happy to stick around with fourTheorem, but I am also looking forward to years to come!

If you ask me why, I probably wouldn’t have a straight answer. Or, at least, I wouldn’t be able to pinpoint a single factor that contributes to my happiness. I think it’s a combination of many things: great culture, interesting projects (always challenging and with lots of variety), amazing colleagues both at human and technical levels, and a lot of freedom to experiment and grow. Finally, I feel that the work we do is meaningful and has a positive impact on our customers and the community. And looking back at my career, I think this is something that is not obvious at all in the tech world.

Most of the cheerful team at fourTheorem

Some of the things we have done in 2023:

Welcomed 2 new colleagues: David Lynam, Senior Cloud Architect, and Conor Maher, Senior Infrastructure Engineer and organiser of the AWS South East User Group in Ireland.
Our team spoke at over 30 conferences in 4 different continents, including AWS Cloud Day Dublin and the AWS AI & Data Conference, where Eoin Shanaghy (our CTO) conducted a one-on-one interview with Werner Vogels, CTO of Amazon.
We celebrated two wins at the JSDAY Awards 2023: Best Place to Work with JavaScript in Ireland and Most Valued JavaScript Open Source Project for Middy.
We proudly became an Amazon ECS Service Delivery Launch Partner, validating our expertise in Amazon ECS and AWS Fargate.
We open-sourced some new very cool projects such as podwhisperer for automating podcast transcripts and episoder for automating chapters, tags, titles, and video descriptions from transcripts.
We also kept investing in an open-source project we launched last year: slic-watch for automating observability on AWS which got mostly rewritten in TypeScript (thanks to all the efforts of our colleague Diren Akkoc).

What can we expect in the next year? Hard to say for sure, but I am sure it’s going to be exciting and rewarding!

I am also happy to see that the company is growing and that we are always on the lookout for talent. If you like the cloud and you think you might be interested in joining us, reach out to me and/or check out the fourTheorem careers page.

Finally, if you and your company are looking for help with AWS and the cloud, whether you are doing a cloud migration, an app modernization, building a scalable MVP for a startup idea or trying to optimize your cloud bill, reach out to us and we’ll be more than happy to help!

It does not matter how slowly you go as long as you do not stop.

— Confucius

Public speaking

I am not sure if I can call myself a public speaker, but I do enjoy speaking at conferences and meetups. I think it’s a great way to share knowledge and to meet new people. I also think it’s a great way to step out of your comfort zone, challenge yourself and grow as a person. If it’s something you haven’t done yet, I would recommend considering it.

I have been doing public speaking since 2016 and in 2023 I reached a total of 137 talks! 🎉

This is me delivering a talk about JavaScript iteration protocols at JSNation in Amsterdam. What an awesome experience!

One of my favourite speaking events was to deliver my first talk at a Rust conference (RustNation UK)! 🦀 This is really important for me because I feel I am still at the very beginning of my Rust journey and being able to speak in front of a crowd interested in Rust and possibly more knowledgeable than me was a great experience. I am looking forward to doing more of this in the future!

The talk is called “What I learned by solving 50 Advent of Code challenges in Rust”. It had a great reception and it’s available on YouTube with more than 12k views and tons of comments!

Spoiler, I’ll be presenting at RustNation UK again in 2024! 🎉

Here’s the full list of talks I delivered in 2023:

Interview with Caffé Sviluppo (Podcast interview in Italian, remote) (Video)
Building an invite-only microsite with Next.js & Airtable (talk at Global Summit for Node.js, remote) (Slides, Video)
What I learned by solving 50 Advent of Code challenges in Rust (talk at Rust Nation, London) (Slides, Video)
Serverless su AWS (podcast interview with Gitbar podcast, remote, Italian) (Episode)
Did you know JavaScript has iterators? (talk at DublinJS meetup, Dublin) (Slides)
The evolution of Async JavaScript and its patterns (talk at Node.js ONE Africa conference, remote) (Slides, Video)
AWS Application Composer e Amazon CodeCatalyst (talk at AWS User Group Roma, remote)
A modern application is ASAP (talk at AWS Cloud Day, Dublin)
Your Lambdas, In Rust! (talk at Conf42: Cloud Native, remote) (Slides, Video)
Production-ready lambdas with Node.js (talk at JSDay Verona) (Slides)
A serverless architecture for high performance financial modelling (talk at Serverless Architecture Conference London) (Slides, Video)
Your Lambdas, In Rust! (talk at AWS Fest remote) (Slides, Video)
JavaScript Iteration Protocols (talk at JSNation Amsterdam) (Slides, Video)
AWS On Air (interview at AWS Summit London) (Video)
Production-ready lambdas with Node.js (talk at Sailsconf) (Slides, Video)
The senior dev: an opinionated take (remote talk at Java Challenger) (Slides, Video)
Easier Observability with SLIC Watch (remote talk at Serverless Office Hours) (Slides, Video)
Rust, Serverless and AWS (talk at Rust Dublin) (Slides, Video)
Automating observability on AWS with SLIC Watch (talk at Ship It Conference Dublin) (Slides
Chats with “La locanda del tech” (remote panel - in Italian) (Video)
Easier Observability with SLIC Watch (talk at ServerlessDays Belfast At Night) (Slides, Video)
Lambda in Rust (talk at AWS User Group Dublin) (Slides)
Did you know JavaScript has iterators? (talk at JavaScript Global Summit) (Slides)
Everything I know about S3 pre-signed URLs (talk at AWS Community Summit Manchester) (Slides, Video)
Did you know JavaScript has iterators? (talk at Node.js Conference) (Slides)
Your Lambdas, In Rust! (talk at Codemotion Milan Milan) (Slides, Video)
Your Lambdas, In Rust! (talk at AWS User Group South East Waterford) (Slides)
AWS Serverless Hero at re:Invent (interview with vBrownBag) (Video)
Serverless Tips from an AWS Hero (online interview) (Video)

What do you think? Is there any topic here that catches your attention? If so, let me know and I’ll be happy to work on more content around the same topic!

It usually takes me more than three weeks to prepare a good impromptu speech

— Mark Twain

AWS Bites

AWS Bites is a weekly podcast that Eoin Shanagy and I started in 2021. We are now in our third season and we are still growing steadily! 🎉

Here’s Eoin and I during the fourTheorem Christmas dinner. Our colleague Conor made for us this awesome cutout so that we can feel like real podcasters! I am doing the Italian pinch gesture, because I am Italian, so I am entitled to it! 😅

This year we released 46 episodes which is almost one every week! It feels great to have been so consistent! We also surpassed the 100-episode mark! 🎉

Let’s have a quick look at our YouTube numbers for 2023:

We more than doubled our number of views from last year reaching ~39k views.
We also doubled the watch time reaching 3k hours of watch time.
We have grown our subscriber base by ~40% reaching ~2.5 subscribers.

Our followers seem to have enjoyed the following 3 episodes the most:

Now let’s compare these with some podcast numbers, based on what Spotify gives us:

We reached almost 60k plays in 2023, which is almost double the number of plays we had in 2022.
We streamed in 79 different countries (with USA, UK, India, Canada and Germany being in our top 5).
We have grown our follower base by 123% reaching ~2k Spotify followers.
The favourite episode was: 60. What is AWS Lambda?
The most shared episode was: 83. Bucket List of S3 Best Practices

If you are curious to see more details you can check out our Spotify for Podcasters wrapped 2023 YouTube short.

If you enjoy AWS Bites, let us know what you think about it, what can we do to deliver more value to you, which topics you care about, etc. We are always looking for feedback and we are always happy to hear from our listeners on what we can do better!

If you have knowledge, let others light their candles in it.

— Margaret Fuller

Live coding on Twitch

In 2023 I continued my activity of live coding on Twitch together with my friend Roberto Gambuzzi. I actually want to take the opportunity to give Roberto a big shout-out not just for the great company during these live coding streams (where he always brings a huge amount of expertise and some great fun too), but also for keeping me on track with streaming (more or less) regularly! So, yes, Thanks Roberto! 🙏

Let’s now share some stats:

Compared to last year we had fewer streams (30 vs 36), fewer new followers (150 vs 250) and fewer hours watched (152 vs 223). A bit of a downward trend, but to be honest I am not too worried about it. We don’t stream because we have some kind of master plan to grow a huge community, but more as an excuse to keep learning about Rust and getting better as software engineers. In that sense, I think that the time spent doing this activity has been absolutely well spent!

This year we focused much less on Advent of Code and we tried to invest more time learning about new interesting Rust crates and working on a few side projects. Here’s a summary of some of the most interesting streams (with a link to the respective recordings on YouTube):

If you think these kinds of topics are interesting, make sure to follow us on Twitch and check out the recording on YouTube.

Finally, a big shout-out to all the people who joined us during the streams and helped us with their comments and suggestions. We really appreciate it and we are looking forward to seeing you again in 2024! 🙏

There’s progression you don’t see and feel. There’s a starting point, and with time and focus. As long as you reevaluate and reassess, and constantly objectively look at what you’re doing, and then pursue it with passion and focus, you get better at things.

— Joe Rogan

Node.js Design Patterns

Node.js Design Patterns is a book I co-authored with the amazing Mario Casciaro. It is s still doing quite well even after 3.5 years since the third edition was published.

The thing that is most rewarding about this book is that it keeps getting praised as one of the best resources to go from beginner to master with JavaScript and Node.js. I am really happy to see that the book is still relevant and that it’s helping people to learn and grow!

In 2023 we kept getting great reviews and we currently have 4.7/5 stars on Amazon (compared to 4.6 last year) and a total of 279 reviews (compared to 227 last year).

Let me just quote a review from 2023 that I particularly liked:

An excellent book, highly recommend it. Extremely useful, I read it 2 times and planning for the third one.

And here’s another one from GoodReads:

As a CS student who just wants to get into backend development with Node.js, all the time a hear terms like microservices, messaging, streams, containers, event loop, and caching, and I just get overwhelmed by the many terms I hear, until someone recommended this book to me, and it just guided me where everything falls in place, it’s not comprehensive it helps you explore and do small examples on how to do these stuff with Node.js. Trust me, it’s not just another “Design Patterns” book :)

At the time of writing the book is also doing quite well in a few categories:

#110,600 in the global Amazon ranking (this one doesn’t mean much but it’s always a funny number to observe)
#18 in JavaScript Programming (compared to #22 last year)
#23 in Web Services (compared to #11 last year)

The book covers the most modern design patterns to make sure you write efficient and scalable Node.js and JavaScript code. It is also up to date to use modern language features such as ESM (EcmaScript modules), Async Iterators, and modern libraries and frameworks. The book targets Node.js 14 which is now deprecates, so, altough I believe the book is still very modern and relevant, it might be a good time to start thinking about a new edition. I am not sure if I can commit to it yet, but I’ll keep you posted if something happens!

Meanwhile, if you have read the book, what do you think? Is there something missing? Something else you’d like to see in it? Let me know!

A book is a garden, an orchard, a storehouse, a party, a company by the way, a counselor, a multitude of counselors.

— Charles Baudelaire

FullStack Bulletin

FullStack Bulletin is a free weekly newsletter about full-stack web development. I have been running this project with my dear friend Andrea Mangano for the last 6 years, now.

A screenshot of one of the latest issues of FullStack Bulletin - Issue #352 published on 2023-12-25

In 2023 we reached more than 350 published issues and a total of 3000 subscribers (compared to 2500 last year)! 🎉

This year we invested a bit in improving our layout, adding more variety to our content, improving the automation around the newsletter, and, generally speaking, we tried to make the newsletter messaging more personal and direct.

We are still running this project at a loss (Mailchimp is expensive! 💸), just for the sheer love of sharing full-stack content that we find interesting (and a bit selfishly as a tool to keep ourselves up to date with the news of the ever-evolving world of web development).

If you are interested in full-stack web development, make sure to subscribe to FullStack Bulletin and let us know what you think about it!

You can also support the project financially by sponsoring an issue! We can offer a sponsored banner, and a sponsored article slot. We can even discuss custom sponsorship packages. Reach out to us if you are interested in knowing the details.

A bit shout-out to our sponsors in 2023: MisterDA, ConfigCat, and Packt. Thanks for supporting us! 🙏

Curation is a natural and necessary extension of content creation. That is, as great as your content may be, your audience wants to learn from other experts and differing perspectives.

— Pawan Deshpande

LifeFoliage: the Landscape Hunt game

In 2023, I was able to carve out some of my free time to support the LifeFoliage project. LifeFoliage is a project funded by The Life Programme, the EU’s funding instrument for the environment and climate action. The project is very active in the Italian territory where it is focused on helping the local ecosystem to preserve and protect the local flora and fauna in a variety of ways.

One of the things LifeFoliage does is monitor and raise awareness about the local flora and fauna. They do that by routinely performing remote sensing through satellite images, which is something that allows them to spot changes in the vegetation and to identify things like fires or deforestation activities.

In this context, they wanted to create a publicly available game that could capture the essence of their work. This is how Landscape Hunt was born.

A screenshot of a game session of LandScape Hunt

Landscape Hunt is a game where you are presented with a series of satellite images and you have to spot guess what’s the right classification for that image. Are you looking at a fire, a forest cutting, a deciduous forest, a quarry or something else? It’s not that easy to guess correctly if you are not trained, so the game can be quite fun to play.

From a technical perspective, this was my first production Rust project, in fact, it features a Rust backend (written with Axum) while the frontend is written using SolidJS and TypeScript.

A big thanks to the team responsible for LifeFoliage and Landscape Hunt for involving me in this awesome project.

The greatest threat to our planet is the belief that someone else will save it.

— Robert Swan

Articles & blog

In 2023 I kept writing articles occasionally on this blog and on other platforms. Here’s the full list of all the articles published in 2023 in this blog:

I also wrote a few articles for other platforms:

The illustrated guide to S3 pre-signed URLs (published on fourTheorem blog)
The case for Serverless Rust on AWS (published in the AWS Comsum Quarterly Newsletter)
La definizione di Sviluppatore Senior (ita) (published on Codemotion’s blog as a liberal translation of my article “The definition of Senior”, kindly translated by Mauro Bardetta).

I was going to publish some stats about the blog, but the migration to Google Analytics V4 has messed up all my historic data and I am left with only ~18K user sessions from July 2023! Sigh. 😭

Blogging is to writing what extreme sports are to athletics: more free-form, more accident-prone, less formal, more alive. It is, in many ways, writing out loud.

— Andrew Sullivan

Open Source

I always like to do some open source whenever I have the chance. I like to contribute to projects I use (even in small ways) and I like to share almost everything I do in an open-source fashion. No surprise that I kept my commit race going in 2023! 🎉

Here’s a list of my most meaningful contributions of 2023:

Various contributions to lmammino/jwt-cracker (mostly merging PRs and general cleanups and improvements). Also, this is the first project where I introduced Biome a new Rust-powered JavaScript/TypeScript formatter and linter which is quite promising!
Continued working on some Advent of Code challenges in Rust (mostly during December 2023). All my solutions are in lmammino/rust-advent.
lmammino/lexicoid: a simple Rust crate implementing short & stable IDs based on timestamps. Based on a Go library (brandur/sorg).
eslint-community/eslint-plugin-security my PR co-authored together with Simone Sanfratello was merged making ESLint able to spot some possible security issues in copy-pasted code.
Some small contributions to an awesome Rust workshop by @jfacchini that I had the fortune to partecipate to during Rust Nation UK 2023.
lmammino/lastfm: An async Rust client to fetch your Last.fm listening history or the track you are currently playing.
lmammino/image-overlay-example: A very simple rust script showcasing how to create an image by overlaying two existing images.
lmammino/unzip-and-resize-images: A simple rust-based CLI that can read images from a zip file, resize them and save the resulting images in a local folder.
lmammino/aws-application-composer-demo: An AWS Application Composer project created during a LIVE demo at the AWS User Group Roma.
lmammino/earthquake-notifier: Lambda in Rust to monitor for earthquakes and generate EventBridge events.
lmammino/mvp-contributions: a CLI tool to review and submit your Microsoft MVP contributions using a YAML file. I was very proud of this one because it saved me so much time when I had to submit my MVP contributions. For better or worse, Microsoft has recently updated the MVP program website and this tool no longer works. I’ll need to find some time to update it to use the new APIs.
lmammino/solidjs-template: A dead simple Solid.js starter template (using npm, ESLint, StandardJS, and TypeScript).
lmammino/zig-python-experiment: A simple Python module written in Zig (mostly work done by Roberto Gambuzzi).
Fixed a small typo in rafaelcaricio/backie, a library to do background task processing for Rust applications with Tokio, Diesel, and PostgreSQL.
lmammino/pyo3-experiment: an example project where we used PyO3 to run Rust code from a Python module.
lmammino/http-check: A dead-simple Lambda to demonstrate some best practices for Node.js Lambdas like logging, custom metrics, testing, etc.
lmammino/playing-with-neon: Some simple experiment with Neon bindings: generating Node.js libraries with Rust
lmammino/js-iter-examples: Code examples from my talk “JavaScript iteration protocols”
FullStackBulletin/tech-quotes: An API to get interesting tech-related quotes (powered by GitHub pages).
Various improvements to lmammino/jwtinfo: A command-line tool (written in Rust) to get information about JWTs (Json Web Tokens).
lmammino/tapo-l920-on-off: A very simple Rust binary that can turn on/off a TP-Link L920 led light strip in your local network.
lmammino/twitch-chat-bot: An (incomplete and experimental) Twitch Chat bot written in Rust.
Contribution to Il-Libro-Open-Source/book: an Italian open-source project that aims to become an open book about best practices and suggestions on how to become a good software developer. I contributed by writing an entire chapter about the skills that are needed to be considered a great Senior software developer.
lmammino/oidc-authorizer: A high-performance Lambda authorizer for API Gateway that can validate OIDC tokens (written in Rust).
Small contribution to aws-samples/serverless-patterns making Node.js imports consistent.
lmammino/whisper-rs-example: An example of how to use Whisper.cpp bindings for Rust to perform speech-to-text on WAV audio files.
A minor contribution aws/aws-cdk making sure that a specific edge case would provide good logging. I had quite some fun working on this one together with my colleague Conor Maher and I am very proud of this contribution since we use CDK quite a lot at fourTheorem.

To close this section here’s my ritual GitHub yearly contribution graph! I know it’s just vanity, but it’s fun! 😀

I often compare open source to science. Science took this whole notion of developing ideas in the open and improving on other peoples’ ideas. It made science what it is today and made the incredible advances that we have had possible. And I compare that to witchcraft and alchemy, where openness was something you didn’t do.

— Linus Torvalds

Other stuff

Some other quick professional news (even though they are not really news).

Still an MVP

In 2023 I was confirmed as a Microsoft Most Valuable Professional for Developer Technologies and Security. This marks 3 years as an MVP! 🎉

Unfortunately, I cannot show you the usual MVP tower trophy where I am adding another disc because my parcel got lost in transit and I haven’t been able to schedule another shipping with Microsoft 😢. BTW, if you work with Microsoft (or you know someone who can help), please let me know. I have sent countless emails and Teams messages already but it seems they are not reaching the right people… 🙁

Still a Codemotion Ambdassador

In 2022 I became a Codemotion Ambassador.

Codemotion is the biggest tech conference (and tech community) in Italy and one of the biggest in Europe. I have been speaking at Codemotion conferences for many years now and I have always been a big fan of their events. I am very happy to be part of the Codemotion Ambassador team and even more to still be part of this group in 2023!

In this picture some members of Codemotion and the Codemotion Ambassador team during the after-party at Codemotion Milan 2023. From left to Right: Michela Bertaina, a random guy, Mattia Tommasone a.k.a. “Mr. DJ”, Gabriele Santomaggio, Francesco Sciuti a.k.a. “Batman”, Emanuele Bartolesi a.k.a “The Machine”, César Alberca, Mara Marzocchi.

Thanks for having me in this awesome community! I look forward to contributing more to it in 2024!

… Also, a tiny extra thank you to Emanuele Bartolesi for reminding me to write this post! 😝

The future of every community lies in capturing the passion, imagination, and resources of its people.

— Ernesto Sirolli

Some random personal stuff

Now, onto some other more personal achievements of 2023 🦸🏻

After 2 years pandemic hiatus from Brazilian Jiu Jitsu (BJJ), this year I finally restarted my training and towards the end of the year, I received the grade of purple belt! 🥋
I took a wonderful road trip visiting some of the most iconic national parks in California, Arizona, Utah and Nevada with my wife and it was probably the best trip I (and we) ever had so far! ❤️
Kept running (even though less regularly than I did in the last 2 years) totaling ~160Km in 27 runs. 🏃

One of my pictures from a visit to the Horseshoe Bend, a horseshoe-shaped incised meander of the Colorado River located near the town of Page, Arizona, United States

Enjoy the little things in life, for one day you may look back and realize they were the big things.

— Robert Breault

Wrapping up

That’s all folks!

I am impressed to see that you got this far. Maybe it wasn’t as boring as I had imagined 😇

It’s great to reflect on everything that happened in 2023 and start to think about what to invest my time and energies in 2024. I am not sure what the future holds, but I am sure it will be fun and exciting!

What about you? What did you achieve in 2023 (or missed)? What’s your masterplan (if any) for 2024? I hope I’ll be reading about it in your own year in review post or in the comments below! 😉

Bye 👋

2024 - A year in Review

Luciano Mammino — Sun, 09 Feb 2025 20:12:00 GMT

2024: Another Year Bites the Dust (in a Good Way!)

Another year has come and gone, leaving behind a trail of commits, conferences, and hopefully, fewer bugs than it introduced! 🐞 I have been meaning to write this review for the last 2 months, but it has been an intense last 2 months, both personally and professionally! Finally, I am finding some time today, so I am going to use it wisely to do this boring task! Yes, the boring task is my annual review of the things I’ve managed to accomplish (or at least attempt) in 2024, both professionally and personally.

As usual, a disclaimer: this post is primarily for my own record-keeping and reflection. I don’t expect anyone to actually read it, but if you do, I hope you’re comfortable, have a beverage of choice, and find it mildly entertaining enough to not induce instant sleep. Think of it as a digital lullaby for the tech-minded.

Launching “Crafting Lambda Functions in Rust” (e-book)

Okay, the title might be a tad misleading! This isn’t a proper launch in the traditional sense, since the book is very much still a work in progress. But it’s also not entirely inaccurate, because the book is already available online, and you can start reading it right now. If you want to dive even deeper into how this book came to be, I wrote a dedicated article about it on this blog.

But let me tell you the story (or at least a condensed version)…

Back in March 2024, I bumped into the amazing James Eastham, again! This time it was at Rust Nation in London. I’ve been following James’s work for a while now, because I love his content and because we both share a keen interest in exploring the world of Rust, especially when applied to serverless architectures.

As we were chatting at the conference, it became apparent that we were both individually thinking about creating a more structured resource around this topic. After comparing notes, we decided to join forces and build something together! If you’re as passionate about serverless and Rust as we are, you’ll want to check this out!

It didn’t take long for us to decide to go with a self-published book. We started drafting ideas for chapters and shaping the overall messaging of the book. We picked a title, designed a website, created a store page on Gumroad, and even created a mascot, which you can appreciate in all its full glory on our book cover. I’m sure you can spot the various bits of inspiration we drew from! Yes, our mascot is a crab (a cRustacean, get it?) with a Lambda hat. If the hat reminds you of something that’s a total coincidence… Mamma mia!

Now, let’s talk a bit about the publishing model we’ve chosen. We could have tried to find a traditional publisher, but our previous experiences tell us that this isn’t generally a viable way to make a project economically sustainable (except for the publisher, of course… 🤫). Plus, we wanted total freedom over the release schedule and how we interact with readers, offer discounts, and so on. That’s why we decided to go the self-publishing route. This gives us full control over how we interact with the community!

Also, in order to keep ourselves committed and to have a steady stream of feedback and reviews, we decided to make the book available as soon as we had the draft of the first two chapters ready. We’re embracing the “early access” approach!

So you might be asking yourself… Why Rust and Lambda? Well, the combination of Rust and AWS Lambda is a game changer. As a compiled language, Rust produces highly efficient binaries that are compact and lightning-fast. This means your Lambda functions run quicker, consume less memory, and ultimately cost less to execute. You can potentially lower both execution cost and resource usage, leading to a lower carbon footprint. You also get reliability and correctness as Rust’s strict compiler and type system are like having an extra pair of eyes on your code. Combine that with Cargo Lambda, a fantastic tool that streamlines the developer experience by handling scaffolding, testing, compilation, and deployment, and you have an incredible tool chain!

As of today, we have six chapters available covering everything you need to know to build and ship a fully functional URL shortener to AWS using Rust, Lambda, and other serverless technologies (e.g., DynamoDB, API Gateway). In those 6 chapters you will learn everything from setting up your environment, creating your first “Hello World” lambda and using SAM to manage your code, to testing the lambda and making it interact with external systems.

There’s still plenty of work to be done, but we plan to finish (or get close to finishing) this book by the end of 2025. This book is your hands-on guide to building efficient, scalable, and cost-effective serverless applications with AWS Lambda functions and using the power of the Rust programming language. You can even get involved by sharing your feedback, help shape the final version, and, if you like, get your name on the book by purchasing the currently available early release bundle!

If you want to find out more, check out our website, and if you want to stay up-to-date with the project, we also have a newsletter and a Discord server. We’re excited to build something amazing with you!

Reading maketh a full man; conference a ready man; and writing an exact man.
— Francis Bacon

Almost 4 Years of Cloud Adventures at fourTheorem

It’s almost been four years since I joined fourTheorem as a Senior Cloud Architect! In fact, as I’m writing this, that “almost” is just a week away! 🎉 They say time flies when you’re having fun, and I can certainly say that time has been flying for me when I think back on all we’ve accomplished at fourTheorem in these four years.

In 2024, I was fortunate enough to help several more enterprises achieve success with AWS and Serverless, and I learned a ton in the process. I’m currently working on a full-stack serverless project that has been incredibly rewarding to build. It’s led me to explore a lot of new things around serverless architectures, TypeScript, OpenAPI, and more. I’m hoping to be able to share some of these learnings in a public form soon! Stay tuned for that.

From a company perspective, fourTheorem is growing strong. So, if you’re passionate about the cloud and looking for a challenging and rewarding career, check out the fourTheorem careers page or reach out to me directly. We’re always looking for talented individuals to join our team. Come join the fun!

2025 is already looking quite busy and I look forward to seeing what we will be building with our clients and the team at fourTheorem!

Try to be a rainbow in someone’s cloud.
— Maya Angelou

This year, I’ve really enjoyed attending various conferences and delivering talks. It’s one of the things I love most about my job. It doesn’t just give me the opportunity to share what I’ve learned; more importantly, it gives me the chance to meet likeminded people and learn from them. Building a network of passionate software engineers is one of the most precious assets you can have in this line of business.

So, I would encourage everyone to attend conferences. You don’t necessarily have to try to deliver a talk if that’s not your thing, but attending should be an absolute no-brainer! If you can take one thing away from this (admittedly selfish) article, it’s to plan a few conferences to attend this year. And if you think conferences are too expensive or too much of a commitment, you can always start with a local meetup… Thank me later! 😉

Here’s a look back at the talks I delivered in 2024:

01-23: Re:Invent recap (panel), BelfAWSt User Group, Belfast
01-30: AWS re:Invent re:Cap (panel), AWS User Group Dublin
03-06: An intro to nom, parsing made easy for Rustaceans, Rust Dublin (Remote) (Slides, Video)
03-19: Serverless Rust su AWS Lambda “for fun and profit” (Workshop), Codemotion Workshop Fest (Remote) (Slides)
03-28: Rust, Serverless, and AWS - Writing Lambdas in Rust, Rust Nation UK, London (Slides)
05-07: Your Lambdas, In Rust!, AWS Community Nordics, Copenhagen (Slides, Video)
05-23: Building Secure and Efficient SaaS Platforms on AWS Serverless, ServerlessDays Belfast (Slides, Video)
06-20: Rust on AWS Lambda, Rustship podcast (Remote) (Video)
08-14: Rust with Lambda, easy-mode Rust & future of Middy, Real World Serverless podcast (Remote) (Video)
08-20: Harder, better, faster, stronger Lambda functions with Rust, Believe In Serverless (Remote) (Slides, Video)
09-06: Faster/Greener/Cheaper Serverless compute with Lambda & Rust, AWS Community Day Belfast (Slides)
09-16: Rust and Serverless Computing, Mainmatter podcast (Remote) (Video)
09-17: Building Secure and Efficient SaaS Platforms on AWS Serverless, AWS Community Day DACH (Slides)
09-27: Managing AWS accounts like a PRO, AWS Community Day Italy, Rome (Slides)
10-03: Building Secure and Efficient SaaS Platforms on AWS Serverless, AWS Community Day NL (Slides)
10-22: Building Secure and Efficient SaaS Platforms on AWS Serverless, Codemotion Milan (Slides)
10-23: Unleashing Serverless Performance: Writing Optimized AWS Lambda Functions with Rust (Workshop), Codemotion Milan (Slides)
11-26: Faster/Greener/Cheaper Serverless compute with Lambda & Rust, Porto Tech Hub (Slides)
12-12: Optimized Lambda functions with Rust (Workshop), Serverless Days Rome (Slides)
12-13: The role of the serverless dev (Panel), Serverless Days Rome

This brings my total number of talks to 157 in 2024. Crazy to think I have done so many!

If you are curious to see the full list and check out videos and slides, you can find them all in the speaking section.

Also worth mentioning that I attended re:Invent again this year, and it was a blast! I got to meet so many amazing people from the incredible AWS community and learned a ton from them. Here’s a randomly selected picture!

From left to right: Anurag Kale, Ivan Casco Valero, Marcus Bladh, it’s a me!

In 2025, I think I will deliberately slow down a bit on the speaking front. I want to reduce the amount of travel and focus more on other side projects, but I will probably still be speaking at a few conferences. So if you want to invite me to speak at your event (especially if it’s a remote one), please reach out to me.

No grand idea was ever born in a conference, but a lot of foolish ideas have died there.
— F. Scott Fitzgerald

Node.js Design Patterns: Still a Relevant Resource After All These Years

Even after four years on the shelves (and in online stores), the third edition of Node.js Design Patterns is still going quite strong. We’re still getting plenty of good reviews and hearing from happy readers.

We are currently holding a solid 4.6 stars with over 300 reviews on Amazon! It’s incredibly rewarding to know that our work is still helping developers improve their skills.

In 2024, we even had some great people in the JavaScript community talking about the book, which gave it another boost in visibility. A huge thank you to everyone who has recommended it to others!

Matt Pocock (Total TypeScript) showcasing his copy of Node.js Design Patterns on LinkedIn

Because the book features Node.js 14, many people ask if it’s still up-to-date and relevant. My answer is absolutely yes! All the patterns we focus on are pretty much timeless, and if you’re just starting with Node.js and JavaScript and want to ramp up your knowledge, there’s still plenty to learn from it.

Of course, there have been some minor improvements in the Node.js and JavaScript landscape since this edition was launched, so there might be a few opportunities here and there to make the book even more relevant and up-to-date. I cannot disclose anything specific right now, but I can only say: stay tuned! 😉

Meanwhile, if you’ve read the book and appreciated it, please leave us a review on Amazon or Goodreads. This is the single best thing you can do to help us and encourage us to invest more in this kind of activity. Your feedback means the world to us!

The future of publishing is about having connections to readers and the knowledge of what those readers want.
— Seth Godin

AWS Bites: Still Serving Up Serverless Goodness (Now in Bi-Weekly Portions!)

AWS Bites, the weekly podcast that Eoin Shanagy and I started in 2021, is now in its fourth season, and we’re still going quite strong! 💪

While we reduced our publishing frequency this year from weekly to bi-weekly, we still managed to release 29 new episodes packed with AWS and serverless insights.

Looking at the stats provided by Spotify, we reached 140k plays with an average of 775 listeners per episode, which I think is quite rewarding.

On YouTube, we have 3800 subscribers and almost 50k views in 2024 alone! Thank you to everyone who listens and watches! 🙏

The episodes that people seem to have enjoyed the most are:

On YouTube: Ep. 111 - How We Run a Cloud Consulting Business
On Spotify: Ep. 120 - Lambda Best Practices

Another cool thing we did in 2024 was to bring in more people from fourTheorem. We had the pleasure of having our CFO, Fiona McKenna, to discuss the financial aspects of running a cloud consulting business. We also had Conor Maher to talk about how to get rid of those annoying IAM users and improve the security and governance of AWS Accounts, and David Lynam talking about AWS Transit Gateway.

We were also super pleased to have had the opportunity to interview the one and only Farrah Campbell and get her to share her history and some suggestions for people who want to grow in tech. If you are curious to know more, check out Ep. 130 - Growing in Tech with Farrah Campbell.

AWS Bites is something we will surely keep investing on in 2025 because we think it’s providing value to the AWS Community, and it’s a great place for us to share what we learn every day while working with AWS.

If there’s a topic you think we should cover, or if you have any piece of feedback you want to share with us, please reach out and let us know! We’re always looking for new ideas.

You’re going to be terrible at a lot of things for years until you’re successful. People probably aren’t going to listen to your podcast initially. But if you like it and you keep putting it out, people will find it.

— Keith Kingbay

Middy: Still the Leading Middleware Framework for AWS Lambda

Middy is a Node.js middleware framework specifically designed for AWS Lambda! I started this project during the early days of Lambda around 2015 (even though the very first public commit only happened on August 3, 2017), and it’s been quite a journey.

Although, a few years back I found myself struggling with maintainer duties, and I passed the ball to Will Farrell, one of the most active contributors, who has since stepped in as a maintainer (and has done a far better job than I could ever dream of doing!).

So today, anything you see happening around Middy is Will’s merit. I only like to talk about it to see how the project is doing, but I can’t really claim any credit! I only have a few occasional chats per year with Will, so you could say I have a very distant role as an advisor, at best…

The main news of 2024 is that Middy v6.0.0 was released in November, bringing the following to the table:

Deprecation of Node.js v18.x
Addition of support for Node.js v22.x
Addition of support for --experimental-require-module

If you want to read the full list of changes and how to upgrade from v5, check out the official Middy 5 to 6 migration guide.

In terms of stats, Middy is still the leading middleware framework for AWS Lambda in the JavaScript/TypeScript landscape, with an average of 360k downloads per week (of the core module alone). It’s awesome to see such continued adoption!

It has almost reached 4000 stars on GitHub (3762 right now), so if you like Middy and want to give us a little hand, give it a star on GitHub – it costs nothing! Make sure to also reach out to Will and consider sponsoring him to support his incredible work (not just about Middy). He’s the one putting in the hard work to keep Middy going strong.

I have yet to catch up with Will in detail, but from what I could tell, his focus for this year will be on security hardening, stability, and improving the open-source processes around the management of the project. These are all great priorities to keep Middy a robust and trustworthy framework.

Software is a great combination between artistry and engineering.

— Bill Gates

Streaming Technology and Learning Rust on Twitch: Fun, Experiments, and (Hopefully) Some Learning

In 2024, I kept streaming live coding sessions on Twitch with my dear friend Roberto. These streams have become a great way for us to explore new technologies, learn together, and generally geek out about code.

We completed a few interesting projects this year, including:

Building a parser for the REDIS protocol using the parser combinator nom
Building an MVP for a lead generation system only using Rust, Lambda, DynamoDB, and SES on AWS (tinykit)
Building a BitTorrent client using Rust following one of the CodeCrafters guided tutorials

Overall, I don’t think we had a massive growth in terms of following or views and I don’t even care enough to check the numbers because this honestly isn’t why we do these streams. We just do them to have an excuse to keep experimenting and learning together, and so far, I think this has been working pretty well for us. It’s more about the journey than the destination!

So, we’ll probably keep these going on a semi-consistent frequency in 2025. If you’re interested in hanging out with us during these sessions, check us out on Twitch. Otherwise, you can check out all our recordings on my YouTube Channel.

Come learn Rust with us (and maybe teach us a thing or two)!

Tell me and I forget, teach me and I may remember, involve me and I learn.

— Benjamin Franklin

My Open Source work in 2024

I always like to do some open source whenever I have the chance. I like to contribute to projects I use (even in small ways) and I like to share almost everything I do in an open-source fashion.

Looking back at my GitHub history, I actually did way more than I originally thought, so it’s probably a good thing to do this kind of reflection to see how much I did actually accomplish over the year! It’s easy to lose track of the smaller contributions, but they all add up.

Here’s a summary of my main open-source contributions of 2024:

Small contributions to Cargo Lambda
I created fm.loige.co, a simple Rust and Lambda powered API to display the song that I am currently listening to on this very website (in the home page). It’s powered by Last.fm
Tinyresp, a parser for the Redis protocol written as a Rust library
Mailchimp List Janitor: a Rust based tool that allows you to remove unsubscribed users from your Mailchimp lists and save money in the process (a bit in an incomplete state)
OBS-countdown: A Rust-powered CLI tool that allows you to manage a countdown file that you can use to display a countdown in OBS
MVP Activities: A simple client to interact with the Microsoft MVP activities API (not official)
A small contribution to the new Node.js website
oidc-authorizer: A high-performance token-based API Gateway authorizer Lambda that can validate OIDC-issued JWT tokens. It’s written in Rust and it also comes with a repo for the benchmarks against a python-based alternative
Tinykit, a super simple and mostly incomplete lead magnet system written using Rust and Lambda
svg-invert: A simple Rust-based CLI utility that inverts colors in an SVG file:
A Small contribution to the AWS cloudfront-hosting-toolkit project to make their CI a bit faster
org-formation-sso-import: A TypeScript based CLI tool to import AWS SSO groups, permission sets and assignments into an OrgFormation workspace
Added free text search on the AWS Bites website using Orama
A rust + Lambda workshop

Here’s a screenshot of my GitHub contribution chart, just for vanity, although it’s slightly skewed in time since I am taking this in Feb 2025 😉

The power of Open Source is the power of the people. The people rule.

— Philippe Kahn

Articles & Blog

I didn’t write very much on this blog in 2024. I only managed to publish 3 articles. Quality over quantity, right? 😉

However, I did spend a good bit of my free time between the end of 2023 and the beginning of 2024 migrating the technology that powers this blog from Gatsby to Astro. If you want to find out more about the motivation and the migration process, well, that’s one of the three articles I wrote: Migrating from Gatsby to Astro. It was a fun project that should lead to better performance and easier maintenance in the future. And in the process I also redesigned the look and feel entirely. I hope you like it! …and be careful of the dreadful 404 page!

The new hero banner on the newly-designed home page. Note how it displays what I am listening to in real-time 🎸

The other two articles are about how I Have Published 404 Issues of a Newsletter for Full-Stack Developers Over the Last 8 Years and announcing that I am co-authoring a book about Rust and Lambda. So, you can see that while the output was low, the topics were significant for me.

Will I write more in 2025? I don’t know… This is the first article of the year, so maybe it’s a good start! I’m not planning to have a particular schedule, so we’ll see what inspiration strikes. Stay tuned!

The best time to plant a tree was 20 years ago. The second best time is now

— Chinese Proverb

FullStack Bulletin: Eight Years and more than 404 published Issues

In 2024, FullStack Bulletin reached a significant milestone: issue 404! It’s hard to believe it’s been eight years since I embarked on this project, aiming to make sense of the fast-paced, ever-evolving world of full-stack web development.

If you’re curious about the story behind the newsletter – why I created it, the technical architecture, and my vision for the future – you can read all about it in my article How I Have Published 404 Issues of a Newsletter for Full-Stack Developers Over the Last 8 Years.

But, to give you a short summary, FullStack Bulletin is a weekly newsletter designed to keep developers informed and inspired. Every week, I curate a selection of articles, a recommended book, and a thought-provoking quote, all tailored for full-stack developers and those aspiring to join the field. It’s the newsletter I wish I had when I started my own full-stack development journey. Andrea Mangano and I originally designed the format together and Andrea is responsible for the awesome look and feel that still represents FullStack Bulletin today.

I have had this quote in the article to celebrate the history. “FullStack Bulletin is our way of making life a bit easier for full-stack developers and those aspiring to join the field. Every week, we carefully curate the most interesting and valuable content, delivering it straight to your inbox. Since our launch in March 2017, our mission has been to help developers stay informed and inspired.”

Since last year, we’re still at around 3000 subscribers, so the growth has been very flat. This isn’t an encouraging sign, but at the same time, I keep getting good feedback from readers, which is what keeps me going.

In 2024, I also decided to give every newsletter a more personal touch, adding an editorial intro where I can share some of my personal news and encourage people to keep being inspired by the awesome content you can freely find on the web. I also spend more time making sure every piece of content has a good description that highlights why I handpicked that particular link.

People seem to have appreciated this more personal touch, so that’s probably a good sign. This year, I am considering moving FSB away from Mailchimp and adopting another provider. If you have any suggestions, let me know! I’m always open to finding a more cost-effective and feature-rich platform. And if you’re not already a subscriber, you can sign up at FullStackBulletin.com and join our wonderful community.

A particularly big shout-out goes to one of our major sponsors last year: PostHog. Thanks for supporting us.

By the way, we are looking for new sponsors, so if you have a company that targets full-stack developers, let me know. I have a nice slide deck I can share with you that illustrates the sponsorship process and the costs in a super-simple way.

Human beings are curators. Each polishes his or her own favoured memories, arranging them in order to create a narrative that pleases.

— Kate Morton

Other Stuff: Personal Achievements and Looking Ahead

Now, onto some other more personal achievements of 2024 🦸🏻. It’s important to remember that life isn’t just about code, conferences, books, podcasts and newsletters! 😆

I have been quite bad at running this year, totaling only 9 runs for a total of ~60Km. I definitely need to improve that in 2025! However, I have been pretty consistent with Brazilian Jiu-Jitsu training, practicing for almost 100 sessions during 2024 and reaching the grade of purple belt with 2 stripes! 🥋 Thanks to all my teammates and professors for keeping me engaged and motivated (although I doubt any of them will ever read this page 😅).

… And in general I kept myself fit, training more at home, getting in better shape and eating healthier! That’s something I’m proud of and I can see making me a lot happier and productive. 💪

Finally, I want to mention I had a lovely road trip to the Norway fjords with my wife during the summer. It was absolutely stunning, and a great way to recharge. I’ll leave you with a random picture from this trip.

That’s all, folks! Please comment to let me know what was your biggest achievement of 2024 and what do you plan to do in 2025. I’m always curious to hear what others are working on.

See you in the next article, hopefully soon! 😉

Luciano Mammino

Finally/First

Symfony security: authentication made simple (well, maybe!)

Interesting links

Symfony Authentication graph

3 invitations to try Atom.io

Introducing ORM Cheatsheet

5 lessons learned at the Bank of Ireland Accelerator

Being an entrepreneur is hell

Understand who you are, what you want to accomplish but don’t forget about your customers

Passion breaks the point!

Surround yourself with great people

You should work really hard if you want to look as good as a swan!

Conclusion

New PHP library: PHPoAuthUserData

Integrating Twig.js and BazingaJsTranslationBundle

Transparent pixel response with Symfony, how to track email opening

Introduction

Track email opening

Symfony implementation

Considerations

Introducing flickr-set-get a command line app to download photos

Why?

How it’s built?

Going asynchronous and execute requests in parallel

How to use it?

Current status

Opinions

Reset your MySql server password

1. Stop the mysql daemon

2. Disable the security checks

3. Login as root

4. Change the root password

5. Restart MySql

Having fun with Node.js, Slack and Chuck Norris

8 invitations to try Keybase.io

A quick demo

Do you want to give it a try?

Symfony, edit the Response globally using the Kernel Response event

The Kernel Response event

Example 1. Add custom Http headers to notify remaining api calls

Example 2. Create a cookie to track referrals

Conclusion

Writing a new Extractor for PHPoAuthUserData

What is an extractor

Writing the Instagram extractor

6 Rules of thumb to build blazing fast web server applications

Rule 1: Avoid premature optimization

Rule 2: Do just what you need to

Rule 3: Mama, I’ll do it tomorrow!

Rule 4: Gotta cache ‘em all!

Rule 5: Avoid the damn N+1 Query Problem

Rule 6: Scale… horizontally!

Conclusions

Write a console application using Symfony and Pimple

Let’s start

Folder structure

The Greeter service

The GreetCommand

Ladies and gents, the Pimple container!

The bootstrap file

The executable console file

Conclusions

Simple echo server written in Go, dockerized!

Introduction

The Go echo server

Dockerizing the application

Images, Containers and Dockerfile(s)

Let’s write the Dockerfile

Let’s build!

Running the container

Conclusions

Backup your server with Dropbox

Prepare the dropbox user

Install the dropbox client

Dropbox as a service

Enjoy

Bonus

Developing a web application with Lumen and MySql

A motivational quote everyday

Extracting data with `grep`

Cleaning up the data with `cut`

Substring with `cut`

Splitting strings with `cut`

Counting and sorting with `uniq` and `sort`

Improved `grep` with the reset match operator