It’s been a long time since this happened, the details are starting to blur.

So I’m writing it down — partly to remember it myself, partly so you might probably learn a thing or two.

Disclaimer: I recreated the problem on my machine so that I can show you screenshots and code examples, because I don’t have have any evidence from the original fault – although, they should be fairly identical.

Prologue

Like most faults in software systems, this one started with something simple going wrong — but it still managed to hand me a few hard-earned lessons.

If you’ve worked with Kafka in Node.js, you’ve probably heard of a library called kafka-node. Back in its prime, it was one of the go-to Kafka client for Node.js. Even at the time of this writing, it still sees over 100k downloads a week, despite being a bit crusty around the edges.

My API running on Kubernetes that also produces messages to Kafka— was built with kafka-node. kafka-node had been losing popularity for a while with better-maintained alternatives becoming available. But, why touch what isn’t (obviously) broken, right? 🤷

The Curious Case of the Grumpy Pods

It was a quiet Friday afternoon — naturally, the perfect time for something weird to happen.

Two pods from my service were failing to connect to the Kafka cluster.

The rest? Totally fine and happily producing messages. Not ideal, but not a fire either.

This wasn’t a big deal, because the Kafka producer connectivity status had been linked to the service’s readiness probe. Therefore, the problematic pods weren’t receiving any traffic due to the readiness probe failure.

Therefore, the API was up and handling traffic.

The Meltdown

Sunday.

The service is completely down — no pods are ready. Zero. Zilch. The thing that was left ignored on Friday? Yeah, that just exploded.

Turns out, it had been a ticking time bomb quietly counting down all weekend.

Here’s what’s happened.

On Friday, something changed that broke new Kafka client connections. The two pods that failed were fresh ones — spun up after the unknown change. The rest, the ones that were already connected to Kafka, kept working fine.

But here’s the catch.

Kubernetes doesn’t care if a pod is healthy and ready right now. It’ll kill and replace pods for all kinds of reasons — autoscaling, resource pressure, you name it. And that’s exactly what happened. Over time, the working pods got cycled out. Their replacements couldn’t connect to Kafka… and boom, no more ready pods.

Debugging the Mystery: VSCode with a hint of Wireshark

Application logs weren’t helpful. All I saw in the logs were loads of messages with text: client disconnected.

There weren’t any debug logs of the service, due to performance and cost concerns. But, of course, I needed more than a vague client disconnected message to figure out what was going wrong.

So, I wanted to reproduce this issue locally and debug on my IDE. In my local machine, I spun up my application with debug logging, pointing to the Dev Kafka cluster which was also indicating the issue.

Then the logs provided some useful insights.

It appears that the ApiVersions request sent by my Kafka client is failing because the connection is closed.

Side note — Lots of different types of messages are sent between a Kafka client and a broker. To ensure the client understand what the broker is sending and vice versa, the client and broker has to agree on the ‘shape’ of each message type. ApiVersions request is the mechanism used for this version negotiation.

I checked the supported Kafka API versions by my client vs on the broker. They seemed to be compatible.

Throughout this investigation, my Kafka provider kept on insisting they haven’t made any changes on their side at all.

I ran my application on debug mode again and played around to see why the ApiVersions request is actually failing.

At the same time, I opened Wireshark to sniff the traffic between my Kafka client and the broker.

What I saw was unexpected.

Light at the end of the tunnel

When a Kafka producer client connects to a broker, the following sequence of actions happen in the indicated order. The interactions highlighted in green with solid border are the parts of the Kafka Protocol itself.

Wireshark told an interesting story from the traffic between my client and the bootstrap server.

TLS handshake was failing right after the client sent the Client Hello message. The bootstrap server acknowledges it but then responded with a TCP packet with the RST (reset) flag, essentially slamming the door shut before the Kafka protocol even got a chance to speak.

So it wasn’t the ApiVersions request that was failing—the connection was already dead before it got that far.

Now the real question became: what was wrong with that Client Hello?

Alternatives

Good progress debugging so far. But I still don’t know what exactly is wrong with the Client Hello message.

It appeared that only the producers written in kafka-node were affected by this issue. So I wanted to check what other clients do right that kafka-node doesn’t.

I wrote a simple Kafka producer script using KafkaJS.

import { Kafka } from 'kafkajs';
import config from './config.json' assert { type: 'json' };

const kafka = new Kafka({
    clientId: config.clientId,
    brokers: [config.bootstrapServers],
    ssl: true,
    sasl: {
        mechanism: config.saslMechanisms,
        username: config.saslUsername,
        password: config.saslPassword
    }
});

const producer = kafka.producer();

(async () => {
    await producer.connect();
    await producer.send({
        topic: 'mytopic',
        messages: [{ key: 'key1', value: 'hello world' }]
    });
    console.log('message succesfully produced');
    await producer.disconnect();
})();

I ran the script, with my friend Wireshark watching the packets.

Unsurprisingly, it worked fine.

And in the Wireshark capture, I saw an interesting extra piece of information in the Client Hello message, which I didn’t see with kafka-node.

KafkaJS appears to be using TLS SNI (Server Name Indication) extension, where kafka-node doesn’t by default.

Both kafka-node and KafkaJS use NodeJS tls package for secure TCP connections and tls.connect function itself doesn’t enable SNI by default. However, KafkaJS adds the SNI extension for all secure connections as implemented here.

In short, Server Name Indication (SNI) is an extension to the TLS protocol that lets a client tell the server which hostname it’s trying to connect at the start of the TLS handshake. This is super handy for servers hosting multiple domains on the same IP address — like shared hosting — because it lets the server present the right TLS certificate for the requested domain.

If you’d like to learn more, there’s a very nice explaination about SNI on Cloudflare blog here: https://www.cloudflare.com/en-gb/learning/ssl/what-is-sni/

kafka-node client been working fine before, which means it was able to establish a connection without SNI. But all of a sudden it can’t?

My Kafka provider seems to be pretty convinced that they didn’t touch anything on their side though.

Time to go back to them with the new evidence about the SNI change.

“Hey, did you make any changes that would make SNI mandatory?”?

The Confession (Sort Of)

“Ah, apparently our network team had made a change to make SNI mandatory. But you shouldn’t be affected as long as you are using a ‘supported’ client”.

Said them.

Finally, with irrefutable evidence, they relented.

They had chosen to enforce SNI on all connections based on the assumption that all customers use ‘supported clients’ by their own definition, which is funny, because they hadn’t listed a ‘supported’ Kafka client for NodeJS on their website.

No heads-up, no changelog, no notice. Just a silent infrastructure change that quietly broke any client not sending SNI.

Fix Now, Refactor Later

So, what options did I have to fix the issue?

kafka-node does has a way of enabling SNI. It’s just not done by default.

import { KafkaClient } from 'kafka-node';

const client = new KafkaClient({
    kafkaHost: "mykafkahost.example.com:9092",
    sslOptions: {
        servername: "mykafkahost.example.com"
    }
});

Providing sslOptions.servername will enable SNI extension when the client connecting to the bootstrap server.

But there’s a catch.

Some Kafka providers have bootstrap servers that only serves the purpose of allowing the Kafka clients to discover the hostnames of actual data-bearing brokers via a Metadata Request. Hardcoding SNI to the bootstrap server’s hostname might not work in that kind of a scenario.

So what did I do?

I hot-swapped kafka-node with KafkaJS, which not only solved the issue but gave us a more modern, better-maintained client at the time in the process.

The Verdict: Who Really Pulled the Rug?

Is SNI a mandatory requirement for Kafka itself?

No. It’s because SNI is not a part of Kaka protocol itself. It is a lower-level network feature so Kafka protocol doesn’t care about it.

So, why am I saying that “they pulled the rug under me”?

My opinion is — A drastic change like this should be communicated upfront. This is because,

• They don’t have their own well-supported SDK that they advise the customers to use. Nor do they have listed a ‘supported’ Kafka library for NodeJS.
• Yes, SNI has legitimate use cases, especially in multi-tenant environments. But I haven’t seen any requirement that Kafka libraries has to support SNI by default.

Is my Kafka provider 100% at fault?

No. I don’t think so either.

kafka-node had become abandonware, but there wasn’t a strong reason to move on from it because it just worked and it didn’t have any alarming bugs/vulnerabilities.

Software vendors do not have unlimited resources to support every possible library on the planet. When popularity moves from one library to another, they do too. They abandon abandonware too.

P.S.

Talking about abandonware, the word around the town is that KafkaJS is no longer maintained too.

So, who knows when someone pulls the rug under KafkaJS soon?

We need to be constantly on the move.

How my Kafka provider pulled the rug under me! was originally published in Deepal’s Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

I had the pleasure of working with Lua with OpenResty a while back. Albeit numerous testing libraries/frameworks being available for Lua unit testing I faced a few challenges to learn how to set up a dockerised unit testing set up for my Lua scripts. I wrote this guide based on what I learned, in case it might be useful for someone else who is new to Lua, or even me down the road.

Framework of Choice

Out of the available Lua testing frameworks including LuaUnit, I found busted testing framework easy to use and feature-rich. You can install busted using luarocks — Lua package manager, but in this article, I will be using a dockerised setup using imega/busted docker image.

busted internally uses luassert for assertions, which is from the same developer as busted itself. If you want to learn more about different assertions available, I recommend referring to luaassert documentation directly.

File Structure for Tests

By default, busted identifies the files with _spec.lua suffix as test files, but it can be configured to your chosen pattern. For this example, I have 3 simple modules, with the corresponding test files sitting at the same level. My file structure would look like this:

You can find the complete example here on GitHub: https://github.com/deepal/lua-testing-sample

Docker Setup

My docker-compose.yml file has one service called test which will run the unit tests inside a imega/busted docker container. imega/busted docker image has busted command pre-installed, which we can use to run the tests.

The content of the docker-compose.yml file looks like this:

It is important to set the LUA_PATH environment variable such that the Lua modules are resolved properly. (You can learn more about LUA_PATH in the Lua docs.)

Writing a simple unit test

My sum.lua module contains a simple function which returns the sum of the given two numbers.

We can test this function easily by writing the following BDD-style test. describe, it, assert are global functions provided by busted framework, therefore there’s no need to explicitly importing those.

Following is the corresponding unit test file (sum_spec.lua) for my sum.lua module.

Easy-peasy.

Stubbing/Spying functions

Stubbing and Spying on Lua functions is also fairly straightforward with busted. It provides a stub method to create stubs or stub existing functions, and spy.on, spy.new functions to create spies.

For the sake of not repeating the documentation, I wouldn’t go into the details of spying. You can have a look at how to spy on functions from the official documentation at your leisure. But let’s look at how to create stubs with busted as it will be useful in the next section as well.

The following is my check_env.lua module. It’s a simple function which returns true if RUN_SUM environment variable is set, and false otherwise.

To test both branches of this function, we need to stub os.getenv and manipulate its behaviour. In the following test file, we stub os.getenv before running each test, which we then modify the behaviour of in the tests. It’s safer if we revert the stub to its original state after running each test by using after_each function.

Similar to describe and it, before_each and after_each functions are also global functions provided by busted.

In the above test file, we have two tests, where the first one would assert check_env function’s behaviour when os.getenv("RUN_SUM") returns “1”, and the second test when os.getenv("RUN_SUM") returns nil.

Stubbing imported modules

Stubbing dependencies is important when test files which have external module imports using require. Let’s look out our app.lua module which depends on our previously discussed lib.check_env and lib.sum modules.

app.run function internally calls both check_env() and sum() functions we previously discussed. Therefore, we will have to test different flows of app.runwhen the behaviours of check_env() and sum() changes.

Because lib.check_env and lib.sum are externally imported modules, we cannot simply stub them the same way we did in the previous section. This is where Lua’s package.preload feature comes to the rescue!

package.preload is a table which Lua’s require function first lookup in when it’s called (official docs). Therefore, we can use package.preload to pre-load a stubbed module, which require will then use instead of the actual module. Let’s look at our test file app_spec.lua to see how we can use package.preload to stub modules. For simplicity of the example, let’s stub check_env the module only in this example.

setup and the teardown global functions provided by busted can be used to define functions that runs at the start and the end of the current test suite respectively.

At the setup stage, we preload lib.check_env module with a stub. Once this is done, any require "lib.check_env" operations will load this stub module instead of the actual lib.check_env module. However, we need to remove this module from the pre-loaded table after the current test suite to avoid leaking it to the other test suites. This can be done by setting package.preload["lib.check_env"] = nil at the teardown stage.

Running Tests

We now have everything set up. A simple docker-compose command will start the test suite.

If you have any questions, don’t hesitate to comment. I hope this was useful in some way, and thanks for reading.

The Journeyman’s Guide to Lua Unit Testing was originally published in Deepal’s Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Advance NodeJS— Part 6

If you fly between two airports that are in two different time zones, would you calculate the time it took for you to fly, by checking the difference between the ‘wall clock’ times of either airport?

If your answer is ‘no’, then let me explain why you shouldn’t do the same in your code, using a little but a very interesting experiment.

This article is the 6th article of my Advanced NodeJS Series which is a sequel to my series on Node.js Event Loop. You can find the other articles of the Advanced NodeJS series below:

Post Series Roadmap

• Crossing the JS/C++ Boundary
• Deep Dive into Worker Threads in Node.js
• Understanding Async Resources with Async Hooks
• A Visual Guide to NodeJS Streams
• NodeJS Streams in Practice
• Why you should avoid Date.now() to measure time durations (This article)

The examples in this article are in JavaScript/Node.js, but the concepts described are valid regardless of the language being used.

Clocks, Time and Measuring Time

I’ve seen countless times where developers (including myself at some point) measure the time spans for metrics and logging using date-time functions. These functions use so called ‘time-of-day clock’, and in JavaScript the usual suspect that falls under this category is Date.now() (or Date constructor). Using Date.now(), this is the code one of those developers would write to measure time duration.

This is a nice little code that works, until it doesn’t.

Before delving into our experiment and explaining why the above wouldn’t work, it’s important to know a few key details about time, how it is measured and maintained by the operating system.

• Not all the clocks across all computers run at the same speed, some may run faster or slower comparatively and eventually may drift apart from each other (like some of our wall clocks sometimes do).
• This ‘clock drift’ can cause various problems in distributed systems.
• NTP (Network Time Protocol), is one of the network protocols that is used to fix this problem, by synchronising the clock of a computer with that of a ‘time server’ that usually has accurate time.
• NTP clients that are usually shipped with the operating system may periodically receive the time from a configured time server, and adjust the local system clock.
• If the NTP client finds that the local time is drifted apart from the time server’s time, it may choose to ‘correct’ it depending on how much the drift is.
• This ‘correction’ of time may result in the system clock suddenly moving forward or backwards.
• The functions that use ‘time-of-day’ clock, such as JavaScript Date.now() returns the current system time, which could be affected by NTP corrections.

Now you may probably understand why I said the snippet 1 wouldn’t work. To see it and understand it clearly, let’s move on to our little experiment.

Experiment 1 — How Not To Measure Time

In this experiment, we are going to measure the time it took to run doSomething() function. But, we are going to simulate an NTP time correction by deliberately setting a fake time in the beginning and correcting the time using NTP while doSomething() function is being executed.

You can find the complete implementation of the example at the end of the article. This example only works on macOS as it relies on a few macOS shell commands/utilities, but you should be able to easily modify it to work on Linux.

• setFakeTime() sets the system clock to 1 min ahead of actual time. This function uses macOS date command.
• correctTimeNTP() uses the macOS sntp NTP client utility to correct the time by synchronising with the time server time.apple.com.
• doSomething() function simulates a task that takes ~2000ms. We are using setTimeout to simulate this task.

If you run the above code, it would measure the time doSomething() took using Date.now() and log it to the stdout.……AND BOY IT’S NEGATIVE!

This is not what a developer would expect at all!

The numeric value you see here may depend on various factors such as how busy your CPU is, how long did it take for the NTP client to receive time from the time server etc. However, it is fairly obvious that the reason for the ‘negative time duration’ is because the NTP client corrected the clock by moving it backwards (because it was ahead by 1 min in the beginning). Therefore, endTimeToD is an older timestamp than the startTimeToD.

Though this example is a simulation, this can happen in real life systems due to various reasons (e.g. time server sending an incorrect time for some reason, NTP client refreshing time after not running for a significant time, a human operator manually changing system time, & etc.)

But the important question is, can we solve this problem? YES. And the answer is by using a function that uses a monotonic clock rather than a time-of-day clock. Let’s read on.

Experiment 2 — How To Measure Time Accurately

Monotonic clock is simply counter that has started from an arbitrary point in the past and moves at the same speed as the system clock. Since this is just a counter, a monotonic timestamp read at a given point in time doesn’t have any use by itself. But the important part is, monotonic clock is not affected by NTP corrections. Therefore, the difference between two monotonic timestamps read at two different points in time will give the accurate time duration between the two points.

Programming languages usually provide functions for both monotonic and time-of-day clock. In Node.js, the functions that returns monotonic time are:

• process.hrtime.bigint() (process.hrtime() in older NodeJS versions)
• require('perf_hooks).performance.now()

Let’s update our experiment to use require('perf_hooks).performance.now() instead of Date.now(). You can also try out the experiment with process.hrtime.bigint() at your leisure.

I’ll leave out const { performance } = require('perf_hooks) part of the following snippets for brevity.

If you run this example, it would log the following output (or similar).

As you can see, we get an accurate result by using monotonic time.

The reason for not getting an exact 2000ms is because, setTimeout is subject to certain Event Loop semantics that I’ve explained in previous article.

Let’s now try to run both experiments in one script, to compare and measure the error of the 1st experiment.

Experiment 3 — Comparison

In this experiment, we run both 1st and 2nd experiments in a single script. And in addition to that, we log:

• Output of the sntp command that’s executed inside setFakeTime() function.
• The error of the duration calculated using Date.now()

If we run the above script, we get the following output:

This output is very interesting! We can clearly see the difference between the time duration measured by both approaches. And if you look closely, the ‘error’ of the time duration measured using Date.now() (-54.053s) is pretty much the same number NTP client adjusted the time by (-54.053600s), if you ignore the precision difference.

This clearly indicates that the time durations measured using a time-of-day clock is directly affected by the NTP corrections.

Isn’t Node.js setTimeout() and setInterval() affected by NTP time corrections?

Node.js event loop uses uv__hrtime to measure loop time, which underneath uses a monotonic clock. Therefore, setTimeout and setInterval functions are not affected by NTP time corrections.

Wrap Up and Recommendations

When measuring a time duration is required for monitoring and logging purposes, it’s always expected to be accurate. Using the incorrect function to measure time spans can yield very inaccurate results as we saw throughout our experiments.

Therefore, always use functions that return monotonic time to measure time durations for metrics and logging.

The following is the complete code example discussed throughout this article. Note that, you’ll need to run this code example as sudo because the date and sntp commands require elevated privileges to run.

I’d also like to mention Dr. Martin Kleppmann’s lecture on Clock Synchronisation as an additional resource, which was an eye-opener for me.

I hope this article helped you get a practical understanding on how to accurately measure time in applications. And I thank you for reading.

Why you should avoid Date.now() to measure time durations was originally published in Deepal’s Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the drawbacks of the microservices architecture is the difficulty to implement atomic transactions. While this limitation can be overcome by using patterns such as Saga, there are certain situations where we can use an alternative approach to solve this problem without having to implement the rather complicated Saga pattern.

Let’s consider a scenario where we need to dispatch an event when a user verifies their email after signing up. The event will then be consumed by other services for reporting, updating the customer profile on a CRM system and sending a customer a welcome email.

Take a look at the following diagram that describes this kind of setup.

Updating the User document, and dispatching the event are two operations to two completely different systems, and therefore cannot be performed as a single atomic transaction.

To make this even clearer, take a look at the following sequence:

In this scenario, the Application sends an update to the Database which is successful, but it is unable to dispatch the event to the Message Broker. In this situation, the Application has the following choices.

Retry dispatching the event — 
The application can exponentially retry, assuming that it’ll eventually be able to dispatch the message. But it cannot be guaranteed that the message will go through.

Revert the database change and send an error response — 
If the message doesn’t go through, the database change can be reverted and an error can be sent back to the user interface. (Customer can then be given a chance to resend themselves the verification email). But, if the message was indeed received by the message broker, and only the acknowledgement from the broker didn’t reach the application, we’ll end up in a tricky situation where we have dispatched an event but the operation is not reflected on the database.

The transactional outbox is a pattern that can be used to solve this problem. In this pattern, the Application would perform the update to the database and at the same time insert a document (outgoing message) that describes the operation to a separate ‘outbox’ collection. Since updating the database and inserting an outgoing message both happen on the same database, the two operations can be performed as a single transaction. The messages inserted into the outbox collection can then be used to generate an event, using a separate event dispatcher service. This way, we can guarantee that either updating the DB and creating a message both occurs, or neither occurs.

In the rest of the article, I’m going to explain how to build this pattern using MongoDB as the data store.

With Polled Outbox

In this approach, the outbox collection is periodically polled by the Event Dispatcher to see if there are any unsent messages in the outbox. If there are any, those will be fetched by the Event Dispatcher to generate events for the message broker. Once the events are sent to the message broker, those messages can be either deleted from the outbox or can be marked as sent.
This architecture has a few advantages as well as some disadvantages.

Advantages:

• This approach is database agnostic and can be applied with MongoDB or any other relational/non-relational database.
• It’s easy to recover after any downtime in the Event Dispatcher. The Event Dispatcher will only have to start polling and process any outstanding messages in the outbox.

Disadvantages:

• Event generation from DB operations won’t be in real-time, because the polling only happens periodically.
• Need additional functionality in Event Dispatcher to mark outbox messages as read, or delete them after producing the message to the message broker.
• The outbox collection has to be polled even if there’s little or no activity in the DB, which wastes resources.

The alternative to the polled outbox approach is real-time outbox processing, using a feature provided by MongoDB called ‘Change streams’.

Atomic Transactions with MongoDB Change Streams

Change streams is a MongoDB feature to allow applications to listen to the changes in a given collection, a database or a deployment. In this approach, the Event Dispatcher is connected to a MongoDB change stream, listens to ‘insert’ events from the outbox collection and dispatch them to the message broker.

Change streams are supported in a variety of languages. You can read more about it here. The following is an extremely simplified example of how the Event Dispatcher code would look like in JavaScript (Node.js).

A sample received changeEvent would look like as this:

In comparison to the Polled outbox approach, using change streams has its advantages and disadvantages.

Advantages:

• Outbox messages can be sent to the message broker in real-time
• No need for the Event Dispatcher having to explicitly delete outbox messages or mark them as read. A TTL index can be set up to clean up the outbox messages after some time.

Disadvantages:

• Change streams is a MongoDB feature. If you decide to switch to a different database vendor, you may need to make significant changes to the Event Dispatcher to build a similar functionality with the database you are switching to.
• The event Dispatcher will need to keep track of the last processed event to resume after a downtime. See resuming a change stream on official docs.
• Change streams are generated using the MongoDB oplog. Oplog is a capped collection with a limited size that stores rolling information about the operations that are happening on the database. In a case where the event dispatcher is down for an extended period, it may not be able to resume from the last processed position if it’s already gone from the oplog. This can particularly happen in a situation where the database is very busy and the event dispatcher cannot process at the same rate the changes are happening on the database.

Also, make sure you are aware of the best practices and the recommendations provided by MongoDB, on using change streams on production applications.

Wrap Up

While the transactional outbox pattern is an excellent way of solving the described problem, it comes with its own complexity. In addition to what I’ve mentioned before, you must have a good understanding of solving the following problems as well when you apply this pattern, regardless of you are using the polled outbox approach or change streams.

Scaling Event Dispatcher — You may want to deploy multiple instances of the event dispatcher, or set up auto-scaling for the event dispatcher in a production setup. In such a situation, you will need to implement a proper ‘partitioning’ mechanism to distribute the outbox messages across all available event dispatcher instances to avoid the same message being picked up by multiple dispatcher instances.

Duplicate Messages — The event dispatcher may generate duplicate messages to the message broker after recovering from a failure, or after an autoscale event. Therefore, the consumers of the events should be idempotent. However, distributed systems often don’t have ‘exactly-once delivery’. Therefore, it’s expected that the consumers to be idempotent in most cases.

If you are already using this pattern on production, or have any experience in these approaches, feel free to let me know in the comments.

Thank you for reading!

References

• https://microservices.io/patterns/data/transactional-outbox.html
• https://docs.mongodb.com/manual/administration/change-streams-production-recommendations/
• https://docs.mongodb.com/manual/core/replica-set-oplog/

Atomic Microservices Transactions with MongoDB Transactional Outbox was originally published in Deepal’s Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Experimenting with HTTP/3 — How the Next Gen Web is being built on UDP

HTTP is the lifeline of the web. Amongst many other application layer protocols that power the modern internet, HTTP is the most prevalent and most known protocol which powers the World Wide Web. Being a protocol that runs on top of TCP (Transport Layer Protocol), HTTP has been benefited from the reliability, congestion control and flow control provided by TCP for decades.

Throughout this time, there were many changes and improvements done to improve the reliability, performance and security of HTTP-based communication. However, the limitations of TCP always prevented further improving the HTTP protocol. Therefore, it looked like it’s time to do something controversial. That is, ditch TCP and switch to its ‘unreliable’ counterpart UDP for the next HTTP version — HTTP3. I’m not planning to explain those reasons in detail in this article, but if you are interested, feel free to check out the References at the end of the article.

At the time of this writing, some major internet companies and CDN providers are already supporting HTTP3 for their services. In this article, I’m going to ‘briefly’ explain HTTP3 and how it’s built, and describe how you can experiment with the new protocol.

Introduction to HTTP3 — a.k.a. HTTP over QUIC

The major problem of switching from TCP to UDP is that the two protocols are very different in terms of reliability and functionality. TCP is a connection-oriented protocol that has congestion control, flow control and retransmission built into the protocol. UDP is a very simple connectionless protocol (fire-and-forget type), and it doesn’t have any of these goodies built into the protocol. Also, the encryption provided by TLS couldn’t be used with UDP because of the unreliable nature of UDP. Therefore, UDP couldn’t directly replace TCP for HTTP connections.

To bridge this gap, a new transport layer protocol called QUIC (originally stood for Quick UDP Internet Connections) was internally developed in 2012 at Google to be used in their services.

QUIC was responsible for providing missing features in UDP as well as encryption provided by TLS, depicted as follows.

With TCP, a secure HTTP connection required both TCP and TLS handshakes to be completed before the actual HTTP communication begins. With QUIC, these two handshakes are replaced by just one QUIC handshake to establish a secure connection to the server, which significantly reduces the time it takes to establish a secure HTTP connection. (More on the handshakes in a future article… 😊)

After further developments and improvements, QUIC protocol was finally standardised by IETF in May 2021 as described by RFC 9000. However, as of Oct 2021, HTTP3 protocol is not yet standardised. However, HTTP3 support is being already implemented by many large scale internet companies and CDN providers.

Here are a few milestones from the journey of the Web towards HTTP3.

Why QUIC? Why not improve TCP?

TCP is an old protocol that is built into the OS kernels of all the devices that make up the internet. Any change to TCP will have require operating system updates to all these devices and may take years or decades to be rolled out across the internet (this is precisely why IPv6 adoption is slow). Implementing a new complex protocol that runs on top of a simple existing transport layer protocol such as UDP seemed to make sense due to this reason. The support for the new protocol could easily be rolled out to HTTP servers/proxies and HTTP clients via software updates, and won’t require OS updates.

Server support for HTTP3

Nginx, which is one of the most widely used servers has experimental support for HTTP3 at the time of the writing. And also Caddy, which is another popular open source web server seems to have experimental HTTP3 support.

However, Apache web server doesn’t seem to be interested in implementing HTTP3 support anytime soon.

Client support for HTTP3

HTTP3 brings significant fundamental changes to how HTTP works. Due to this reason, major changes are required in many tools (such as SSL libraries) that are used to develop HTTP clients. Though the progress towards HTTP3 is a bit slow, there are many HTTP clients that already support HTTP3 or can be custom-built from the source to support HTTP3.

Browsers

At the time of the writing, a majority of the major browsers support HTTP3. You can check the current status of HTTP3 support at https://caniuse.com/http3.

Obviously, establishing HTTP3 communication will require both the client and the server to support HTTP. The HTTP server will need to first let the browser know that it supports HTTP3, as the browsers will not try to establish an HTTP3 connection by default. This is done by the HTTP servers using the alt-svc (alternative service) response header.

In the following screenshot, www.google.com indicates that it supports a few draft versions of HTTP3 on port www.google.com:443 using alt-svc header.

When this header is received by an HTTP3-capable browser, it will switch to HTTP3 communication with the server, which can be verified using the browser’s network panel.

Also, the browser will cache the alt-svc parameters for the number of seconds set by ma parameter (max age) in the header value. For example, alt-svc: h3=":443"; ma=2592000 will instruct the browser to cache this alternative service for 2592000 seconds, which is 30 days. This helps the browser to temporarily remember that the server supports HTTP3 and directly use HTTP3 for future connections.

cURL

Though the browsers are catching up with HTTP3 support, our beloved HTTP client curl doesn’t yet support HTTP3 by default. This is because neither OpenSSL nor LibreSSL which curl is usually shipped with supports QUIC at the time of this writing. However, BoringSSL — Google’s fork of OpenSSL supports QUIC and you can easily build curl from the source to use BoringSSL.

If you are using macOS, you can build curl with HTTP3 support using the homebrew formula provided by Cloudflare as follows.

Alternatively, I found this useful docker image ymuski/curl-http3 of curl built with HTTP3 support. You can also use this image to try out the curl commands I’ve mentioned in the rest of the article.

Once you build and add the new curl binary to PATH, you can verify HTTP3 support by simply running curl -V.

You can now send HTTP3 requests to any server that supports HTTP3. There are two ways you can instruct curl to use HTTP3.

Forcing curl to use HTTP3

If you know that the server supports HTTP3, you can provide --http3 flag to curl to force it to use HTTP3 as follows:

curl -I --http3 https://www.google.com

If you prefer using ymuski/curl-http3 docker image, you can use the following command instead.

docker run --rm ymuski/curl-http3 curl -I --http3 https://www.google.com

If the server supports HTTP3, it will respond to the request, and you can verify that the communication is HTTP3 by inspecting the response.

curl with alt-svc

Unlike browsers, curl doesn’t honour alt-svc header automatically because it doesn’t cache alt-svc information by default. However, you can instruct curl to cache alt-svc info, which curl can then use in subsequent requests. This can be done by using --alv-svc flag and providing a file name to cache alt-svc info as follows.

curl will by-default connect to the server with HTTP 1.1, and store any alt-svc information provided by the server in the cache file we provided (in our case, altsvccache.txt)

If you inspect altsvccache.txt, you will see that it’s populated with the information similar to the following.

If you send a subsequent request to www.google.com providing the cache file, curl will then read the cache file and send the request in HTTP3 directly.

REST Clients

There are feature requests to Postman or Insomnia — two of the commonly used REST clients requesting for HTTP3 support. However, no active discussion is going on towards implementing HTTP3 support in these clients at the moment. This is likely due to the lack of HTTP3 support in Node.js (which I explain in the following section).

Language/Runtime Support

Built-in language/runtime support for HTTP3 and QUIC is not mature yet, but there seem to be 3rd party libraries offering HTTP3/QUIC support in some languages such as quic-go for Golang and kwik for Java. (I would love to know the HTTP3 support in your favourite language. Feel free to let me know in a comment.)

HTTP3 support for Node.js has been underway for some time, but it’s still unclear which future Node version will be shipped with it. This is being relied upon by bunch of other tools such as Electron.js thereby impacting a lot of applications such as the Postman and Insomnia REST clients I talked about in the previous section.

Future of HTTP3

The transport layer changes introduced by HTTP3 is the major fact which slows down the roll out of HTTP3. Apart from its usage by DNS, UDP is not a protocol which as frequently used as TCP. Therefore, software used in many L4 devices on the internet are not optimised for UDP.

Though all these circumstances affect negatively on HTTP3 adoption, it’s interesting to see many internet companies and CDN providers already supporting it and the others are actively working towards it. However, it might take a long time for HTTP3 to become the dominant HTTP version, considering how slow even its predecessor HTTP2 is being adopted. But knowing how this novel protocol works, and experimenting with it early will give you the upper hand for quick adoption.

References

• https://blog.cloudflare.com/quic-version-1-is-live-on-cloudflare/
• https://blog.cloudflare.com/head-start-with-quic/
• https://www.nginx.com/blog/introducing-technology-preview-nginx-support-for-quic-http-3/
• https://datatracker.ietf.org/doc/html/rfc9000
• https://www.youtube.com/watch?v=dQ5AND4DPyU
• https://dl.acm.org/doi/10.1145/3098822.3098842
• https://www.akamai.com/blog/performance/http3-and-quic-past-present-and-future
• https://daniel.haxx.se/blog/2019/01/21/quic-and-missing-apis/
• https://daniel.haxx.se/blog/2021/04/02/where-is-http-3-right-now/
• https://everything.curl.dev/http/altsvc
• https://http3-explained.haxx.se/
• https://everything.curl.dev/http/altsvc

Experimenting with HTTP/3 — How the Next Gen Web is being built on UDP was originally published in Deepal’s Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

How to hide your communication in plain sight with Network Covert Channels

Attempt for Secret Communication is nothing new. Ever since we started to communicate with each other, we wanted to hide the details of our conversations from prying eyes and ears. Over the years, people have been working on this leading to the invention of “Encryption” as a result. But, encryption has a problem. Even if the encryption can hide the content of the conversation, it cannot hide the fact that a secret conversation is present. For that, we need something uncannier than encryption: one of the most popular ways to achieve that is called Covert Channels.

To understand this in a practical way, let’s have a look at a popular dilemma called The Prisoners’ Problem.

The Prisoners’ Problem

Two prisoners locked in two near-by cells are planning their escape. They are allowed to talk with each other, but there’s a jailer who can hear their conversation. Talking with each other is the only way of communication between the two of them, but they need to be careful. If the jailer senses that they are planning for something, he might move them to distant cells, which will jeopardise their escape plan. How can the prisoners plan their escape without making the jailer suspicious?

Encryption vs Covert Channels

Let’s say that the prisoners decided to transform their conversation using a secret code, which the jailer is unaware of. This is called “Encryption” in general. In this case, only the two prisoners can translate the transformed messages back to the original because only those two know the secret code used to transform the messages. But, even though the jailer cannot understand the conversation, the inability to understand the conversation could easily make him suspicious. Therefore, Encryption is not the solution for the prisoners’ problem.

Finding a solution to the prisoners’ problem introduces a new way of secret communication called Covert Channels. The goal of a covert channel is to provide communication between two parties in an unusual way so that an intermediate party can hardly notice that there’s such communication.

For example, a covert channel between the two prisoners could be that the number of words in a sentence serves as the actual information to be passed.

e.g: The prisoners can successfully pass the message “345” by using 3 irrelevant sentences as follows.

“Are you okay? I feel like sleeping. The food was not good.”

• Are you okay? => 3
• I feel like sleeping => 4
• The food was not good=> 5

This is a primitive example for a covert channel. But the most important fact is, both entities in the covert channel should understand how it works in order to successfully pass messages.

Covert Channels are not efficient in terms of throughput. But they are quite good at hiding their presence, hence providing security by obscurity. If a covert channel is detected, it’s very likely to be entirely compromised unless the content is further protected by encryption.

Covert Channels are established over the network by using networking protocols in unexpected ways. In this article, I’ll be talking about, Network-based covert channels and how they work, practical examples and identifying and blocking covert channels.

Covert Channels vs Steganography? If you have heard about Steganography, you might find it very similar to the concept of Covert Channels. However, Steganography does not misuse any network protocols to transport secret data. Instead, it conceals secret data inside another payload such as an image/video.

Abusing Network Protocols

A network protocol is basically a contract between a sender and a receiver, where the sender sends structured information in a format that can be interpreted by the receiver. This ‘structure’ of the information is defined as the ‘network packet’ structure. A packet mainly consists of 3 parts which are called the header, data, and an optional trailer. Firewalls and Intrusion Detection Systems (IDS) are particularly interested in the data section of a packet as it carries the actual payload of the transmission.

Many network protocols were implemented in early days where security was not a major concern, but ‘reliable communication’ was a concern. Therefore, passing information secretly by misusing header fields was certainly possible without triggering firewall or IDS rules, hence giving birth to covert channels.

Transmitting information in the header fields is not the only way for setting up covert channels. There are two main kinds of covert channels based on how data transmission is done.

Storage Channels — Storage Covert channels encode covert data in the packets themselves (encoded in headers as I mentioned before)

Timing (temporal) Channels — Delay between the packets is used to transmit data.

Timing channels are not as reliable as storage channels and are mostly research-oriented. Therefore, during the rest of the article, we’ll be discussing storage covert channels.

Storage Covert Channels

Single Protocol Covert Channels

A single protocol covert channel is the most common type of covert channels which are established using only one protocol for covert communication. This type of covert channels usually encapsulates covert data into a protocol header. A popular example of this type is a tool called ptunnel.

Ping Tunnel

ptunnel lets you tunnel TCP connections using ICMP Echo Request and ICMP Echo Reply packets reliably. We’ll see an example of a covert channel using ptunnel at the end of this article.

Protocol Channels

As its name implies, a Protocol Channel is a type of covert channel which is established over two or more protocols. An example would be a covert channel which uses ICMP and HTTP packets interchangeably to transmit data. This type of covert channels are more obscure compared to single protocol covert channels and makes it difficult.

Protocol Switching Covert Channels (PSCC)

PSCC also uses more than one protocol but unlike Protocol Channels, data is not encoded in the packets. Instead, in a PSCC, a packet itself should be interpreted as a message. Consider the following example.

In the above example, an HTTP packet is binary 1 and a POP3 message is binary 0. Therefore, to transmit the message 110100, the sender has to send the sequence of packets as HTTP, HTTP, POP3, HTTP, POP3, POP3. The bandwidth of a PSCC can be improved by using more than two protocols and assigning a larger binary value for each protocol.

e.g: Using 4 protocols and each protocol is assigned values subsequently 00, 01, 10, 11.

Protocol Switching Covert Channels are not efficient in terms of throughput compared to Protocol Channels. But for small messages, they are much more stealthy.

Use Cases of Protocol Channels

Enterprise Data Exfiltration — The most popular use case of covert channels is to exfiltrate confidential data out of a protected network.

Bypassing Firewalls to access forbidden content — Covert Channels can also be used to access forbidden content otherwise forbidden by a firewall. An example is tunneling TCP traffic over ICMP using a ptunnel proxy as described at the end of this article.

Botnet Communications, Remote Backdoors, and Reverse Shells — Some Botnets use protocol channels in order to stealthily communicate with their command and control (C&C) centers. Also, remote backdoors or reverse shells can also use covert channels to download additional malicious scripts without detection.

Implementation Concerns

Although Covert Channels are a great way for obscure communication, setting up a reliable covert channel is not easy. The following are the key problems which have to be catered when implementing a covert channel.

Throughput vs Stealth

• Encoding more data into packets could increase the risk of the channel being exposed because the packets contain a significantly high amount of data.
• The number of packets sent at a time has to be increased if less amount of data was encoded into a single packet, leading to the channel being discovered due to the unusual network activity.

Catering packet loss and out-of-order delivery

Covert channels do not use network protocols as is, but rather misuse them. Hence, they do not have the luxury of using network protocols’ reliability mechanisms such as TCP re-transmission, re-assembly, etc. Therefore, the implementation of the covert channel should define its own protocol which implements these reliability mechanisms so that it can withstand the packet loss and the out-of-order delivery of packets.

Setting up an ICMP-based Covert Channel with ptunnel

In this example, we’ll see how to access an unauthorized web resource through a firewall with the help of a covert channel. For this, we use ptunnel(i.e, Ping Tunnel) which provides the facility to tunnel a TCP connection over ICMP Echo Requests/Replies.

Let’s imagine the following hypothetical scenario.

• We are located in a secure corporate network and we need to access a particular webpage which is blocked by the company’s firewall.
• But we identify that we can successfully ping through the firewall, which indicates that ICMP Requests and ICMP Replies are not blocked by the firewall.
• Then, we set up a ptunnel proxy outside the firewall. ptunnel proxy will establish a TCP connection with the destination server and will tunnel our HTTP requests to the webserver and server response back to us via ICMP.

Our setup is as follows:

• Our local machine. Runs the ptunnel client (macOS — 192.168.4.1/24)
• ptunnel proxy (Kali linux — 192.168.4.111/24)
• Destination webserver (Metasploitable Ubuntu — 192.168.4.100/24)

Setting up ptunnel proxy

Starting up the ptunnel proxy is as trivial as running the ptunnel command.

Setting up ptunnel client

We then start up the ptunnel client on our local machine (located inside the protected network). Ptunnel client will start listing on a local port (-lp)for TCP connections which it will tunnel to the destination address (-da) and port (-dp) via the proxy (-p).

Accessing the remote resource

Once the server and the client are up and running, we can now access the restricted resource via the local TCP port as follows.

If we intercept the communication between the ptunnel client and the proxy, we could see how HTTP response secretly encoded inside the ICMP replies as indicated below.

Obviously, using a ptunnel covert channel to create a covert channel is not bandwidth efficient (150 kb/s download and about 50 kb/s upload as per the author). However, it can still come in handy for an attacker in order to secretly download a malicious script or exfiltrate secret data outside a secure network.

Identifying and Blocking Covert Channels

Covert Channels can be used with good intent, as well as malicious intent. Therefore, identifying and blocking them is also equally important. Though it is very hard to identify and 100% block covert channels, there are certain ways to make it difficult to set up covert channels.

IDS Packet Signatures — Configuring Intrusion Detection Systems with rules to identify the packet signatures of the common types of covert channels.

Traffic Anomaly Detection — Mostly research-oriented. For example, analyzing packet size variation, header size variation and bandwidth usage vs time of the day can lead to discovering unusual network activities.

Active Warden (Wendzel, Steffen & Keller, Jörg. (2012). Design and Implementation of an Active Warden Addressing Protocol Switching Covert Channels) — A firewall-like service which randomly delays packets if protocol switching is detected, making it hard to establish reliable covert communication. This is useful to prevent protocol switching covert channels.

Packet Data Padding — Intercepting and padding packet header fields which are likely to be used for covert communication.

Still, though, it’s difficult to 100% prevent covert channels without disrupting the legitimate traffic. There’s so much research work done in this area and new types of covert channels and protection mechanisms are invented from time to time.

References:

• Wendzel, Steffen & Keller, Jörg. (2012). Design and Implementation of an Active Warden Addressing Protocol Switching Covert Channels.

Abusing Network Protocols for Secret Communication was originally published in Deepal’s Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

NodeJS Streams in Practice — Advanced NodeJS Part 5

Turbo boost your Node.js application with Streams

In my previous article “A Visual Guide to NodeJS Streams”, I described what a stream is and how streams can be used in NodeJS using visuals. Visuals are great to get a broad understanding to start with, but it’s even better to do some hands-on work to understand how streams can be applied in practical applications. Therefore, in this article, I’m going to describe a few examples to demonstrate how streams can be used in day-to-day applications.

If you are not yet familiar with the concept of Streams in Nodejs, I highly recommend you to read my previous article first. 🙂

This article is the 5th article of my Advanced NodeJS Series which is a sequel to my series on Node.js Event Loop. You can find the other articles of the Advanced NodeJS series below:

Post Series Roadmap

• Crossing the JS/C++ Boundary
• Deep Dive into Worker Threads in Node.js
• Understanding Async Resources with Async Hooks
• A Visual Guide to NodeJS Streams
• NodeJS Streams in Practice (This Article)
• Why you should avoid Date.now() to measure time durations

Understanding a Readable Stream

Before moving on further, let’s try to understand how a readable stream works using the following example. In this script, I’m reading a file using a readable stream created by fs.createReadStream. The file I’m reading is foo.txt and it has the content “This is foo!!!”

If we run the above program, we will see the following output:

stream in flowing mode!
file opened!
stream paused!
waiting for 3 seconds...
stream in flowing mode!
got some data:  This is foo!!!
stream ended!
stream closed!

Let’s try to understand this output:

1. A readable stream is created as soon as we call fs.createReadStream, but it’s by default not in flowing mode. This means that the stream will not be generating data from the underlying file ‘foo.txt’.
2. As soon as we attach an event handler for the data event, the stream switches to flowing mode and emits the resume event. This triggers our event handler that is listening to the resume events.
3. The operating system opens ‘foo.txt’ which causes the readable file stream to emit open event, which triggers the event handler listening to the open events.
4. Inside our event handler for open event, we explicitly call reader.pause(). This will cause the stream to switch from flowing mode to paused mode. In paused mode, the stream will not generate any data from the underlying file. Therefore, no data event will be emitted until the stream is put back to flowing mode explicitly.
5. After waiting for 3 seconds, we call reader.resume() which switches the stream back to flowing mode. Then the stream will start generating data, which causes data events to trigger.
6. One or more data events will be emitted depending on the size of the file. In our case, it’s just a small amount of text which the readable stream is perfectly capable of buffering in memory. Therefore, we only see one data event.
7. Once the data is completely read from the stream, the stream emits end event to indicate there are no more data to be read from the stream.
8. Finally, the operating system closes the underlying file which is indicated by the close event emitted from the readable stream.

Depending on the purpose of the readable stream, they may emit other implementation-specific custom events as well. In this example, we only saw the different stages of a readable file stream. But the concept of flowing mode and paused mode is common in all readable streams.

A trivial pipe

Just like a pipe used to connect a water source to a particular destination, a ‘pipe’ in streams is used to connect a readable stream to a writable stream. A pipe. Let’s recall what a pipe is from the previous article.

In the following snippet, we are creating a readable file stream using fs.createReadStream and pipe it to process.stdout writable stream to be printed on the shell.

process.stdout is probably the most common and well-known writable stream. It’s called the “standard output” and is used by console.log.

If you run this example on the terminal, you will see the content of the foo.txt gets printed on the terminal.

This is foo!!!

An echo server

Let’s now look at how we can use pipes in a slightly different use case. The following is an ‘echo server’ which accepts a request from a client and reflects the request content (body) back to the client.

Let’s try to understand what req and res mean.

When a client and a server communicate with each other, they do it via ‘sockets’ created and maintained by the operating system at each end. In NodeJS applications, we can communicate with such a socket using a TCP Socket stream. As we saw in the previous article, a TCP socket in NodeJS is a duplex stream. Therefore it can be read from and written to.

However, NodeJS http module provides additional functionality for HTTP on top of TCP sockets and exposes two separate streams for reading from the socket and writing to the socket. They are,

• IncomingMessage — Readable stream from the socket. Can be accessed by req object in the request handler in the example. NodeJS http module reads the incoming request content from the socket and pushes into the IncomingMessage readable stream.
• OutgoingMessage — Writable stream to the socket. Can be accessed by res object in the request handler in the example. NodeJS http module takes the data written to OutgoingMessage writable stream and passes it to the socket stream. This data is then written into the actual TCP socket created by the operating system and is sent to the client.

With this information in mind, we can visualise our echo server as follows:

In this echo server, we are simply piping req readable stream to res writable stream so that the request body content is reflected back to the client.

To see this in action, start the server and run the following curl command.

curl -d"hello" http://localhost:3000

You will see “hello” returned in the response.

Pipe multiple times

So far we saw how pipe can be used to connect a readable stream to a writable stream. pipe can also be used to connect one readable stream to more than one writable stream. In that case, the content from the readable stream is copied to the two writable streams.

We can demonstrate that using the following example. In this script, we are creating two copies of foo.txt as copy1.txt and copy2.txt using pipes.

Though this is a very naive example, multiple pipes can be very useful in real-life applications. One such example would be in an http server, where you want to proxy an incoming request to an upstream server at the same time reading the incoming request content for logging and monitoring.

⚠️ Problem with pipes:

Pipes are very useful and convenient, but it can be very tricky to handle errors when you pipe multiple streams together. The reason is that pipe does not automatically destroy and clean up readable or writable side if an error occurred in the other end. This can lead to memory leaks. Therefore, it’s wise to use require('stream').pipeline function to pipe streams in production applications. It takes care of tracking errors and cleaning up in those types of situations. You can read more about pipeline function here in the official docs.

A hash server

In the previous examples, we discussed readable and writable streams. Let’s now look at how we can combine a readable stream, transform stream and a writable stream together.

The following is a simple server that generates a SHA-512 hash for a provided request body content.

In this example, I’m using pipeline function instead of using .pipe like in the previous examples.

The following diagram visualises our hash server’s functionality.

Let’s try to understand this flow step by step:

• As we discussed before, req is a readable stream that we can use to read incoming request content.
• createHash function in crypto module creates a hash stream. It is a transform stream that we can write our content into and receive the hash of the content back. By piping req readable stream to hasher transform stream, we are passing the incoming request body to be hashed.
• res object is a writable stream. When hasher is piped to res, generated hash data flows to the res stream and eventually gets written to the socket.

Let’s first start the server and use the following curl command to see it in action.

curl -d"test" http://localhost:3000/hash

You will see the following output in response, which is the base64-encoded SHA-512 hash of the text “test”.

7iaw3Ur350mqGo7jwQrpkj9hiYB3Lkc/iBml1JQODbJ6wYX4oOHV+E+IvIh/1nsUNzLDBMxfqa2Ob1f1ACio/w==

Streams are widely used in NodeJS applications due to their efficiency and low memory footprint. Although we looked at how we can use them using a few small examples, they can be very useful in real-life applications to process large files and large amounts of data with relatively low memory consumption. If you are keen on learning more, I also strongly recommend reading the official NodeJS documentation for streams, which is very detailed and full of useful examples.

I’d like to thank you for reading, and appreciate it if you could let me know your feedback.

Further References:

• HTTP Module https://nodejs.org/api/http.html
• HTTP Module Implementation https://github.com/nodejs/node/blob/3ac223ed25db4e4b89799f3e60ad8dd6ce454884/lib/http.js
• Stream Module https://nodejs.org/api/stream.html
• File System Module https://nodejs.org/api/fs.html

NodeJS Streams in Practice was originally published in Deepal’s Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Tips and tricks for building your technical writing portfolio

I love writing. I’ve been writing since 2012 on various platforms including Blogger, Wordpress, dev.to, Hashnode etc. on topics including NodeJS, JavaScript, Security, Developer productivity. I feel that, blogging shaped me into a better written communicator as well as a better developer than I was in terms of technical knowledge, thirst for knowledge and curiosity on how things work under the hood.
From the responses I received and through Medium story statistics (Views, View-to-read ratio, Claps etc.), during the past couple of years I’ve learned a few tricks in terms of structuring an article and conveying an idea. Though I still have a lot to learn and improve, I thought of sharing what I learned so far and my approach to writing technical content. Though I might use some Medium-related references, this guide is not necessarily limited to writing on Medium.

Why should you write…

I don’t have the greatest memory. So, I started writing what I learn as a way of refreshing my memory in case if I forget what I learned. Yes, it’s a great reason to write. I frequently revisit my own articles to recall certain topics I no longer remember. Apart from that, there are a couple of other reasons why writing could be beneficial to you.

• Feedback from readers — Positive/Negative feedback, as well as follow-up questions from your readers, are invaluable for learning. Sometimes readers will have questions that you need to learn even more in order to answer.
• Refer to your own writing later — As I mentioned before, you will find yourself occasionally going back to read your own articles to recall your memory.
• Improves your writing skills — Writing technical content will improve your written communication skills. It will help you explain technical things to others clearly and concisely in documentation, emails, chats etc.
• Good for your profile — Having your own blog or a technical writing portfolio will definitely help you stand out. It’ll help you even more, and draw more attention if you carefully plan your writing to cover in-demand but less-documented topics.
• Give back to the community which you thrive upon — How many technical tutorials, blog posts and articles have you read during your career? Now it’s time for you to share what you know and help someone like you.

What to write and How to write…

Everyone has a unique way of writing. Here’s my approach to writing a new article.

• Select a good topic — Internet is vast. Therefore, don’t worry too much that you don’t have some unique topic to write about. No matter how unique we may feel our write up is, it’s always possible that there’s similar content somewhere on the internet. That said, if you write something less common yet in-demand, you will likely draw more crowd.
• Create a draft outline — Once you have selected a topic to write about, create a draft first and then outline the sections and sub-sections to fill later. This can be done based on what you have off the top of your head and doesn’t have to be the final outline of the article.
• Spend a lot of time researching — When the draft outline is ready and you know what you want to write, then start researching about the topic. In my case, I refer to as many resources as possible about the topic and try to get a deep enough understanding of the subject. The more you research, the more questions you will have which will force you to learn even more. Having a deep understanding of the topic will help you write a concise article and answer questions readers may have.
• Write — Once you are done researching, check if your outline is still relevant. If not, modify it and get the final outline. Once done, fill the sections and subsections with the content you’d like to write. I wouldn’t worry too much about typos, grammar, understandability and how detailed my writing is at this stage. The goal is just to fill the sections and subsections and get a complete draft.
• Refine, Refine, Refine — Now it’s time to refine your write-up. Read the article from start to bottom from a reader’s perspective and fix any typos and grammar issues. As you read, you will find unclear content in the article. Try to reword those content or add more details to convey the point clearly. Do this a few times until you are confident that it’s ready to be published. It might also be a good idea if you could ask someone to proofread it before publishing.

What you shouldn’t do…

• Too detailed — If an article is too long and detailed, it’s very likely that readers lose interest in reading it halfway through. Medium’s product science team has found out that the optimum post is 7 minutes long based on the reader engagement statistics. But it doesn’t mean that you shouldn’t write longer articles. You can use code snippets, diagrams, graphs, images, GIFs etc. to keep the reader engaged in a long article.
• Unnecessarily broken down to multiple parts — I’ve seen many occasions where writers on Medium are too focused on earning money therefore have unnecessarily broken down their articles into a series of very small articles as Part 1, Part 2 etc. This makes each individual part of the article series useless on its own. Doing this won’t give you any significant short term monetary benefit, but will only hurt your profile in the long run.
• Text-only —It’s not a secret that the readers are very likely to engage with an article more (more followers, more reads and more claps) if it contains visuals such as diagrams, graphs, images etc. Text-only articles are very hard to follow and boring, especially if they are too long. If you embed code snippets, make sure that they are short and easily understandable. Also, it would be better to embed code snippets in ‘pretty’ format using GitHub gists or a similar tool rather than pasting them directly.
• Clickbait titles — Medium is bloated with clickbait titles. There’s nothing wrong with starting your article title as “5 things you should…”, “Top 10 things you must…” & etc. But if you do, make sure your article is actually worth a visit for a reader. If the content is also as juicy as the title, people are very likely to follow you and engage more. Clickbaits can harm you in the long run and damage your portfolio. They are likely to annoy readers and repel them away from your future content.

I wrote this guide from my personal experience as a writer as well as a reader. I’d be very glad to hear your thoughts if you have a different opinion on any of the points I mentioned. In that case, please don’t hesitate to respond.

Starting Your Own Tech Blog was originally published in Deepal’s Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Imagine you have a pile of bricks somewhere. And you want to build a wall with those bricks here. Let’s say you have a friend to help move the bricks. To start building, you now have two options. You can either wait until your friend brings the whole pile of bricks to you, or you can start building as soon as you have a few bricks to start with, while your friend keeps bringing more bricks.

Which would be efficient? Clearly, it will be efficient to start building as soon as you have a few bricks to start with. This is a classic example of how a ‘stream’ (in this case a stream of bricks) can improve the efficiency of a process. Another common example you might be very familiar with would be streaming a movie instead downloading the whole movie first and and watch it.

Wikipedia says,

In computer science, a stream is a sequence of data elements made available over time. A stream can be thought of as items on a conveyor belt being processed one at a time rather than in large batches.

In NodeJS, stream module provides the capability to work with streams. Even if you haven’t used stream module explicitly there are a lots of underlying functionality in NodeJS applications which use streams. “Streams” is an easy concept, but it may sound very complex if you are unfamiliar with it. Therefore, I thought of describing a few key concepts in NodeJS streams in visuals so that it can be easily understood.

This article is the 4th article of my Advanced NodeJS Series which is a sequel to my series on Node.js Event Loop. You can find the other articles of the Advanced NodeJS series below:

Post Series Roadmap

• Crossing the JS/C++ Boundary
• Deep Dive into Worker Threads in Node.js
• Understanding Async Resources with Async Hooks
• A Visual Guide to NodeJS Streams (This article)
• NodeJS Streams in Practice
• Why you should avoid Date.now() to measure time durations

Water flows, Information flows.

Information is like a liquid. It flows from one place to another place as a stream of bits. For example, this happens when two peers talking to each other via a network, or even when your application is communicating with the disk or a peripheral device. When such an I/O operation happens, the information is read from a device and is flowed towards an application, or vice versa.

However, it’s possible that one end of the above transaction is slower than the other for various reasons. Therefore, some of the data might need to be “buffered” in-between, while the receiver-end is ready to accept more data.

Have a look at the following picture where two faucets of different sizes are connected via a tank. The rate water flows from upstream is higher than the rate downstream can consume. Therefore, the tank has to temporarily store (“buffer”) the excess water while the downstream slowly consumes.

This is the fundamental idea of streams in NodeJS. stream module provides functionality to implement this behaviour when working with streaming data. There are two basic types of streams provided by NodeJS.

They are,

• Readable Streams
• Writable Streams

However, there are two additional types of streams which are hybrids of Readable and Writable streams, and serve special purposes.

• Duplex Streams
• Transform Streams

Let’s get into more details, and try to visualise each of these.

Readable Stream

A readable stream can be used to read data from an underlying source such as a file descriptor. Data can be stored in a Buffer within the readable stream if the application consumes slower than the operating system reads from the source.

A few of the most common readable streams in NodeJS are process.stdin, fs.createReadStream and IncomingMessage object in an HTTP server.

Writable Stream

A Writable stream is used to write data from an application to a certain destination. To prevent data loss or overwhelming of the destination in case the destination is slower than the writing application, the data may be stored in an internal Buffer.

The most common writable stream which you might use every day would be process.stdout which is used underneath console.log. In addition to that, two other very common writable streams in NodeJS are process.stderr and fs.createWriteStream

Duplex Stream

As I mentioned earlier, a duplex stream is a hybrid of a Readable stream and a Writable stream. An application connected to a duplex stream can both read from and write to the duplex stream. The most common example of a duplex stream is net.Socket. In a duplex stream, read and write parts are independent and have their own buffers.

Transform Stream

A transform stream is an even special hybrid, where the Readable part is connected to the Writable part in some way. A common example would be a crypto stream created using Cipher class. In this case, the application writes the plain data into the stream and reads encrypted data from the same stream.

Probably the most trivial transform stream is PassThrough, where the input data is simply passed as the output with no transformation whatsoever. While it may sound very trivial, I’ve used it many times to implement custom behaviours with streams.

Piping Streams

In many cases, streams are even more useful when connected together. As obvious as it may sound, this is called ‘piping’. You can connect a Readable stream to another Writable/Duplex or a Transform stream by using the Readable stream’s pipe() method.

A trivial example for this would be copying a file from one place to another place by piping an fs.createReadStream() to an fs.createWriteStream().

Copying Data with Streams

One interesting fact about piping is that you can pipe the same stream more than once. This can be very useful in situations where you need to read the same stream twice, as you cannot read from a Readable stream once again after it’s completely read by another consumer. However, by piping a Readable stream more than once, more than one consumers can read the same stream by copying the data from the original Readable stream.

Have a look at the following simple example, where we create two copies of the original.txt file.

The above program can be visualised as follows:

Backpressure

Let’s go back and recap our water tank analogy. In that analogy, the upstream flows at a higher rate than the downstream can consume. In this case, the water level of the tank will keep rising until at some point it overflows and the water goes to waste.

What if we can ‘detect’ it’s going to happen and let the upstream know to stop the flow? We can mark the tank below the highest level, and ask the upstream to stop when the water level rises above the mark.

This is very similar to how streams work. While Readable and Writable streams can buffer data internally, the amount of data they can buffer is limited by the total available memory of the system. Therefore, streams have a threshold called highWaterMark which can be used to detect if the rate at which data is passed into the stream is way higher than the rate at which data is flushed out of the stream.

As an example, when a readable stream is piped to a writable stream, the writable stream can ask the readable stream to stop the flow if the writable stream’s buffer fills beyond the highWatermark.

While highWaterMark is not a hard limit but only a threshold, it’s important to adhere to that threshold when building custom streams to avoid data losses or undesirable memory usage. You can read more about this in the official documentation.

Let’s put them all together

So far, we discussed different types of streams and how to use them. Let’s now try to put them all together and visualise a real-life example.

The following is a design of a simple image service I built some time back. In this service, an image is retrieved from an S3 bucket, and served as a resized image to the end-user.

If we build this without streams, we will fetch the whole file from the S3 bucket and keep it in the memory, resize the whole image at once, and then send the resized image to the end-user.

However, with streams, we can chain the above process in a very efficient manner to improve speed and optimise the memory usage in our application.

Since the data stream coming from the S3 bucket is a readable stream, we can pipe it straightaway to a transform stream where it’s resized. Since the transform stream is also readable, we can pipe it to the response stream directly so that a chunk of data coming from the S3 bucket is resized and sent to the user without having to wait for the whole file from S3 bucket.

If you are curious, feel free to try to build this API on your own and let me know if you have any questions. Tip: You can have a look at sharp NPM module which can be used to build the image processing capability. Although, sharp uses Duplex streams instead of Transform streams at the time of this writing due to the reason mentioned here.

Ready to dive into how NodeJS streams can be used to build complex applications? Check out my the next article about ‘NodeJS Streams in Practice’ below.

NodeJS Streams in Practice

In this article, I wanted to visualise a few of the key concepts in NodeJS streams for ease of understanding. Please let me know if you have any feedback on this article which will be very helpful for me not only to improve this article but to write more helpful and detailed content in the future. I plan to write a few more detailed articles about the advanced concepts in streams, memory leaks and many more. So stay tuned in. Thanks.

A Visual Guide to NodeJS Streams was originally published in Deepal’s Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Advanced NodeJS — Part 3

This article is the third article of my Advanced NodeJS Internals Series, which is a sequel to my previous series on Node.js Event Loop. You can find the other articles of the Advanced NodeJS Internals series below:

Post Series Roadmap

• Crossing the JS/C++ Boundary
• Deep Dive into Worker Threads in Node.js
• Understanding Async Resources with Async Hooks (This article)
• A Visual Guide to NodeJS Streams
• NodeJS Streams in Practice
• Why you should avoid Date.now() to measure time durations

Prologue

A real-life NodeJS application can be very complex in terms of the asynchronous operations it performs, and the asynchronous resources that are created and destroyed as part of those operations. The ability to inspect the lifecycles of these asynchronous resources from user-land code can be very useful because it can provide us a lot of insights about the application, especially about its performance and potential optimisations that can be done.

There has been a lot of work done by the Node community to expose this advanced functionality to user-land code (previous attempts with AsyncListener and async_wrap) and thanks to all that effort we now have async_hooks, a mature implementation to trace the lifecycle of asynchronous resources. It’s still documented as “Experimental” at the time of this writing, but it has been polished and shaped over time, to be more stable than a typical experimental API.

async_hooks module provides a world of new capabilities, but what I’m mostly interested in as a Node.js enthusiast is, it allows you to easily understand what happens under the hood of some of the tasks we frequently perform in our applications such as reading a file, making an HTTP request, spawning an HTTP server, perform a database query, etc., in a way that was not possible before.

In this article, I will try to demonstrate and explain the life cycle of a typical asynchronous resource with the help of async_hooks module.

Before moving on…

Please note that this is not a tutorial for async hooks since I didn’t want to repeat the official documentation in a blog post. If you are completely new to async_hooks or would like to learn learn how to use async hooks, I’d strongly recommend first reading official docs and this very useful article on “A Pragmatic Overview of Async Hooks API in Node.js” from Andrey Pechkurov. Also, I would assume that you have a basic understanding of how Node JS asynchronous model works.

The lifecycle of an Asynchronous Resource

Async resources are created as part of asynchronous operations. Async resources are nothing but objects that are used to keep track of asynchronous operations. Therefore, they are naturally associated with a callback that will be executed once the operation completes. Once the async resource serves its purpose, it can be garbage collected like any other object.

A very simple example would be setTimeout. setTimeout function returns the async resource (Timeout) that is creates to track the timer as the return value of the function. You can see it by calling setTimeout in the Node REPL as follows:

This object contains useful information such as:

• Its timeout value (_idleTimeout)
• Timer callback (_onTimeout)
• Whether it’s a timer or an interval (_repeat)
• whether the timeout is active or not (_destroyed)

The typical lifecycle of such an async resource is similar to this:

What “Async hooks” facilitate you is the ability to spy on different stages of the above lifecycle by providing “hooks”, to which we can attach callback functions. There are four common types of such hooks as init, before, after and destroy. And they are fired at the following four stages of an async resource’s lifecycle.

Depending on whether the async resource is persistent or not (I will get to this later), the async resource’s callback may be executed 0 to many times. Therefore, it can be expected that before and after hooks may be called multiple times for certain async resources, or may not be called at all. For example with setTimeout, before and after each will only be fired at most once followed by an init, whereas with setInterval, there can be multiple before and after invocations followed by one init.

For example, both setTimeout and setInterval create Timeout async resources. However, the Timeout resource created by setInterval is a persistent Timeout resource and can be identified by the non-null __repeat property.

Promise objects are slightly different and they have one more additional hook called promiseResolve which is fired right after the promise is resolved or rejected.

Async Resources in real-life applications

In a real-life application, one async resource may trigger lots of async resources during its lifetime. With the above-discussed details in mind, let’s have a look at the following hypothetical example of an HTTP request handler.

The purpose of this HTTP request handler is to

• Receive some data via HTTP
• Store it in a database
• Notify an upstream service via HTTP
• Write a log message to stdout

Assuming that the above four operations create only four asynchronous resources, let’s try to plot the timelines of those four resources in a graph as follows (the lengths of the timelines are not drawn to scale). Feel free to spend a few minutes to read the graph, and try to correlate the timelines of the resources with what we discussed about the lifecycle of the async resources.

A few important details from the above graph are:

• DB Operation timeline doesn’t span beyond the end of HTTP Client Request and Write Log Message timelines because the callback of the storeInDB function doesn’t wait until notifyUpstream and writeLog functions complete their async operations.
• Write Log Message is also an async operation, hence it creates an async resource. This resource is short-lived compared to the other resources.
• Incoming HTTP Request is the last resource to be destroyed because it will only be destroyed once the notifyUpstream completes and the response has been completely sent.

Practical examples with Timers

Since we now have a theoretical understanding of the lifecycles of the async resources, let’s have a look at some practical examples. For this demo, I’ll be using a few code examples with async hooks set up. You can check out the source code for the examples here: https://github.com/deepal/async-hooks-demo. And I’d like to suggest reading the README.md of the repository and set up the examples in your local machine before continue reading.

Setting a timer

(Source code, Used NodeJS version: 14.16.1)

As a start, let’s look at the simplest example. In this example, I’m creating a timer with 1000ms timeout and writing “timer callback” text to a log file using logger.write function.

When I run the above piece of code along with my async hooks set up, I will see the following output in my log file.

According to this output,

• A Timeout async resource was initialised when setTimeout was called.
• After the 1000ms timeout expired, before async hook was invoked right before the timer callback was executed.
• Timer callback executed and "timer callback" text was logged.
• After the timer callback executed, after async hook was invoked.
• Timeout resource was destroyed and the destroy hook was invoked.

Setting a nested timer

(Source code, Used NodeJS version: 14.16.1)

The above example produces the following log.

In this example, the Timeout resource of the outer timer triggers another Timeout resource when the timer callback is executed. You can see that the inner timer’s Timeout resource has triggerAsyncId as 2, which is the asyncId of the outer timer’s Timeout resource. This indicates that the inner timer was triggered by the outer timer.

However, it’s worth mentioning that the outer Timeout resource was destroyed before the inner Timeout resource because the outer timer doesn’t wait until the inner timer is expired and the callback is executed.

Clearing a created timer

(Source code, Used NodeJS version: 14.16.1)

In the following example, we create a new timer by calling setTimeout, and the created timer is immediately removed by calling clearTimeout and passing the created Timeout object to it.

In this case, we don’t see BEFORE or AFTER log messages because the timer was immediately removed and the callback didn’t get the chance to be executed due to the clearTimeout call. Therefore, before and after hooks were not invoked.

Creating an interval

(Source code, Used NodeJS version: 14.16.1)

The following code creates an interval to schedule a callback to be executed every second, 3 times.

Similar to setTimeout, this also creates a Timeout async resource. However, the Timeout resource created by setInterval is a persistent async resource. In persistent async resources, before and after hooks may be called multiple times before the resource is destroyed. In this example, the timer callback is called 3 times. Therefore we can see that the before and after hooks fired 3 times before the timer was destroyed by clearInterval.

Practical examples with Custom Async Resources

So far we talked about Timeout objects which are native NodeJS async resources. async_hooks module also provides the facility to write your own custom async resources by using the JS Embedder API’s AsyncResource class.

You can learn more about AsyncResource class and how to use it from the official docs.

A Custom Async Resource with automatic destroy

(Source code, Used NodeJS version: 14.16.1)

The following is a custom async resource I’ve written which aims to track a database query. For the sake of simplicity, I haven’t added any actual DB operations.

Before moving any further, I’d like to point out a few key pieces from the above code.

• requireManualDestroy flag is set to false. Which instructs that, if there are any destroy hooks for the resource, they should be automatically fired when the resource is garbage collected. This is internally done in NodeJS by registering the destroy hook as a “Weak Callback” on the resource object in v8. (v8 Weak Callbacks are scheduled to run when the object is garbage collected).
• At the end of the code, I’ve added a large timer which causes the application to keep running for some time before exiting. I’ll get to the reason behind this in a moment.

Let’s run this program with --trace-gc Node CLI flag so that we can see the garbage collection logs.

As you may have noticed, the resource is garbage collected during the immediate run of Mark-sweep and it triggered the destroy hook.

Since setTimeout callback doesn’t use the dbquery object, after the setTimeout is executed there will be no references to the dbquery object to keep it from being garbage collected. This explains why it was garbage collected immediately.

What if we do not use that timer at all?

If we do not use this timer, the application will exit immediately and the garbage collector will not get the opportunity to run. Therefore, the destroy hook won’t be fired at all. For this purpose you can also use any other async operation which keeps the program from exiting. But I used a timer to avoid the logs being cluttered too much.

Let’s now look at another example, where we keep the reference to dbquery object inside the timer callback.

If we run this with --trace-gc Node CLI flag, this will produce the following result.

You might notice that the destroy hook was not called at all because the resource was not garbage collected even if the garbage collection was run. The reason for this is that the created Timeout async resource is keeping a reference to the dbquery object inside the timer’s callback, thereby preventing the dbquery from being garbage collected.

A Custom Async Resource with manual destroy

(Source code, Used NodeJS version: 14.16.1)

In the previous example, we discussed async resources which are constructed with requireManualDestroy as false and observed how they automatically fired destroy hooks upon garbage collection. However, if we set requireManualDestroy to true when it was constructed, destroy hooks won’t automatically fire when the object is garbage collected, and you will have to fire them manually by calling emitDestroy() on the async resource.

Let’s look at the following example. In the following async resource, we have a destroy method that calls AsyncResource’s emitDestroy prototype function.

The above code produces the following output when run.

As you can see, destroy hook was immediately called after emitDestroy. If we omit dbquery.destroy() statement, the destroy hook will not be fired even if the object was garbage collected.

In this article, I tried to cover a few practical examples of using async hooks to understand the lifecycle of an async resource. To avoid this article being too lengthy, I didn’t delve into other types of async operations such as HTTP requests, File system access, Crypto operations, etc. But feel free to experiment with the other examples in the examples repository, and let me know if you have any feedback/questions or suggestions. Thank you very much for reading.

You can follow me and https://blog.insiderattack.net to get updates about future posts.

Further References

• Async Hooks | Node JS Docs https://nodejs.org/api/async_hooks.html
• A Pragmatic Overview of Async Hooks API in Node.jsA Pragmatic Overview of Async Hooks API in Node.js https://itnext.io/a-pragmatic-overview-of-async-hooks-api-in-node-js-e514b31460e9
• async_hooks: add AsyncLocal API mk2 https://github.com/nodejs/node/pull/31016
• async_hooks: introduce mandatory store argument in AsyncLocalStorage run methods https://github.com/nodejs/node/pull/31930
• src: remove TimerWrap https://github.com/nodejs/node/pull/20894

Understanding Async Resources with Async Hooks was originally published in Deepal’s Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.