The 3-Second Production Mystery That Took Me 20 Days to Solve

This is a story about how we found a 3s EventLoop lag (p99) in one of our microservices while exploring the Kafka consumer lag… and how I tracked it down and fixed.

It’s Feb 6. I notice an increase in Kafka lag in our Grafana chart, so I ask team members if they performed any actions that could have increased the traffic. I get a negative answer.

Together with two other senior engineers, I jump on a call to understand what’s happening. One engineer suggests a hypothesis:

Every time we restart the servers and reconnect to Kafka, things behave differently at the beginning. Kafka Cluster works efficiently in production at first, but after some time, it might do internal calculations/optimisations and start processing the queues with bigger lags.

I was very sceptical of this idea, and didn’t believe the Kafka cluster could behave like that, even though I see the Grafana chart showing the lag drops to 0 every time we restart our Node.js consumer services.

I request approval to perform a test in production: simply restarting the services to see if the same thing happens. I do it; the lag drops to 0, and I still don’t believe that's the real reason.

We spent 3 hours checking the logic, Grafana charts, and anything else we could think of, because I can’t accept that Kafka would behave like that.

After 3 hours, we find this picture…

And yet it’s February 26th: after 20 days, I was finally able to start writing this article, because it took that long to track down the issue, reproduce it, test it in dev, and fix it.

The Event Loop usually works very fast: it’s on the order of milliseconds. Around 10 ms is a pretty good number, but if you see it increasing (or getting much bigger), you’re in huge trouble.

Even though we had this issue in production for a month, we didn’t notice it because our alerting wasn’t set up properly. We also didn’t feel a big impact because traffic wasn’t that high yet. But once we discovered the issue, the first thing I said was: we can’t accept more and more traffic, because this will only get worse over time. We need to understand the root cause and fix it as soon as possible.

I could clearly see when it started because we had historical Event Loop lag data in Grafana. And guess what the first thing I checked was: what we released that day (Jan 16, based on the picture above) and what changes we included. After reviewing it, it was pretty obvious which commit/MR introduced the problem.

The issue was that it was a big refactor: we couldn’t simply revert it, and we couldn’t easily understand the root cause just by looking at the file changes.

So, guess what: I got really happy, because it was a challenging and interesting problem that had to be solved. And I jumped into the work…

Reasons Event Loop Lag Can Grow

There are a couple of reasons why event loop lag can grow. It’s mostly because we have:

• Synchronous JavaScript work that blocks the event loop.
• Too many variables/objects are being created and not cleaned up, so GC takes longer and ends up blocking the event loop.

Thanks to the proper Grafana charts, we also found this chart, along with the previous one above.

Resident Set Size (RSS) is the portion of a process’s memory currently held in physical RAM.

So it’s already obvious that we’re creating things that aren’t being cleaned up, and because of that, the event loop slows down.

Now … we know what we’re looking for.

Fixes and Changes I Made

The first thing that caught my eye in that MR was a caching solution for Kafka batch processing. Basically, the consumer was handling 100 messages per batch, but before processing them it was creating a cache using JavaScript Maps and loading data from the DB into those maps, so the batch processing could reuse it for each message in the batch.

function createCache() {
  return {
    entity1: new Map(),
    entity2: new Map(),
    entity3: new Map(),
    ...
  };
}

So it was doing this for every batch. The naming also bothered me and kept pushing me toward the idea that this could be the memory leak. I tried to track down whether those Maps could be referenced by a function or object that lives longer than the batch (or maybe something running in the background), so they wouldn’t get cleaned up. I couldn’t find anything obvious, but I still removed that logic and implemented it differently, just to be sure my changes didn’t leave any references that could prevent the maps from being garbage-collected.

In addition to those changes, I tried to clean up anything unnecessary that caught my eye and disable any additional processes related to batch-metrics performance tracking.

Performance Tracking Implementation

There was a class responsible for collecting some statistics about batch processing, and it was enabled in the main file. The related functions were injected into the Kafka consumer, so they could generate data for those statistics.

Inside the performance tracker class, there was a setInterval running every 1 second, that printed the results (simply console logs). The environment check (“don’t run outside dev”) existed only inside the print function, so even with that check, the other methods still ran regardless of the environment and kept generating statistics data.

I simply disabled this performance tracking implementation entirely (for all environments), and along with my other changes, prepared everything for the next release, which I labelled as a potential fix for the event loop lag.

Reproducing the Issue in the Dev Environment

We decided to do some stress testing to reproduce the memory leak and observe the event loop lag. Even though we didn’t have Grafana charts for the dev environment, we exposed Prometheus collectDefaultMetrics via an HTTP endpoint. That helped me stress-test the service while monitoring RSS and event loop lag to make sure I was actually reproducing the issue.

I was able to reproduce it by sending 800K Kafka messages through the batch processing. On top of that, I also proved that my changes fixed the situation.

Even though I still couldn’t 100% understand which exact line caused the issue.

Releasing the Fix to Production

So the important day came, and we decided to release the changes at 8 AM during low traffic.
We kept a rollback plan. If the new changes didn’t work, we could revert to the previous broken version, which was slow, but at least it was working reliably.

We released it and monitored the logs and charts closely, along with the data generated in the database after batch processing.

And this was the picture after a couple of hours.

Beautiful, isn’t it!?

Even though we were doing batch processing and committing the offset only after 100 messages were processed and saved to our database in a single transaction, I was sure the event loop lag would drop. We would still have some lag because of batch processing, but it wouldn’t be that big.

And here is proof that I was right, again!

It was finally the time when I could rest and breathe without thinking about the issue anymore.

Solving the Mystery: The Root Cause

Even though I fixed the issue, I still didn’t know which exact line caused it: and I really didn’t like that.

• How could I be sure it wouldn’t happen again?
• How could I write this article without understanding the root cause? (hehe)

So I spend few hours during the night to dive deeper into the MR changes from Jan 16.

After some investigation, I finally understood…

Remember the performance tracker I was talking about? It was enabled, but it simply didn’t log anything in prod. And the related functions were generating a huge amount of data inside an array. It kept pushing data into the array and never cleaned them up, so they just kept growing.

this.latencies.push(...);

JavaScript Garbage Collector

If we have too many JavaScript object references in the heap, GC will slow down the event loop because it runs on the main thread. GC has to scan the JS objects and their references, and that’s exactly what was happening to us.

Even though there are ways to keep data in memory and reduce how much GC blocks the event loop, in general, using too much JS heap can cause big trouble like this.

The AI-generated code (Claude Opus 4.6)

Obviously, this code was generated with AI. But we weren’t using it properly, and it also wasn’t generated with safeguards.

It was a huge class with many methods: around 500 lines in total. It had a reset method that would clean the data, but reset was never used. That was the primary issue. Another problem was that the AI could have added safeguards to prevent the arrays from growing infinitely.

So you probably get my point: AI can’t fully replace engineers. Who else would have tracked this issue down and fixed it if it… hehe

Summary

This was an interesting production bug that I really enjoyed tracking down, understanding, and fixing: although in this case, it was the other way around: I fixed it first, and only then understood the real reason.

It took around 20 days, but luckily, I was able to track it down relatively easily without using advanced tools (e.g. Clinicjs) or doing a heap dump to monitor Node.js memory usage. Otherwise, it could have taken much longer.

I hope you enjoyed reading this article and that it gave you some insights and ideas on how to handle situations like this. If you’re interested, I highly recommend reading my article about the event loop: “You Don’t Know Node.js Event Loop.”

Thank you for taking the time to read this comprehensive article. I hope you found it informative and gained valuable insights from it.
Feel free to ask any questions or tweet me @nairihar

Also, follow my JavaScript newsletter on Telegram: @javascript

• You Don't Know Node.js EventLoop
• An Advanced Retry Mechanism in Node.js with Kafka
• How to Scale Node.js Socket Server with Nginx and Redis

Node.js EventLoop Lag + Kafka Consumer Lag: One Root Cause was originally published in JavaScript in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

Reliable Event Delivery with an Easy-to-Monitor Approach

Have you ever wondered how systems are designed to ensure reliable event delivery and processing? How can we minimise event loss to a very low level: or even achieve near-zero message loss and highly reliable processing?

In this article, I’ll show you an example of Node.js architecture that ensures event processing and minimises event loss, keeping it almost zero. Even during deployments or server crashes, data loss stays minimal. I’ll also demonstrate how this approach enables you to monitor and debug every part of the system, including failures and the retry processes.

Here is the list of topics covered in this article.

— Acknowledgment
— Basic Retry Mechanism
— Why do we need to store the event?
— — — Kafka as an Alternative
— — — Kafka over SQL DBs and Redis
— Kafka as a Solution
— — — Example 1: speed-over-safety
— — — Example 2: lossless-over-speed
— — — CRONJob Retries with Kafka
— Processes should be idempotent
— Kafka UI: Monitoring and Debugging
— — — Visualisation and Alerts
— Summary

Acknowledgment

Lately, I ran into an interesting issue related to this “Retry” topic.
An Internal Service should send an event to an External System.

But after looking closely at the requests and service behaviour, I realised that the Internal Service can fail to send some events to the External System for many reasons:

• If E.S. is unavailable for a while, I.S. will fail to send the event.
• If I.S. starts the request but crashes because of an unexpected failure, then the request won’t be delivered.
• And so on.

It was clear that I needed to implement a retry mechanism to solve this problem, because event loss can be a serious issue.

Before jumping straight into the topic, I want to highlight that every system is unique, and every problem needs its own solution. You can’t just copy this approach into any system that needs a retry mechanism. Depending on your system, available tools, environment, and overall situation, you might choose a different approach. This solution worked well for my case, but you might need something much simpler, or even a more advanced version, based on your needs.

I’ll explain why I chose this approach, what other options I considered, and why I didn’t pick any of those.

Basic Retry Mechanism

Here is a very basic retry mechanism that tries to run a function (for example, a fetch request). If the provided function fails, then our retry mechanism waits for 3 seconds and tries again, up to 3 times.

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function retry(fn, retries = 3, delayMs = 3000) {
  let lastError;

  for (let attempt = 1; attempt <= retries; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err;

      if (attempt === retries) {
        throw lastError; // final fail
      }

      console.log(`Attempt ${attempt} failed. Retrying in ${delayMs}ms...`);
      await sleep(delayMs);
    }
  }
}

What do you think:

• Is this an ideal solution?
• Does it prevent event loss?
• What kind of issues might you face with this approach?

Of course, this can help a bit, but it’s not an ideal solution. Here are a couple of reasons why:

• What if the service crashes during the retries? The message won’t be delivered, and we’ll lose that event.
• What if the destination system/service is unavailable? We will retry three times over 9 seconds, but what if the service is down for longer than 9 seconds? In that case, we’ll still lose the event.

So this solution is not bulletproof.

Why do we need to store the event?

It’s quite obvious that if we want to prevent such losses, we need to store the event somewhere first. But where should we store it? That’s the main question.

Should it be MySQL/PostgreSQL, Redis, or something else?

If you think about it, an SQL database is probably a better option than Redis, or maybe that’s just my preference. But I’ll try to explain why.

With Redis, of course, everything feels easier because we don’t need a strict structure, and the event payload can change over time. But for this case, I don’t like Redis as an option because it can make debugging harder. In general, we can install Redis Insight and monitor the data inside Redis, but if we compare it with other solutions, and if you’ve tried them, you might understand why I say this.
Btw, one very critical point: Redis must be configured with persistence. Otherwise, we might lose the data if Redis restarts. For me, it was an option because Redis was already configured to be persistent.

With an SQL database, we need a basic structure and at least one JSON column to store the event payload. But debugging becomes much easier: we can quickly inspect events and check their retry count with a simple query.

Whatever storage we choose, we need a job that runs periodically, checks for failed processes, and retries them. But there is one very important part in this flow which makes everything a bit harder when using an SQL database or Redis. I’ll explain it in a bit.

Kafka as an Alternative

If you’ve never worked with Kafka, I’ll briefly introduce it to you. It’s a message queue where one or multiple service(s) produces messages and another service consumes them. Messages stay in the queue until the consumer marks them as processed.

You probably get the point: Kafka fits our problem much better than the storage options we discussed above. Why? Because it already provides a native way to produce and consume events, and it has a built-in mechanism to mark the messages as processed. With the other solutions, we would need to implement some things by ourselves.

For monitoring and debugging, we can install Kafka UI, which shows all the queues(topics) and their messages with the details.

Kafka over SQL DBs and Redis

Remember, I mentioned that there is one part that can be implemented much better with Kafka compared to the other options.

Let’s say there is a requirement that when the Internal Service has an event, it should send it as fast as possible, without any extra delays.

Here we have two choices:

1. speed-over-safety: Either the Internal Service immediately tries to send the event to the External Service once it’s available. And only in case of failure, it moves the event to the storage.

function processEvent(event) {
  try {
    await fetch(event);
  } catch (err) {
    moveToStorage(event);
  }
}

2. lossless-over-speed: Or the Internal Service immediately stores the event, and only after that, another job/service processes it accordingly and sends it to the External Service.

function processEvent(event) {
  moveToStorage(event);
}

In the first option (speed-over-safety), the downside is that if the server crashes during the fetch request, we will lose the event. Also, fetch requests can take much longer because the request goes outside of your system, and you don’t control it anymore. On the other hand, the benefit of this solution is that you won’t add extra delay to the request if everything is fine.

In the second approach (lossless-over-speed), we are much more secure because the chances of losing data are significantly lower. Once the event is available, we store it first. Of course, that’s also a request, but it can be an internal request inside our cluster, meaning we control everything. So writing to storage should be much faster, which reduces the risk of data loss if the server crashes. However, the downside is that we delay the delivery of the event to the External Service.

First of all, for both options (speed-over-safety & lossless-over-speed), we would need to create a cron job that runs every X minutes and retries sending the failed events.

For the lossless-over-speed approach, a cron job can’t be the primary solution, because it runs periodically with a fixed interval, and that interval alone could already be too long a delay. We need a better mechanism to deliver the event as soon as possible once it’s stored. We need a solution where the event is emitted in real time for processing. The idea is that we save the event to storage first, and the first fetch request should happen as soon as possible. After that, the remaining retries can be handled with a cron job, which will bring some delays.

Of course, we can use Redis Pub/Sub or PostgreSQL with additional extensions to work with real-time events instead of manually pulling them via read operations. However, Redis Pub/Sub doesn’t provide delivery guarantees: it simply lets us publish events and subscribe to them, so we’d still face issues with non-guaranteed delivery in case of system failures. And the PostgreSQL with extensions also feels a bit heavy/overkill for this use case.

And this is the point where Kafka will shine over the other solutions.

Kafka as a Solution

Here is how the record inside the Kafka can look.

{
  "id": "evt_1737429812_001",
  "event": {
    "type": "USER_REGISTERED",
    "timestamp": "2026-01-20T12:34:56Z",
    "userId": "u_98231",
    "email": "user@example.com"
  },
  "retry_count": 2,
  "is_manual_review_needed": false
}

In Kafka, you don’t simply read and write like in traditional storage solutions. Instead, you produce (write) messages, and then consumers receive them once they’re sent to Kafka by a producer.

// PRODUCER (writes to Kafka)
await producer.send({
  topic: "user-events",
  messages: [
    { key: "user_1", value: JSON.stringify({ type: "USER_REGISTERED", ... }) }
  ]
});

In an SQL database, you have tables, but in Kafka, you have entities called topics. A topic is a queue where your messages are stored in the correct order. Once you start consuming, you will receive messages in order from the specified topic.

// CONSUMER (reads from Kafka)
await consumer.subscribe({ topic: "user-events" });

await consumer.run({
  eachMessage: async ({ message }) => {
    console.log("Received:", message.value.toString());
  }
});

This means that when the Internal Service has a message, it can produce the event into Kafka, and it will be delivered in real time to another service (or even the same service) that is consuming the topic. If there is no active consumer, the message will stay in the topic until someone consumes it and marks it as processed.

Here, I mentioned something really important: “mark it as processed.” Kafka has the concept of acknowledgement (also known as committing). When you consume a message, you can acknowledge it to confirm that it was processed successfully, so it won’t be sent to you again.

If you consume a message, but your internal processing fails, and you skip the acknowledgement (commit), you will receive the same message again once your service is back online. This gives you a strong message processing guarantee.

// CONSUMER (reads from Kafka)
await consumer.subscribe({ topic: "user-events" });

await consumer.run({
  eachMessage: async ({ message }) => {
    console.log("Received:", message.value.toString());

    ...

    // ACKNOWLEDGE => message won't be re-delivered
    await consumer.commitOffsets(...);
  }
});

Btw, the consumer can also auto-commit your message if the eachMessage callback function finishes without errors.

Let’s see what the solutions will look like for both options:

Option 1: speed-over-safety

Once we have the event first, we try to fetch, and only after initial failure, we send it to storage for a later retry process.

function processEvent(event) {
  try {
    await fetch(event);
  } catch (err) {
    moveToStorage(event); // producer.send - for retries
  }
}

If the fetch system crashes, we will lose the message. But if the fetch fails, we will continue with the retry processes, and eventually it will be delivered.

Option 2: lossless-over-speed

In this case, once we receive the event, we first publish it to Kafka. Then we consume it from the topic and process it via fetch. This way, we guarantee the message is processed and delivered. But this will slow down the event delivery to the External System.

function processEvent(event) {
  moveToStorage(event); // producer.send - for main processing (fetch)
}

await consumer.subscribe({ topic: "user-events" });

await consumer.run({
  eachMessage: async ({ message }) => {
    ...

    try {
      await fetch(event);
    } catch (err) {
      moveToStorage(event); // producer.send - to another topic for retries
    }
  }
});

Keep in mind that if the internal service is down and there’s no consumer available to process the queued messages, those messages will remain in the Kafka topic (queue) until the service comes back up and starts consuming them.

We will need to add cron job support here, because if we keep producing events into the same topic that also contains events that have never been fetched, we’ll slow down their process. In that case, we won’t be able to prioritise the events properly, and retry processing may negatively impact new events’ delivery time. So we should add a separate topic and a cron job to ensure proper prioritisation.

CRONJob Retries with Kafka

Ideally, we should have a couple of topics for proper retry with a cron job:

• topicName.retry
• topicName.manual-review

When the fetch fails, we produce the message to the user-events.retry topic. Then we create a job that consumes messages from this topic and tries to re-execute the fetch method. If it fails again, we produce the message back to the same topic. If it succeeds, we simply commit the offset and mark everything as done.

During each retry, we also update the message headers (metadata stored alongside the message). Similar to HTTP, where we have a body and headers, Kafka messages can include headers, too. For example, we can set an x-retry-count header (if it’s not available yet) and keep increasing it every time the job consumes the message and produces it back to the retry topic after a failure.

Finally, after X retries, we can move the message to user-events.manual-review. The name literally describes what needs to happen next. We can also store the latest error message in a header, so we can quickly understand why the message failed and why multiple retries didn’t resolve it.

// Retry Job

await consumer.subscribe({ topic: "user-events.retry" });

await consumer.run({
  eachMessage: async ({ message }) => {
    ...
    const retryCount = Number(
      message.headers?.["x-retry-count"]?.toString() || 0
    );

    try {
      await fetch(event);
      // Success
    } catch (err) {
      const nextRetry = retryCount + 1;

      if (nextRetry >= MAX_RETRIES) {
        // Move to manual review topic
        await producer.send(...);
        return;
      }

      // Re-send to retry topic with updated headers
      await producer.send(...);
    }
  },
});

Again, depending on the option we choose above, we can have 2 or 3 topics, and they can always be adjusted based on our needs, regardless of the exact setup I’m showing here.

So, for the second option (lossless-over-speed), we have a user-events topic for the initial fetch processing and fast delivery. This means a consumer should always be running for this specific topic. If the fetch fails, we send the event to the user-events.retry topic, and the rest works the same way.

await consumer.subscribe({ topic: "user-events" });

await consumer.run({
  eachMessage: async ({ message }) => {
    ...

    try {
      await fetch(event);
    } catch (err) {
      moveToStorage(event); // producer.send - user-events.retry
    }
  }
});

Just to highlight again, topics are like tables; you can name them however you want. There’s no strict rule for topic names.

At the beginning of this section, I showed two topic names (retry and manual-review). That’s the case for the speed-over-safety option.

For the lossless-over-speed, we can simply have two topics. As we already have the user-events topic, we can treat it as the same retry topic: simply republish messages there with an updated header, and we’d only need the manual-review topic.

But as I mentioned before, retries will delay the other newly delivered events’ delivery time. So it’s better to use this 3-topic approach:

Remember, it’s not strictly necessary to have three topics. For retries, two topics are enough. But if you also consume your initial event via Kafka, then this approach works better.

Processes should be idempotent

Kafka will send the message at least once, so we can be confident it won’t be lost. However, depending on configuration, network issues, and other factors, you might receive duplicate messages. That’s why it’s best to make your functions and processes idempotent: meaning that running them multiple times with the same input produces the same result.

You can also add a deduplication mechanism using Redis. In my case, I implemented this with Redis by using the message key as a unique identifier: I store the key in Redis when processing starts and mark it as processed with a TTL (for example, 1 minute).

Here, we should use an atomic operation, because the same message can be delivered to different services at the same time. I suggest looking into the SETNX command (or SET with NX).

When you receive a message, try to lock its key using SETNX. If you succeed, you can proceed and keep that lock until the TTL expires. If processing fails, delete (DEL) the key so the message can be reprocessed.

Kafka UI: Monitoring and Debugging

This article’s subtitle includes something like “…Easy-to-Monitor Approach.” That’s the part I really like, because it helped me monitor the growth and processing of topic messages in real time in my project. And in case of failures, I could quickly inspect the headers and payloads to see which events were failing, how many times they had been retried, and why, or even review the messages that eventually ended up in the manual-review topic.

Recently, I had an incident in my project, and because of it, we ended up with a lot of messages in the manual-review topic. As you understand, we don’t have anything that automatically processes those messages.

During the investigation, I found a bug inside the main processor and the retry processor, which is why the events ended up in the manual-review topic after multiple retry attempts.

Once the bug was fixed, I needed to retry the messages from the manual-review topic. What I did was simply use a shell command to move the messages from the manual-review topic back to the retry topic, and then rerun the job.

kafka-console-consumer --bootstrap-server localhost:9092 \
  --topic source-topic \
  --group move-to-destination-v1 \
  --property print.key=true \
  --property key.separator=":" | \
kafka-console-producer --bootstrap-server localhost:9092 \
  --topic destination-topic \
  --property parse.key=true \
  --property key.separator=":"

When the job was successful, I simply cleaned all the messages from the manual-review topic. This makes it much easier to debug future cases. The same idea applies to retry topic: once they are processed and the messages are committed, they can eventually be deleted.

Btw, you can configure Kafka retention, and Kafka will remove old messages after a specified amount of time (by default it’s 7 days).
You don’t necessarily need to treat Kafka like a database. Even in a database, you would probably remove those records at some point, because you no longer need them. Kafka acts as a storage layer and comes with many useful features. It’s a distributed event-streaming system that can be used for internal service communication, and for similar cases, depending on your needs.

Visualisation and Alerts

It would also be good to set up a tool like Grafana to visually monitor pending messages in Kafka topics that are waiting to be processed. You can also configure thresholds, so you get a notification in Slack, for example, when the manual-review topic is growing (or any other topic), indicating that messages aren’t being processed.

Summary

In this article, I showed how to implement a proper retry mechanism to minimise data loss and provide eventual consistency. I also explained why an SQL database or Redis is not ideal in this case, and how Kafka shines as a solution. Finally, I showed how you can use Kafka UI to visualise existing messages and processes.

This Kafka cron job solution will fit most systems where you need a retry mechanism. Remember that each problem is unique and requires specific research and focus. You might find a better approach for your particular case, which is totally normal.

Thank you for taking the time to read this comprehensive article. I hope you found it informative and gained valuable insights from it.
Feel free to ask any questions or tweet me @nairihar

Also, follow my JavaScript newsletter on Telegram: @javascript

• You Don't Know Node.js EventLoop
• How to Scale Node.js Socket Server with Nginx and Redis
• Node.js Health Checks and Overload Protection

An Advanced Retry Mechanism in Node.js with Kafka was originally published in Bits and Pieces on Medium, where people are continuing the conversation by highlighting and responding to this story.

When and How a Backend Monorepo Can Be a Better Choice

In this article, I will discuss multi-repo and monorepo structures in the context of backend microservices. I’ll share my journey, why I used to hate monorepos and how I eventually came to love them, while also covering the pros and cons of each architecture. The article mainly focuses on the backend, as I see the benefits of monorepos primarily on one side, not both (i.e., not frontend and backend together). That said, I will also briefly touch on the idea of having both frontend and backend in the same monorepo.

There are many ways to build and manage the monorepo architecture, but believe me, it’s a challenging task.

Here is the list of topics covered in this article.

— Briefly About Multi-repos & Monorepos
 — Sharing Common Logic Across Multi-repos
 — Why I Didn’t Like the Idea of a Monorepo
 — You Don’t Need a Library for Everything
 — — Database migrations
 — — Node.js Native Test Runner
 — Decision-Making Between Engineers
 — Backend Monorepo with NPM Workspaces
 — TypeScript & ESLint Configuration
 — — Common Paths in Monorepo using TSConfig
— There is Always a Need to Adjust the Monorepo
 — CI/CD: Challenges with Setting up Docker files
 — Summary

Briefly About Multi-repos & Monorepos

The simplest way to build an application with frontend and backend communication is to have two separate repositories (or folders): one for the backend code and one for the frontend application. Later, if you decide to introduce a mobile app, you can simply create a third folder to hold everything related to the mobile app.

But things can get more complex. What happens when you need to support both iOS and Android? Or when your backend grows and you realise that a single service isn’t enough, and now you need multiple services for scalability.

What do you do then? Is it okay to have 10 different projects or repositories just to support the entire application end-to-end?

Well, there’s no simple answer to that question.
If you had asked me this question a year ago, I would have definitely said “yes.” Keeping projects isolated and as small and simple as possible seemed like the best approach. That way, everything stays clean: we can have one backend service just for reporting, another for billing, and a third for the admin panel. Simple, right?

Even though I was happy with that setup, I often found myself wanting to share logic between services. I didn’t want to duplicate the same code across multiple projects. When those thoughts start creeping in, it’s usually a sign that your experience is evolving.

There are many ways to solve this issue, which we’ll explore soon.

Having 10 separate repositories for 10 different services is a multi-repo structure. On the other hand, the whole point of a monorepo is to collect multiple services into a single repository. But that doesn’t necessarily mean putting everything, frontend, backend, mobile, into one giant repo. I used to think that too, and honestly, that mindset was one of the main reasons I avoided monorepos for a long time.

In reality, you can have multiple monorepos: one for backend projects, one for frontend, maybe even separate ones for web and mobile, depending on your needs.

Sharing Common Logic Across Multi-repos

Over the years, while working on different projects, I often needed to share code or logic across multiple repositories.

On the frontend, I wanted to reuse things like form validations, HTTP request/response logic, and Redux-related functionality between React and React Native projects.
On the backend, I wanted to share database models and cryptographic logic used across 6 or 7 different services.

I didn’t want to duplicate the same 30+ files across every repository. That just didn’t feel like a respectable or maintainable choice.

So, that marked the beginning of the private packages era for me.

I started building private packages for internal services, things like:

• @company/db
• @company/integrations
• @company/crypto
• @company/validations, etc.

Each of these modules had its internal structure, tests, documentation, and a versioning file. Every time we introduced a change, we’d update the version manually.

For example:

# Version history of the @company/database library

## 2.3.0
- Update `users` model, add `name` field

## 2.2.5
- Fix vulnerabilities; update crypto lib

This approach worked well, and I used it in a large multi-repo setup for many years. I was happy with it and had no desire to change anything.

The only downside was that when we updated a common library, we had to manually bump the dependency version in every service that used it. But honestly, that wasn’t a big deal at the time.

Why I Didn’t Like the Idea of a Monorepo

To be honest, I think I didn’t have enough experience to set up a monorepo properly at the time. I was also worried that putting both frontend and backend code in the same place would make decision-making between the frontend and backend teams much harder. Things like formatting, coding styles, etc. It felt like a constant debate. I preferred having my own space where I could make decisions independently instead of constantly negotiating everything with frontend engineers.

This second point still holds for me today, and I’ll dive deeper into it later.

But regarding the first point: as I mentioned before, setting up a monorepo can be tough. Managing deployments, testing, and shared code logic isn’t always straightforward. There are many tools that help with this, but you don’t always need to rely on existing libraries. In fact, I believe you should first understand the fundamentals and ideally build things yourself. That way, you maintain full flexibility. Otherwise, your plans may get blocked by limitations in a tool you adopted too early.

It’s similar to using Expo in React-Native; it helps you get started fast, but eventually, you’re stuck in a bubble where you can’t do everything you want. Or take ORM libraries as another example. Many developers write migrations in JavaScript, mixing JS with SQL. I don’t agree with that. If you want to create a table, just run CREATE TABLE using SQL. You don’t need a library for everything. Sometimes the best way is the direct way, with the right tool for the job.

That’s why I always avoided tools like:

• Lerna
• Nx

… and similar tooling for managing monorepo structure.

In addition to my lack of knowledge on setting up monorepos properly, I also noticed that testing and deployment could sometimes get harder and more time-consuming. In a monorepo setup, you might need to build or test everything together. That has its pros and cons. For example, maybe you just updated a shared library to introduce a feature in Service A. It works perfectly there, but it unexpectedly breaks Service B and C. In a monorepo, you’ll find that out much faster than in a multi-repo setup.

I’ll tell you soon about how I managed a monorepo starting from configurations, testing, and deployment, don’t worry, we won’t miss the details.

You Don’t Need a Library for Everything

There are so many people who always try to solve problems using libraries, without even checking the source code.
If you’re a professional engineer, you should understand every dependency you install: why it’s there, how it works, and what trade-offs it introduces.

You should always be mindful not to get trapped in a “library bubble.” At some point, you might need to remove that library or do something it doesn’t support, and you’ll be stuck.

I’ll give you an example involving database migrations and testing to show you how I prefer to approach things. That should help explain my point more clearly. After that, I’ll dive into the important part: why I used to hate monorepos, and how I eventually started to love them.

Database migrations

Having a proper way to handle migrations is straightforward, you don’t need to mix JavaScript and SQL code in the same file. That’s like a cocktail that gives you nausea after drinking it.

What’s stopping you from simply keeping your SQL code in .sql files, and having a single JavaScript/TypeScript script that reads and runs those files? It’s clean, simple, and each tool does exactly what it’s meant to do.

+-- run-migration.ts
+-- migrations/
|   +-- migration-1.1.sql
|   `-- migration-1.0.sql

Node.js Native Test Runner

I’m sure you’ve used Mocha or Jest at some point. They’re great tools that help with testing, I’ve used them too. But I don’t need them anymore, because Node.js now comes with its built-in test runner.

If the native tool does the same job, why shouldn’t I prefer it?
Fewer libraries mean fewer vulnerabilities. Simple as that.

Nowadays, Node.js includes more and more native features, and that’s a great thing. We’re finally moving in the right direction. Hopefully, at some point, we’ll have all the essential tools needed to build solid backend systems, without relying too much on external dependencies.

I think you get my point, it’s like enjoying engine-based cars over electric ones. Or preferring to drive a stick shift over an automatic. There’s just more control, and it feels right.

Because of this, I never ended up using a dedicated tool for setting up a monorepo until I discovered NPM workspaces.

But before jumping into that, I also promised to talk about decision-making. Decision making

Decision-Making Between Engineers

I have more than 10 years of experience, and I want to point out that the decision-making process within a team is critical. It can slow down your progress or even lead to the failure of an entire project.

Frontend and backend engineers live in completely different universes. Even though we may use the same language, JavaScript, and we’re all engineers, we think in very different ways.

From variable naming to code style, formatting, and dependency choices, our mindsets differ significantly. This difference always kept me from seriously considering putting frontend and backend code in the same place. So it’s often better to define clear boundaries.

And eventually, I realised: maybe I could set up a monorepo just for backend projects. And that’s where the fun and the challenge started, especially when I discovered NPM workspaces.

Backend Monorepo with NPM Workspaces

Recently, I was starting a new project and needed to set up the architecture. It was clear from the beginning that we’d have around 7–10 services, since it was intended to be a microservice-based system. I was on the verge of going with a multi-repo structure.

But then found this: NPM Workspaces

Workspaces is a generic term that refers to the set of features in the npm cli that provides support to managing multiple packages from your local files system from within a singular top-level, root package.

docs.npmjs.com

So basically, you can have different packages in the same folder and manage them through a single root-level package.json. Shared dependencies can live at the root, while each service can have its own dedicated package with its dependencies.

I started diving deeper into this approach and eventually decided to set up the project using a monorepo structure that includes only the backend services.

.
+-- node_modules
+-- package-lock.json
+-- package.json
+-- api
|   `-- package.json
+-- core
|   `-- package.json
+-- reporting
|   `-- package.json
`-- sdk
    `-- package.json

It was straightforward in the beginning, until I had to deal with TypeScript, ESLint configurations, and CI/CD.

A monorepo is like a BMW: it requires constant maintenance and attention. You can’t just set it up once and expect it to work smoothly for the next five years.

I spent 1–2 weeks setting up the initial structure, but I still make adjustments whenever I need something specific.

TypeScript & ESLint Configuration

This was truly a challenge for me, because I’m not a fan of TypeScript, yet I had to set it up properly for the monorepo. I’ve always said: if we need typing, why not just use a statically typed language in the first place?

Well, that’s a big topic for another day. The reality was, TypeScript was a team decision, and I had to work with it.

Both TypeScript and ESLint support extending configurations, so I immediately created a generic root-level config file and extended that in each package.

.
+-- node_modules
+-- package-lock.json
+-- package.json
+-- tsconfig.json
+-- .eslintrc.js
+-- api
|   +-- package.json
|   +-- tsconfig.json
|   `-- .eslintrc.js
+-- core
|   +-- package.json
|   +-- tsconfig.json
|   `-- .eslintrc.js
+-- reporting
|   +-- package.json
|   +-- tsconfig.json
|   `-- .eslintrc.js
`-- sdk
    +-- package.json
    +-- tsconfig.json
    `-- .eslintrc.js

I needed this approach because I wanted the ability to run, test, and work with individual services/packages independently. I didn’t want everything to be controlled from the root level.

These configurations should always be based on the actual needs of the team and the project. There’s no single package that will perfectly fit and satisfy all of those needs. That’s why I believe in solving each problem individually, rather than just installing a library that tries to do everything.

It helps you to define your boundaries.

I won’t show specific code examples because the configuration might look different tomorrow: packages are constantly being updated.

However, you can check out the extend functionality of TypeScript and ESLint by following these links:

• Extend ESLint
• TSConfig Option: extends

And in a similar approach, you can configure other tools.

But what I do want to show you is how to handle shared logic: where to store it and how to set it up properly.

Common Paths in Monorepo using TSConfig

In the TypeScript config, you can define paths like this:

{
  "compilerOptions": {
    "paths": {
      "@common/*": ["./_common/*"],
      "@db/*": ["./_db/*"]
    }
  }
}

You can place reusable code in the root of the monorepo and treat it as an external dependency within each service.

.
+-- node_modules
+-- package-lock.json
+-- package.json
+-- tsconfig.json
+-- .eslintrc.js
+-- _db/
|   `-- user.ts
+-- _common/
+-- configs/
|   `-- env.ts
+-- api
|   +-- package.json
|   +-- tsconfig.json
|   `-- .eslintrc.js
+-- core
|   +-- package.json
|   +-- tsconfig.json
|   `-- .eslintrc.js
+-- reporting
|   +-- package.json
|   +-- tsconfig.json
|   `-- .eslintrc.js
`-- sdk
    +-- package.json
    +-- tsconfig.json
    `-- .eslintrc.js

As an example, inside both the api and core packages, I can import my User entity like this:

import { User } from '@db';
import { env } from '@common/env';

The reason I use the @ symbol in the config is for better observability: it helps me immediately recognise that it’s a custom or shared path. I apply the same logic with underscores (_) in folder names. When you prefix a folder with an underscore, it appears at the top in most code editors, which improves visibility.

Of course, these are just conventions for clarity: you’re free to name them however you prefer.

There is Always a Need to Adjust the Monorepo

I want to point out a very important example that shows why a monorepo always requires maintenance.

At some point, I introduced an SDK. It wasn’t part of the microservice architecture: it was just a shared SDK that I planned to publish to npm. I decided to keep it inside the monorepo as I reused some things from the common folder.

Because of that decision, I had to separate the ESLint and TypeScript configs for the SDK instead of extending from the root. Sharing the root configuration caused issues with building, packaging, and publishing.

You have to keep in mind that when publishing an SDK, you don’t want to expose your entire backend architecture. We will discuss the solution to this particular issue in the next section.

So the key takeaway is: over time, as you make different decisions, your monorepo setup will evolve. There’s no one-size-fits-all solution. You should evaluate your specific needs and choose the setup that works best for you.

CI/CD: Challenges with Setting up Docker files

When you’re working with a monorepo and have reusable code, and you want to keep everything as close to professional as possible so that you’re proud of your solution, you need to put in real effort.

For example, if the SDK were in a separate repo, I could simply build and publish it without much hassle. But I chose to keep it inside the monorepo because it relied on reusable functionality. On the other hand, I don’t want to publish the entire common folder with the SDK: only the files actually used. So as you can see, there are pros and cons to every approach.

The same challenge applies to building and deploying services. We have to carefully control which files get exposed during service deployment or package publishing. And that makes our Dockerfile setup a bit more complicated.

First, you need a build environment where you isolate the specific service or module (like the SDK) you want to build. Then you need to include only the necessary files from the reusable/shared folders. Only by doing this can you avoid exposing your entire architecture or internal codebase.

It might all sound easy in theory, but believe me, it’s challenging in practice.

Example of what they Dockerfile could look like:

FROM node:20 AS builder

WORKDIR /app

COPY package.json package-lock.json tsconfig.json .eslintrc.js ./

COPY sdk ./sdk
COPY _db ./_db
COPY configs ./configs
COPY _common/number.ts ./_common/number.ts

COPY sdk/package.json sdk/tsconfig.json sdk/.eslintrc.js ./sdk/

RUN npm install

RUN cd sdk && npm run build

FROM node:20-alpine

WORKDIR /app

COPY --from=builder /app/sdk/dist ./dist
COPY sdk/package.json ./

RUN npm install --only=production

RUN npm publish

And as you might already have guessed, you may need to adjust your monorepo as it grows: copy-pasting more files becomes harder, and this approach might no longer be ideal

Summary

Having a monorepo structure is much more challenging than it might seem. It requires a lot of experience if you want to do everything properly. If you don’t have much experience and you’re just starting a project, it might be better to work with multiple repositories, at least until you have a mentor or gain enough knowledge to configure and manage all the edge cases on your own.

I would personally recommend keeping frontend and backend in separate repositories unless there’s a very specific reason to combine them.

In any case, avoid introducing tons of libraries for every issue. Sometimes, it’s better to do some research and spend time understanding how certain things should be done properly without using any libraries. Also, you will have fewer vulnerabilities.

Both monorepos and multi-repos have their pros and cons, and we can’t simply say one is always better than the other. I just shared my experience, why I used to hate monorepos and how I slowly began to understand their value and became comfortable using them. But that doesn’t necessarily mean I’ll stop using multi-repos for future projects.

It depends on factors like:

• How many engineers are working on the project
• How much code needs to be reused
• The overall experience level of the team

If there’s not much to share across services, I don’t think a monorepo adds much value; in that case, a multi-repo setup will likely be a safer, faster and simpler choice.

Even though I haven’t gone into very much detail about how to do certain things, I shared my experience with multi-repos and monorepos, and showcased some of the problems and solutions, including resourceful links.

Thank you for taking the time to read this comprehensive article. I hope you found it informative and gained valuable insights from it.
Feel free to ask any questions or tweet me @nairihar

Also, follow my JavaScript newsletter on Telegram: @javascript

• You Don't Know Node.js EventLoop
• How to Scale Node.js Socket Server with Nginx and Redis

Monorepo: From Hate to Love was originally published in Bits and Pieces on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, we will review the notable changes introduced in the new Node.js version:

• add node:sqlite module (Colin Ihrig) #53752
• add matchesGlob method (Aviv Keller) #52881
• add postMessageToThread (Paolo Insogna) #53682

Let’s get started!

node:sqlite

The node:sqlite module facilitates working with SQLite databases.

// index.js

const { DatabaseSync } = require('node:sqlite');
const database = new DatabaseSync(':memory:');

// Execute SQL statements from strings.
database.exec(`
  CREATE TABLE data(
    id INTEGER PRIMARY KEY,
    name TEXT
  );
`);

// Create a prepared statement to insert data into the database.
const insert = database.prepare('INSERT INTO data (id, name) VALUES (?, ?)');

// Execute the prepared statement with bound values.
insert.run(1, 'Bob');
insert.run(2, 'John');

// Create a prepared statement to read data from the database.
const query = database.prepare('SELECT * FROM data ORDER BY id');

// Execute the prepared statement and log the result set.
console.log(query.all());

// [ { id: 1, name: 'Bob' }, { id: 2, name: 'John' } ]

The following example shows the basic usage of the node:sqlite module to open an in-memory database, write data to the database, and then read the data back.

This built-in lib becomes available when using the --experimental-sqlite flag.

node --experimental-sqlite index.js

We developers, along with Node.js native modules, use many external modules to make the server fully functional. It’s excellent that Node.js tries to include those important tools on its own. For example, they recently added a native support of test runner, so we switched from jest and mocha to the native library. Now, it's time for the databases and ORMs.

This is currently an experimental module, and it will take some time for it to become stable and have more methods.

Let’s appreciate the Node.js Core team’s effort. One day, we will only use a small number of external libraries as most key modules will be available natively.

matchesGlob

This is another experimental method which determines if path matches the pattern.

path.matchesGlob('/foo/bar', '/foo/*'); // true
path.matchesGlob('/foo/bar*', 'foo/bird'); // false

postMessageToThread

Last but not least, a method that sends a value to another worker, identified by its thread ID.

Previously, we could communicate with the worker threads using Message Channels. But now, thread communication is much simpler.

Here is how we can get the thread ID:

const { threadId } = require('node:worker_threads');

And this is how they can communicate:

postMessageToThread(id, { message: 'pong' });

To be able to get the messages first, you need to have the listener:

process.on('workerMessage', (value, source) => {
    console.log(`${source} -> ${threadId}:`, value);
});

The source is the sender's thread ID, so you can do something like this if you want to message back.

process.on('workerMessage', (value, source) => {
  console.log(`${source} -> ${threadId}:`, value);
  postMessageToThread(source, { message: 'pong' });
});

If you want to communicate with the main thread, you can simply use 0, as the main thread’s ID is 0.

postMessageToThread(0, { message: 'ping' });

Full example here.

I hope you enjoyed this summary. I aimed to deliver the information promptly and highlight the most important updates.

Also, follow my JavaScript newsletter on Telegram: @javascript

• You Don’t Know Node.js EventLoop
• How to Scale Node.js Socket Server with Nginx and Redis

What is new in Node.js V22.5 was originally published in Bits and Pieces on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are numerous programming design patterns that you’ve likely used, but you’re not aware of them.

In 2019, I presented a talk at JS Conf Armenia on this topic. The presentation is in Armenian, and you are welcome to watch it if you’re interested. Today, however, I have chosen to conduct the same research in written form

I will cover some common and useful design patterns, providing JavaScript examples.

In programming, design patterns are invaluable as they provide significant support. By following certain principles, we ultimately achieve well-structured, easily understandable, reusable, and manageable code. When starting the journey of learning design patterns, it may pose challenges, and certain concepts may appear unclear initially, especially when you can express the same idea more briefly. But remember, high-quality code doesn’t always mean it’s the shortest.

As I mentioned earlier, most developers use different kinds of design patterns daily, but they are not aware of them. There are so many design patterns, and each has its principles, scopes, and names.

Design patterns are not programming language-specific. They are more likely programming paradigm-specific. Functional programming, Object-oriented programming, and all those different types of programming paradigms have their design patterns.

Design Pattern types

Think of design patterns as helpful guides in the world of Objects and Classes. When we talk about classes, we are talking also about function constructors. These patterns help us to maintain and organize high-quality code.

According to this map, there are three main types of design patterns.

• Creational Design Patterns focus on the process of object/class creation.
• Structural Design Patterns are concerned with the composition of classes or objects.
• Behavioral Design Patterns focus on the interaction and communication between objects and classes.

We will not delve deeper into each of those design patterns from the image. I just want to introduce some design patterns and their examples. Afterwards, you can dedicate time to books on similar topics. This is something that should be taken seriously, and you can’t learn everything in just a few hours. It requires constant practice.

1. Singleton

This pattern ensures that a class or object has only one instance and provides a global point of access to that instance.

Database connection example

In this example, the MongoClient is typically used to connect to a MongoDB database. The Singleton pattern here would ensure that there is only one instance of the MongoClient throughout your application, preventing multiple unnecessary connections to the database.

const { MongoClient } = require('mongodb');

const url = 'mongodb://localhost:27017';

// here you may have other options as well ....

export default MongoClient.connect(url);

When we import and use the exported object in different parts of our application, it will always refer to the same instance of the MongoClient. This helps in managing and efficiently reusing the database connection.

2. Factory

In simple terms, this pattern encapsulates the instantiation logic, providing a way to create objects without specifying their exact class.

class User {
  constructor(name) {
    this.name = name;
  }
}

function createUser(name) {
  return new User(name);
}


const myUser = createUser('Sara');

Image example

function createImage(name) {
  if (name.match(/\.jpeg$/)) {
    return new JpegImage(name);
  } if (name.match(/\.gif$/)) {
    return new GifImage(name);
  } if (name.match(/\.png$/)) {
    return new PngImage(name);
  }
  throw new Exception('Unsupported format!');
}

In this example:

• createImage is the factory method responsible for creating instances of different image types.
• JpegImage, GifImage, PngImage are the product classes, each representing a specific type of image.

Here are the benefits of using the Factory Pattern in this example:

• Encapsulation: The creation logic is encapsulated in the createImage function, isolating it from the rest of the code.
• Flexibility: If you want to add a new image format or modify the creation process, you can do it within the factory function without affecting the client code.
• Readability: The client code that uses createImage is abstracted from the details of how each image type is created, making the code more readable and maintainable.

Overall, this example nicely demonstrates the Factory Pattern by providing a centralized way to create instances of different image types based on a common interface.

3. Strategy

This is a behavioural design pattern that defines a family of algorithms, encapsulates each one, and makes them interchangeable. It allows the client to choose an algorithm from a family of algorithms at runtime.

const ups = new UPS();
const usps = new USPS();
const fedex = new Fedex();

const shipping = new Shipping();

shipping.setStrategy(fedex);

shipping.calculate();

// YOU CAN ALSO DO THIS
// shipping.setStrategy(ups);
// shipping.setStrategy(usps);

In this example:

• UPS, USPS, and Fedex are different strategies or algorithms for shipping.
• Shipping is the context class that uses a strategy.

The key benefit of this Pattern is that it allows the client (Shipping class in this case) to switch between different algorithms (strategies) at runtime without altering its structure. It promotes code reusability, flexibility, and easy maintenance.

4. Adapter

It allows us to access the functionality of a class using a different interface.

// Existing class with a different interface
class OldSystem {
  requestInfo() {
    return "Information from the old system.";
  }
}

// Adapter to make the OldSystem compatible with the new interface
class Adapter {
  constructor(oldSystem) {
    this.oldSystem = oldSystem;
  }

  fetchDetails() {
    const oldInfo = this.oldSystem.requestInfo();
    return `Adapted: ${oldInfo}`;
  }
}

// Using the Adapter we make OldSystem compatible with another interface
const oldSystem = new OldSystem();
const adaptedSystem = new Adapter(oldSystem);

So, the overall purpose is to make an existing class (OldSystem) work with a new interface by creating an adapter (Adapter) that translates the methods from the old class to match the expected interface. The client code can then interact with the adapted system (adaptedSystem) without worrying about the differences in the original and desired interfaces.

5. Command

Here we have a simple calculator with undo functionality using the Command Pattern. It defines basic math operations, creates command objects for each operation, and utilizes a calculator object to execute, undo, and log these operations. The example performs calculations, undoes two operations, and displays the final result. The Command Pattern enables flexible and extensible command execution in a clean and organized manner.

const calculator = new Calculator();

// Perform calculations via commands
calculator.execute(new AddCommand(100));
calculator.execute(new SubCommand(24));
calculator.execute(new MulCommand(6));
calculator.execute(new DivCommand(2));

// Reverse last two commands (undo calculations)
calculator.undo();
calculator.undo();

console.log(`Value: ${calculator.getCurrentValue()}`);

It’s obvious, that this isn’t the complete example. You can find it here.

Why bother writing a basic calculator in 100 lines when it could be done in less? The thing is, the impact may not be obvious immediately.
In big and complicated projects, being clear is super important. Opting for consistent design patterns ensures that you and your team uniformly approach the code. This not only makes the project more understandable but also facilitates smoother collaboration, even when dealing with sizable projects/codebases.

Imagine you’re in an interview, tasked with writing a calculator. How do you think the interviewer would perceive a basic calculator compared to one crafted using this design pattern? It becomes a showcase of your understanding and knowledge, leaving a lasting impression.

6. Module

With the help of this design pattern, we can prevent pollution of the global namespace by keeping variables and functions within a private scope.

Numerous implementations exist, but in this article, I’ll present two straightforward examples that are exceptionally simple and clear within the context of Node.js.

Implementation 1

// utils.js

const privateList = [];
const privateObject = {};

function find() {
  // ...
}

function each() {
  // ...
}

module.exports = {
  find, each
};

Implementation 2

// utils.js

const privateList = [];
const privateObject = {};

exports.find = function () {
  // ...
};

exports.each = function () {
  // ...
};

In both examples, the functions and variables inside the module are encapsulated (hidden), providing a clean and organized way to structure code. Users of the module can only access what is explicitly exported, helping to maintain a clear separation of concerns and avoiding naming conflicts with other parts of the codebase.

7. Pub/Sub

This behavioral design pattern facilitates communication between different parts of a software system without them having to directly reference each other. It involves a mechanism where entities can subscribe to receive messages (publishers) and other entities can publish messages to the subscribers.

const subscriber1 = new Subscriber();
const subscriber2 = new Subscriber();

const publisher = new Publisher();

subscriber1.sub('t.me/javascript', (msg) => {
  console.log(msg);
});

subscriber2.sub('t.me/javascript', (msg) => {
  console.log(msg);
});

publisher.pub('t.me/javascript', 'Quiz #123');

Subscriber is a class that represents an entity that wants to receive messages. Each instance of Subscriber can subscribe to specific channels. And the Publisher is a class that represents an entity that sends out messages to specific channels.

Every time specified callback function will be executed when a message is published to that channel.

In summary, the Pub/Sub pattern provides a flexible way for different components or modules in a system to communicate without being directly aware of each other. Publishers send messages to specific channels, and subscribers express interest in receiving messages from particular channels, creating a decoupled and scalable communication system.

8. Observer

In the world of design patterns, numerous patterns may seem similar at first glance, but in reality, they have subtle differences. Similarly, this design pattern shares similarities with the Pub/Sub pattern.

const observer = new Observable();

// Subscriber 1
observer.subscribe('channelName', (msg) => console.log(msg)); // Hello
// Subscriber 2
observer.subscribe('channelName2', (msg) => console.log(msg)); // Hello

observer.notify('channel', 'Hello');

You can find the full example here.

Number of Subjects

In the Observer pattern, there’s typically one subject (or observable) that maintains a list of dependents (observers) and notifies them of changes in its state. But in Pub/Sub, there can be multiple publishers and multiple subscribers, and they communicate through channels. Publishers send messages to specific channels, and subscribers express interest in receiving messages from particular channels.

Flexibility and Scalability

Pub/Sub is more flexible and scalable, especially in distributed systems. Publishers and subscribers can be added or removed without directly impacting each other, making it easier to extend the system. However the Observer pattern might require more careful management of dependencies, and adding or removing observers may involve modifications to the subject.

The granularity of Notification

In the Observer pattern, when the subject notifies observers, it sends the same message to all observers. In Pub/Sub, when a message is published to a channel, only the subscribers of that channel receive the message.

Coupling

The Observer pattern often involves a more direct relationship between the subject and its observers. Observers are aware of the subject and its changes.
Pub/Sub promotes a more decoupled approach. Publishers and subscribers are not explicitly aware of each other; they communicate through a central hub (the message broker) without direct references.

While the design patterns may appear similar, there are crucial differences between them.

9. Constructor

This is a way to define and initialize object instances with their properties and methods.

class Movie {
  constructor(name, year) {
    this.name = name;
    this.year = year;

    this.about = () => {
      return `${name} movie has been shot in ${year}`;
    };
  }
}

const hp = new Movie('Harry Potter', 2001);
const insatiable = new Movie('John Wick', 2014);

The Constructor Pattern provides a convenient way to create multiple instances of objects with shared properties and methods.

The Constructor Pattern is often associated with classes in JavaScript, but it’s not limited to classes. In JavaScript, a constructor is essentially a function that creates instances of objects.

function Movie(name, year) {
  this.name = name;
  this.year = year;

  this.about = () => {
    return `${name} movie has been shot in ${year}`;
  };
}

const johnWick = new Movie('John Wick', 2014);

It’s more about the concept of using a function (whether a class constructor or a regular function) to create and initialize objects. It’s a way to structure and create instances of objects with shared properties and methods.

function createMovie(name, year) {
  const about = () => {
    return `${name} movie has been shot in ${year}`;
  };

  return {
    name, year, about
  };
}

Summary

Remember, every design pattern can have multiple implementations, and implementations can differ in every language. However, the idea is the same.

Every design pattern comes with its pros and cons.

The cool thing is, that these patterns work in all sorts of programming languages. So, if you learn them in Java, you can apply the same tricks in JavaScript.

It’s like having both advantages and disadvantages, but in a language that all programmers can understand!

Consider using up-to-date books to understand those patterns in your favorite programming language. For JavaScript, here are a few examples of helpful books.

• Node.js Design Patterns by Mario Casciaro
• Learning JavaScript Design Patterns by Addy Osmani

Thank you for taking the time to read this article. I hope you found it valuable. Feel free to ask any questions or tweet me @nairihar

Also follow my JavaScript newsletter on Telegram: @javascript

• You Don’t Know Node.js EventLoop
• How to Scale Node.js Socket Server with Nginx and Redis

Essential JavaScript Design Patterns was originally published in JavaScript in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you have over 5 years of programming experience, it’s quite likely that you’ve experienced burnout at some point.

You’re facing a 10-day deadline, and you end up procrastinating for the first 8 days because you just didn’t have the mood to work. However, you managed to complete the task within the last two days out of necessity.
It’s a familiar scenario — you open your laptop, but the motivation to code consistently for the 5th or 6th year isn’t there.

I began my journey in the programming world almost 10 years ago. I have been working as a software engineer for 8 consecutive years. The last few years lacked proper vacations, and at times, I experienced burnout. Nevertheless, I still find joy in my work every day, and in this article, I will share my secret…

The Journey into the World of Programming

When I was learning programming, at some point I understood that this is the thing that I want to do every day. I fell in love with programming, and it became one of the prioritized hobbies in my life.

I got a bit crazy, didn’t sleep well, and missed my breakfasts and lunches. I was like a zombie, undisturbedly consuming new information from the internet. And it boosted my career.

You may say — we all understand that if you’re doing something too much, then you will lose your interest too early.
- Fortunately, my case is quite different.

Hobbies

I’m not quite sure why, but it feels like I have too many hobbies (magic tricks, cycling, running, podcasting, writing etc). You might think that I simply adopt a hobby for a couple of weeks and then promptly forget about it. However, that’s not the case; I genuinely strive to deeply immerse myself in any aspect that captivates me.
I spent 4–5 years learning magic tricks, 3–4 years cycling and running long distances, and so on … Sometimes I even sensed that my focus leaned more towards one of those aspects rather than on programming itself.

While having those types of hobbies, I never felt that I was losing my interest in programming. Programming still is a hobby and that pursuit helps me make money and concentrate on other hobbies.

We, engineers, spend lots of time doing the same thing every day. Even though it can be a hobby for us, we may lose interest if we do it too much.

Somehow, we need to maintain a balance, just like on the cover of this article. On the left side, there’s a large stone symbolizing programming, which often consumes most of our time. Finding a similar stone for the right side to achieve balance is challenging, as we have limited time in our day. So that’s why on the right side of the cover we have multiple stones on top of each other. Each of those can carry a different weight, and each can bring various emotions and joy to our day.

It’s quite similar to my situation; every day, I spend hours coding, but I make an effort to maintain my interest and prevent burnout through other hobbies.

If you like dancing, why you don’t practice every day?

In addition to my hobbies mentioned above, I also enjoy Latin dancing, and I practiced for 1–2 years. I prioritized it in my daily routine because I enjoyed dancing.

One day, I asked the question above to one of my friends, and he replied:

Even though I have plenty of time and enjoy dancing, I don’t want to lose my dancing interest. If I practice every day, it’s likely that I will lose the interest in dancing in a short period.

The thing is, that unintentionally, I had too many hobbies at the same time, and they all helped and continue to help me maintain my interest in programming.

Sometimes, I feel like I have a new hobby addiction. However, I understand that if I try to balance my hobbies, as my friend mentioned, I can’t get the maximum from each one, and it will definitely affect my interest in programming. Because I have some expectations from me, and I need to satisfy them mentally and not only. Just programming can’t always satisfy me. I strive to extract the most from each hobby, and whenever I feel it’s losing its appeal, I seek out another one or an old one.

I wouldn’t say that I disagree with my friend. He is right. However, each of us has our priorities and interests in life. Somehow, I ended up in this situation that magically works, and I am not displeased with it.

No burnout? You don’t lie?

I would lie If I said I never had a burnout situation. Even though I shared my story, I want to mention that I still have burnout sometimes. But the level of that burnout isn’t significant. It has a really weak effect.

We need to understand that it’s not possible to prevent burnout completely. We just need to minimize it. If it weren’t minimal in my case, I wouldn’t continue to enjoy coding, and I wouldn’t love to continue programming like some of my friends who decided to switch from programming to business or project management.

Programming as a hobby

If this is not the case, then it’s probably harder for you. Because you’re not enjoying your job. You shouldn’t restrict yourself to programming just because it’s popular or lucrative. No, you need to start loving it; you need to take it as a hobby. Otherwise, you might find yourself in a huge struggle, at least in the long run for sure.

Summary

If you are a beginner, if you have just started your journey in programming, try to gather as much information as you can. The longer you do the same thing, the more likely you are to get tired of it. In the beginning, your productivity is high, but over time, it tends to decrease. Try to understand that and pace yourself. As someone who has been in software engineering for more than 5 years, also concentrate on other things. Make sure that you are satisfying your needs.

Thank you for taking the time to read this article. I hope you found it valuable. Feel free to ask any questions or tweet me @nairihar

Also follow my JavaScript newsletter on Telegram: @javascript

• You Don't Know Node.js EventLoop
• How to Scale Node.js Socket Server with Nginx and Redis

Load environment variables without an additional library

Many programming languages have a handy tool called dotenvfor loading environment variables from a .env file.

The Node.js dotenv library is no different, and it's one of our go-to choices for managing environment variables. It's a simple and important addition to any project.

With the exciting release of Node.js V20.6.0, there’s now native support for .env files. This means you can effortlessly load environment variables from a .env file into your Node.js application’s process.env, all without the need for external dependencies.

node --env-file .env

Differences

1. When Node.js starts up, it automatically loads and parses the .env file. This allows you to include environment variables that configure Node.js itself, such as NODE_PRESERVE_SYMLINKS.
2. While this may not align with the behaviour of current dotenv packages, it’s crucial to recognize that Node.js handles it differently. In Node.js, variables defined in the .env file will not override existing environment variables.

There is currently an ongoing discussion regarding this matter, and it’s highly likely that it will be changed.

3. While many dotenv packages offer additional features, such as supporting multiple environment variables, Node.js takes a simpler approach. It allows us to specify only one environment variable. Attempting to specify multiple variables will result in Node.js reading only the last specified environment variable.”

node --env-file .env --env-file .env.staging

Adding the dotenv library might seem like a small step, but it’s a significant one. Dotenv library increased our package list by one. Even though that simple configuration can be one line, it’s not good to mix technical things with business logic.

Node.js is always evolving, and you can stay up-to-date with all the latest changes by referring to the documentation.

Thank you for taking the time to read this comprehensive article. I hope you found it informative and gained valuable insights from it.
Feel free to ask any questions or tweet me @nairihar

Also follow my JavaScriptnewsletter on Telegram: @javascript

Additional reading:

• Executing Node.js Callbacks in Separate Threads
• You Don't Know Node.js EventLoop

Thank you for reading until the end. Please consider following the writer and this publication. Visit Stackademic to find out more about how we are democratizing free programming education around the world.

Native support of .env in Node.js was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.

It has already been five years since Node.js introduced the worker_threads native library, designed for creating workers, essentially functioning as threads.

When it was released, I was incredibly enthusiastic and eager to build something interesting in the scope of that.

When discussing worker threads, I noticed that it’s not straightforward for everyone to build a proper system that would use worker_threads. Consequently, I embarked on a quest to find abstractions that could simplify and enhance the potential for more intriguing solutions.

I came across several intriguing abstractions, but ultimately, I chose to develop my own — one that is lightweight and straightforward.

funthreads

NPM | Github

The idea is simple: you can execute your function in a dedicated thread by utilizing Promises.

import executeInThread from 'funthreads';

async function calculate() {
    const values = await Promise.all([
        executeInThread(() => 2 ** 10),
        
        executeInThread(() => 3 ** 10)
    ]);
    
    console.log(values);
}

calculate();

You can relocate CPU-intensive operations to separate threads and easily retrieve the results using Promises.

Just try yourself and see.

$ npm i funthreads

https://github.com/nairihar/funthreads

I didn’t spend much time on this library, and though I haven’t used it in a real project, I stayed excited and made small improvements over time.

After publishing this library, I noticed other extended implementations of the same idea.

node-worker-threads-pool

As the authors describe: Simple worker threads pool using Node’s worker_threads module. Compatible with ES6+ Promise, Async/Await and TypeScript🚀.

But this is not all, there is another library that had been released even before worker_threads.

Previously, they used various strategies and approaches to support multithreading. However, after worker_threads, their implementations significantly improved.

Here’s a concise example that illustrates how the library operates.

// main.js
import { spawn, Thread, Worker } from "threads"

const auth = await spawn(new Worker("./workers/auth"))
const hashed = await auth.hashPassword("Super secret password", "1234")

console.log("Hashed password:", hashed)

await Thread.terminate(auth)

// worker.js
import sha256 from "js-sha256"
import { expose } from "threads/worker"

expose({
  hashPassword(password, salt) {
    return sha256(password + salt)
  }
})

It seems that this is the biggest library that provides a bunch of tools to work with multithreading in Node.js.

P.S. Both of the libraries offer a method to establish a thread pool and employ it for your specific use cases.

Unfortunately, I haven’t used any of them in production, as there wasn’t a need. However, I’m confident that many projects can benefit from them in real-world scenarios.

It’s interesting to observe how they work and understand the mechanics of the abstraction.

funthreads library is quite straightforward, consisting of just a few files with small functions. You can begin with it, and I’m confident that you’ll grasp the concept easily, realizing how simple the provided abstraction is.

Wrapping up

Compared with other programming languages in JavaScript everything is quite different. Everything operates in its own unique way and has its proper explanation.

JavaScript has classes, and Node.js supports multithreading. While it may seem overwhelming compared to other languages, it’s important to embrace JavaScript on its own uniqueness.

Thank you for taking the time to read this comprehensive article.
Feel free to ask any questions or tweet me @nairihar

Also follow my JavaScript newsletter on Telegram: @javascript

I hope you found it informative and gained valuable insights from it.

Read more

You Don’t Know Node.js EventLoop

Executing Node.js Callbacks in Separate Threads was originally published in Bits and Pieces on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Ultimate Guide to Understanding EventLoop in Node.js

You might be wondering if the title of this article is a bit ambitious, but fear not! I’ve put in the time and effort to cover everything you need to know about the EventLoop in Node.js. And believe me, you’ll discover so many new things along the way!

Before beginning this article, I’d like to inform you that it’s important to have a basic understanding of Node.js and its EventLoop to fully comprehend the core concepts discussed.

This comprehensive guide is going to take some time to cover every detail that you need to know, so grab a cup of coffee and settle in for an exciting journey to the fascinating world of Node.js. Let’s get started!

Here is the list of topics covered in this article.

— What is Node.js
 — Reactor Pattern
 — Node.js Architecture
 — Event Queues (I/O Polling, Macrotasks, Microtasks)
 — Changes from Node v11
— Unblocking EventLoop using unref() (added in Jul of 2024)
 — CommonJS vs ES modules
— Libuv (thread pool)
 — DNS is problematic in Node.js
 — Custom Promises
— Bluebird
 — Summary
— A challenging interview question (added in Feb of 2024)
 — Translated versions of this article

What is Node.js

If you take a look at the official documentation, you’ll find a brief explanation like this.

Node.js is a JavaScript runtime built on Chrome’s V8 JavaScript engine. It uses an event-driven, non-blocking I/O model that makes it lightweight and efficient.

Well, that brief explanation doesn’t tell us much, does it?

There are so many important details and concepts related to the EventLoop in Node.js that require a more in-depth explanation.

Let’s explore them together!

Reactor Pattern

event-driven model

Node.js is written using the Reactor pattern, which provides what is commonly referred to as an event-driven model.

So, how does the Reactor pattern work in event-driven programming?

Let’s say we have an I/O request — in this example, it’s a file system action.

fs.readFile('./file.txt', callback);

When we make a function call that involves an I/O operation, the request is directed to the EventLoop, which then passes it on to the Event Demultiplexer.

I/O stands for Input/Output and refers to the communication with the computer’s central processing unit (CPU) .

After receiving the request from the EventLoop, the Event Demultiplexer decides what type of hardware I/O operation needs to be performed, based on the I/O request. In the case of a file system read, the operation is delegated to the appropriate unit, which reads the file.

A specific C/C++ function will read the requested file and return the content to the Event Demultiplexer.

uv__fs_read(req)

When the requested operation is completed, the Event Demultiplexer generates a new event and adds it to the Event Queue, where it can be queued with other similar events.

Once the JavaScript Call Stack is empty, the event from the Event Queue will be processed, and our callback function will be executed.

Neither the Event Demultiplexer nor the Event Queue is a single component. This is the abstract view. For example, the implementation of Event Demultiplexer and Hardware I/O can vary depending on the operating system. Additionally, the Event Queue is not a single queue but rather consists of multiple queues.

— Where does all of this come from?

libuv

The EventLoop in Node.js is provided by the libuv library, which is written in C language specifically for Node.js. It provides the ability to work with the operating system using asynchronous I/O.

libuv is a multi-platform support library with a focus on asynchronous I/O.

— Where does it fit in the Node.js architecture?

Node.js architecture

Using JavaScript, we interact with the operating system. But if we put it bluntly, JavaScript is a high-level language that is limited to basic operations such as creating variables, loops, and functions. It can’t do much more on its own.

— How does it work with the Operation System?

Many programming languages interact directly with the operating system. Therefore, if we integrate JavaScript with those languages, we can essentially work with the operating system using JavaScript.

Here is how it is already done.

The middle layer (Node.js) takes care of our JavaScript code and interacts it with the Operation System. Now let’s discuss the components of Node.js architecture.

V8

This should be a well-known engine that parses and executes our JavaScript code.

libuv

This is the library we previously discussed, which provides the EventLoop and most of the interactions needed to work with the operating system.

Core modules

Node.js provides various native modules, such as fs, http, and crypto. Those are called native modules and include JavaScript source code.

C++ bindings

In Node.js, we have an API that allows us to write C++ code, compile it, and require it in JavaScript as a module. These are called addons. Core modules may have their addons as well.

Node.js provides a compiler that generates addons, which essentially creates a .node file that can be required.

require('./my-cpp-module.node');

The require function in Node.js prioritizes loading .js and .json files, followed by addon files with the .node type.

c-ares, zlib, etc

There are also smaller libraries written in C/C++ that provide specific operations, such as file compression, DNS operations, and more.

We are not limited to this. We will revisit libuv later in this article. For now, let’s continue forward.

Event Queues

The EventLoop is a mechanism that continuously processes and handles events in a single thread until there are no more events to handle. It is often referred to as a “semi-infinite loop” because it runs indefinitely until there are no more events to handle or an error occurs.

As previously mentioned, the EventLoop consists of multiple queues, each with its priority level. In the following sections, we will delve into more detail about these priorities.

Once the Call Stack is empty, the EventLoop goes over the Queues and waits for an event to execute. It checks for timer-related events first.

If a setTimeout or setInterval function has finished executing, the Event Demultiplexer will enqueue an event to the Timers queue.

Let’s say we have a setTimeout function which is scheduled to execute in one hour. It means that after one hour, the Event Demultiplexer will enqueue an event into this queue.

When the queue has events, the EventLoop will execute the corresponding callbacks until the queue is empty. Once the queue is empty, the EventLoop will move on to check other queues.

In the second position, we have the I/O queue which is responsible for most of the asynchronous operations such as file system operations, networking, and more.

Next, we have the Immediate queue which is responsible for setImmediate calls. It allows us to schedule operations that should run after I/O operations.

And at the end, we have a specific queue called close events for the closed event.

It is responsible for handling all connections that have a close event, such as database and TCP connections. These events are queued here for execution.

In each cycle, the EventLoop checks all of these queues to determine if any events need to be executed. The EventLoop typically takes only a few milliseconds to review all the queues and check if any events are to be executed. However, if the EventLoop is busy, it may take longer. Fortunately, there are many libraries available on npm that allow us to measure the duration of the EventLoop cycle.

You may think that we have already covered most of it, but I will say it’s just the beginning.

The events which are queued in these queues are also referred to as Macrotasks.

There are two types of tasks in the EventLoop: Macrotasks and Microtasks. We will discuss microtasks later on.

Now, let’s take a look at how the EventLoop determines when it’s time to stop the Node.js process because there are no more events to handle.

EventLoop maintains a refs counter, which starts at 0 when the process begins. Whenever there is an asynchronous operation, the counter is incremented by one. For example, if we have a setTimeout or readFile operation, each of these functions will increment the counter by one.

When an event is pushed to the queue, and the EventLoop executes the callback of that particular macrotask, it also decreases the counter by one.

After processing the closed events queue, EventLoop checks the counter. If it is zero, it means there are no ongoing operations, and the process can exit. However, if the counter is not zero, it means there are still ongoing operations, and the EventLoop will continue its cycles until all the operations are completed, and the counter becomes zero.

OK, enough theory, let’s do some practice!

const fs = require('fs');

setTimeout(() => {
  console.log('hello');
}, 50);


fs.readFile(__filename, () => {
  console.log('world');
});

Let’s take a look at a simple example with one timeout and one file system read to better understand how the EventLoop works in practice.

1. When the Node process starts, V8 begins by parsing the JavaScript code and executing the setTimeout function. This triggers a C/C++ function (C_TIMEOUT) to execute inside libuv and increase the counter (refs++).
2. When V8 comes across the readFile function, it does the same thing. Libuv initiates the file read operation (C_FS), which again increases the counter (refs++).
3. Now there is nothing left for V8 to execute, and EventLoop takes over. It starts by checking each queue one by one until the counter reaches zero.
4. Once C_TIMEOUT is finished, an event is registered in the timers queue. When EventLoop checks the timers queue again, it detects the event and executes the corresponding callback, resulting in the “hello” message appearing in the console. The counter is then decremented, and the EventLoop continues to check each queue until the counter is zero.
5. At some point, depending on the file size, the C_FS operation is completed, and an event is registered in the I/O queue. Once again, EventLoop detects the event and executes the corresponding callback, which outputs the “world” message in the console. The counter is decremented again, and EventLoop resumes its work.
6. Finally, after checking the close event queue, EventLoop checks the counter. Since it is zero, the Node process is exited.

Visualizing the diagram can help you easily understand how the asynchronous code will work, and you won’t have to wonder about the results.

While we’ve been discussing Macrotasks, let’s discuss another important thing.

I/O Polling

This process often confuses those who attempt to learn about the EventLoop. Many articles mention it as part of the EventLoop, but few explain what it does. Even in the official documentation is hard to understand what does it.

Let’s take a look at this example.

const fs = require('fs')
const now = Date.now();

setTimeout(() => {
  console.log('hello');
}, 50);

fs.readFile(__filename, () => {
  console.log('world');
});

setImmediate(() => {
  console.log('immediate');
});

while(Date.now() - now < 2000) {} // 2 second block

We have three operations: setTimeot, readFile and setImmediate.
In the end, we have a while loop that blocks the thread for two seconds. During this time, all three corresponding events should be registered in their respective queues. This means that when V8 finishes executing the while loop, EventLoop should see all three events in the same cycle and based on the diagram execute the callbacks in the following order:

hello
world
immediate

But the actual result looks like this:

hello
immediate
world

It’s because there is an extra process called I/O Polling.

Unlike other types of events, I/O events are only added to their queue at a specific point in the cycle. This is why the callback for setImmediate() will execute before the callback for readFile() even though both are ready when the while loop is done.

The issue is that the I/O queue-checking stage of the EventLoop only runs callbacks that are already in the event queue. They don’t get put into the event queue automatically when they are done. Instead, they are only added to the event queue later during the I/O polling.

Here is what happens after two seconds when the while loop is finished.

1. The EventLoop proceeds to execute the timer callbacks and finds that the timer has finished and is ready to be executed, so it runs it.
   In the console, we see “hello”.
2. After that, the EventLoop moves on to the I/O callbacks stage. At this point, the file reading process is finished, but its callback is not yet marked to be executed. It will be marked later in this cycle. The EventLoop then continues through several other stages and eventually reaches the I/O poll phase. At this point, the readFile() callback event is collected and added to the I/O queue, but it still doesn’t get executed yet.
   It’s ready for execution, but EventLoop will execute it in the next cycle.
3. Moving on to the next phase, the EventLoop executes the setImmediate() callback.
   In the console, we see “immediate”.
4. The EventLoop then starts over again. Since there are no timers to execute, it moves to the I/O callbacks stage, where it finally finds and runs the readFile() callback.
   In the console, we see “world”.

This example can be a bit challenging to understand, but it provides valuable insight into the I/O polling process. If you were to remove the two-second while loop, you would notice a different result.

immediate
world
hello

setImmediate() will work in the first cycle of EventLoop when neither of the setTimeout or File Systems processes is finished. After a certain period, the timeout will finish and the EventLoop will execute the corresponding callback. At a later point, when the file has been read, the EventLoop will execute the readFile’s callback.

Everything depends on the delay of the timeouts and the size of the file. If the file is large, it will take longer for the read process to complete. Similarly, if the timeout delay is long, the file read process may complete before the timeout. However, the setImmediate() callback is fixed and will always be registered in the event queue as soon as V8 executes it.

Let’s discuss some other interesting examples that will help us practice the diagram.

setTimeout & setImmediate

In this example, we have a timeout with a delay of 0 seconds and a setImmediate function. This is a tricky question, but if you answer correctly, it can leave a good impression on your knowledge during the interview.

setTimeout(() => {
  console.log('setTimeout');
}, 0);

setImmediate(() => {
  console.log('setImmediate');
});

The thing is, you never know which one will be logged first.

The thing is, you never know which one will be logged first. This is because sometimes a process can take longer(this is about milliseconds) to execute, causing the EventLoop to move past the timers queue when it is empty. Alternatively, the EventLoop may work too quickly, causing the Demultiplexer to not manage to register the event in the Event Queue in time. As a result, if you run this example multiple times, you may get different results each time.

setTimeout & setImmediate inside fs callback

In contrast to the previous example, the result of this code is predictable. Take a moment to examine the code and consider the order in which the logs will appear, using the diagram as a guide.

const fs  = require('fs');

fs.readFile(__filename, () => {
  setTimeout(() => {
    console.log('setTimeout');
  }, 0);
  
  setImmediate(() => {
    console.log('setImmediate');
  });
});

As the setTimeout and setImmediate are written inside the readFile function, we know that when the callback will be executed, then the EventLoop is in the I/O phase. So, the next one in its direction is the setImmediate queue. And as the setImmediate is an immediately get registered in the queue, it's not surprising that the logs will always be in this order.

setImmediate
setTimeout

Now that we have gained a good understanding of macrotasks and the workflow of the Event Loop, let’s continue exploring further.

First, let’s improve our diagram slightly by adding markings to indicate the phases that depict JavaScript executions.

We have a single JavaScript execution when we run the node.js process. After that, when there is nothing left to execute, V8 waits until the Event Loop receives an event and commands the execution of the corresponding callback. As you may have observed, there is a JavaScript execution phase after each queue phase. For instance, in the diagram, the execution of the callback for the timeout occurs during the second JavaScript phase.

So far, our discussion has focused on macrotasks, which didn’t include any information about Promises and process.nextTick. Those two are called Microtasks.

During each JavaScript execution phase, there is additional processing takes place.

These two types of microtasks have their dedicated queues. Additionally, there are other microtask schedulers as well, known as MutationObserver, queueMicrotaskbut for our discussion, we will focus on nextTick and Promise.

Once V8 executes all the JavaScript code, it proceeds to check the microtask queues, just like it does with macrotasks. If there is an event registered in the microtask queue, it will be processed, and the corresponding callback will be executed.

In the diagram, the light gray color represents the process.nextTick() queue, which holds the highest priority among scheduled tasks. The next dark gray color represents the Promise queue, which follows next in terms of priority.

Let’s try some examples.

process.nextTick & Promise

This is a basic example which demonstrates the workflow of Microtasks.

console.log(1);

process.nextTick(() => {
  console.log('nextTick');
});

Promise.resolve()
  .then(() => {
    console.log('Promise');
  });

console.log(2);

In terms of output sequencing, the process.nextTick() callbacks will always be executed before the Promise callbacks.

During the execution process, V8 begins with the first console log statement and then proceeds to execute the nextTick function, which registers an event in the queue. A similar process occurs with the Promise, where its callback is stored in a separate queue.
After V8 completes the execution of the last function call, resulting in the output of 2, it moves on to execute the events stored in the queues.

process.nextTick() is a function that allows a callback function to be executed immediately after the current operation completes, but before the Event Loop proceeds to the next phase.

When process.nextTick() is invoked, the provided callback is added to the nextTick queue, which holds the highest priority among scheduled tasks within the Event Loop. As a result, the callback will be executed before any other type of task, including Promises and other microtasks.

The primary use of process.nextTick() is for time-sensitive or high-priority operations that require prompt execution, bypassing the wait for other pending tasks. However, it is essential to exercise caution when using process.nextTick() to prevent blocking the Event Loop and causing performance degradation.

As long as at least one event remains in the Microtasks queue, the EventLoop will continue to prioritize it over the timers queue.

If we recursively run process.nextTick(), the EventLoop will never reach the timers queue, and the corresponding callbacks in the timers queue will never be executed.

function recursiveNextTick() {
  process.nextTick(recursiveNextTick);
}

recursiveNextTick();

setTimeout(() => {
  console.log('This will never be executed.');
}, 0);

In the above code, recursiveNextTick() function is invoked recursively using process.nextTick(). This causes the EventLoop to continuously process the nextTick queue, never allowing it to reach the timers queue.

As a result, the callback passed to setTimeout will never be executed, and the console statement inside it will never be printed.

Similarly, the same scenario will occur if we recursively use other microtasks. The EventLoop will be continuously occupied with processing the microtask queue, preventing it from reaching the timers queue or executing any other tasks.

Consequently, the callback passed to setTimeout will not be executed, and the console statement within it will never be printed.

function recursiveMicrotask() {
  Promise.resolve().then(recursiveMicrotask);
}

recursiveMicrotask();

setTimeout(() => {
  console.log('This will never be executed.');
}, 0);

This can lead to the EventLoop being blocked, which can cause scheduled timeouts to run with inaccurate timing or potentially never execute at all.

setTimeout(() => {
  console.log('setTimeout');
}, 0);

let count = 0;

function recursiveNextTick() {
  count += 1;

  if (count === 20000000)
    return; // finish recursion

  process.nextTick(recursiveNextTick);
}

recursiveNextTick();

As you can observe, the timeout that was scheduled for 0 milliseconds executed after 2 seconds.

So be careful with Microtasks!

As a quick exercise, let’s try to predict the output of the code.

process.nextTick(() => {
  console.log('nextTick 1');

  process.nextTick(() => {
    console.log('nextTick 2');

    process.nextTick(() => console.log('nextTick 3'));
    process.nextTick(() => console.log('nextTick 4'));
  });

  process.nextTick(() => {
    console.log('nextTick 5');

    process.nextTick(() => console.log('nextTick 6'));
    process.nextTick(() => console.log('nextTick 7'));
  });
  
});

Here is the explanation:
When this code is executed, it schedules a series of nested process.nextTick callbacks.

1. The initial process.nextTick callback is executed first, logging 'nextTick 1' to the console.
2. Within this callback, two more process.nextTick callbacks are scheduled: one logging 'nextTick 2' and another logging 'nextTick 5'.
3. The callback logged as ‘nextTick 2’ is executed next, logging ‘nextTick 2’ to the console.
4. Inside this callback, two more process.nextTick callbacks are scheduled: one logging 'nextTick 3' and another logging 'nextTick 4'.
5. The callback logged as ‘nextTick 5’ is executed after ‘nextTick 2’, logging ‘nextTick 5’ to the console.
6. Inside this callback, two more process.nextTick callbacks are scheduled: one logging 'nextTick 6' and another logging 'nextTick 7'.
7. Finally, the remaining process.nextTick callbacks are executed in the order they were scheduled, logging 'nextTick 3', 'nextTick 4', 'nextTick 6', and 'nextTick 7' to the console.

Here is an overview of how the queue will be structured throughout the execution.

Proess started: [ nT1 ]
nT1 executed: [ nT2, nT5 ]
nT2 executed: [ nT5, nT3, nT4 ]
nT5 executed: [ nT3, nT4, nT6, nT7 ]
// ...

Meanwhile, it’s worth noting that referring back to the diagrams will greatly assist in understanding the underlying logic.

Microtasks & Macrotasks in practice

For the next exercises, you will need to work with the complete diagram to fully grasp the concept.

process.nextTick(() => {
  console.log('nextTick');
});

Promise.resolve()
  .then(() => {
    console.log('Promise');
  });

setTimeout(() => {
  console.log('setTimeout');
}, 0);

setImmediate(() => {
  console.log('setImmediate');
});

This should be an easy one!

Here’s an explanation of how each of these functions behaves:

1. process.nextTick: This function schedules a callback to be executed, immediately after the current execution process completes. In the code, the callback logs 'nextTick' to the console.
2. Promise: The Promise.resolve creates a resolved promise, and the attached .then method schedules a callback to be executed. In the code, the callback within the .then() logs 'Promise' to the console.
3. setTimeout: This function schedules a callback to be executed as a macrotask after a specified delay. In the code, the callback logs 'setTimeout' to the console. Although the delay is set to 0 milliseconds, it still gets queued as a macrotask and will be executed after any pending microtasks (nextTicks, Promises).
4. setImmediate: Similar to timeout, this function also schedules a callback which will be executed as a macrotask.

The execution order will follow this sequence:

1. process.nextTick
2. Promise
3. setTimeout
4. setImmediate

It’s important to note that the Event Loop processes microtasks (such as process.nextTick and Promise) before macrotasks (such as setTimeout and setImmediate), and the order within each category is respected.

Okay, now let’s dive into something more challenging.

const fs  = require('fs');

fs.readFile(__filename, () => {
  process.nextTick(() => {
    console.log('nextTick in fs');
  });

  setTimeout(() => {
    console.log('setTimeout');
    
    process.nextTick(() => {
      console.log('nextTick in setTimeout');
    });
  }, 0);
  
  setImmediate(() => {
    console.log('setImmediate');

    process.nextTick(() => {
      console.log('nextTick in setImmediate');

      Promise.resolve()
        .then(() => {
          console.log('Promise in setImmediate');
        });
    });
  });  
});

Looks scary, isn’t it? It will be even worst if it’s an interview…
Well just remember the diagram and everything will be much easier.

When V8 executes the code, initially there is only one operation, which is fs.readFile(). While this operation is being processed, the Event Loop starts its work by checking each queue. It continues checking the queues until the counter (I hope you remember it) reaches 0, at which point the Event Loop will exit the process.

Eventually, the file system read operation will be completed, and the Event Loop will detect it while checking the I/O queue. Inside the callback function there are three new operations: nextTick, setTimeout, and setImmediate.

Now, think about the priorities.

After each Macrotask queue, our Microtasks are executed. This means “nextTick in fs” will be logged. And as the Microtask queues are empty EventLoop goes forward. And in the next phase is the immediate queue. So “setImmediate” will be logged. In addition, it also registers an event in the nextTick queue.

Now, when no immediate events are remaining, JavaScript begins to check the Microtask queues. Consequently, “nextTick in setImmediate” will be logged, and simultaneously, an event will be added to the Promise queue. Since the nextTick queue is now empty, JavaScript proceeds to check the Promise queue, where the newly registered event triggers the logging of “Promise in setImmediate”.

At this stage, all Microtask queues are empty, so the Event Loop proceeds and next, where it founds an event inside the timers queue.
Now, at the end “setTimeout” and “nextTick in setTimeout” will be logged with the same logic as we discussed.

You can further enhance your understanding of Microtasks and Macrotasks by engaging in similar exercises independently. By doing so, you can gain insights into how these tasks operate and develop the ability to anticipate the sequence of results.

Just use this diagram and don’t forget about the I/O polling phase (which is a specific case)!

Btw if you check the Node.js source code in GitHub, you will notice that Microtasks are at the JavaScript level, and it’s easy to understand and see all those queues and logics… because it’s JavaSciprt, not C++.

Changes from Node v11

When running this example in a web browser …

setTimeout(() => console.log('Timeout 1'));
setTimeout(() => {
    console.log('Timeout 2');
    Promise.resolve().then(() => console.log('promise resolve'));
});
setTimeout(() => console.log('Timeout 3'));

The result would be as follows.

Timeout 1
Timeout 2
promise resolve
Timeout 3

However, in Node versions prior to 11.0.0, you’ll receive the following output:

Within the Node.js implementation, process.nextTick, promise and other microtask callbacks are triggered during the transitions between each phase of the EventLoop. Consequently, during the timers phase of the EventLoop, all timer callbacks are handled before the execution of the Promise callback. This particular order of execution is what ultimately produces the output that has been observed and mentioned above.

Extensive discussions have taken place within the Node.js community regarding the need to address this issue and align the behaviour more closely with web standards. The aim is to bring consistency between Node.js and web environments.

In this scenario, instead of using setTimeout, I have incorporated another macrotask(setImmediate) to expand upon the example.

The release of Node.js version 11 brings forth noteworthy changes, enabling nextTick callbacks and microtasks to execute between every individual setTimeout, setImmediate and other macrotasks.
This update harmonizes the behaviour of Node.js with that of web browsers, enhancing the compatibility and reusability of JavaScript code across both environments.

Changes introduced by the Node.js team have the potential to impact the compatibility of existing Node.js applications. Therefore, it is crucial to stay informed and remain vigilant about Node.js updates. Being attentive to these updates is essential as it ensures that you are aware of any modifications that may occur. By staying tuned to Node.js updates, you can proactively address any changes that might affect your applications and ensure their smooth operation in the face of evolving technologies and frameworks.

Unblocking EventLoop using unref()

You will not wonder if I say that this code will keep the EventLoop alive and prevent the Node process from exiting.

setInterval(() => console.log('interval 2s'), 2000);

But what if I say that we can have an active timer and also remove that restriction that prevents the EventLoop from stopping?

So, we can make the interval process unimportant for the EventLoop.

const interval = setInterval(() => console.log('interval 2s'), 2000);

setTimeout(() => interval.unref(), 6000);

Interval works fine as soon we don’t call the unref function.

Unlike the .clearInterval() & .clearTimeout()methods, .unref() doesn’t stop the timer. It acts like a flag for your timers, marking tasks that EventLoop doesn’t need to wait for. They’ll run as usual while the process runs, but if the event queue is empty, they are ignored, and the process exits.

You can also make the timer important again by using .ref().

In Node.js 11 and later, you can check if the timer is currently blocking the process from exiting with timer.hasRef().

const interval = setInterval(() => console.log('interval 2s'), 2000);

console.log(interval.hasRef()); // true

CommonJS vs ES modules

process.nextTick(()=>{
    console.log('nextTick');
});

Promise.resolve().then(()=>{
    console.log('promise resolve');
});

console.log('console.log');

This is a rather straightforward example that we have previously discussed.

console.log
nextTick
promise resolve

However, when you attempt to utilize it with ES modules, you will observe a notable distinction.

If you understand how ES modules work and what they provide, then it is highly probable that you will comprehend the reason.

ES modules operate asynchronously, and when you compare the usage of require with imports, you will observe a significant disparity in their execution order. The key point to note is that ES modules function asynchronously, meaning that when the program begins, it does not initiate solely as a program in the conventional CommonJS fashion.

This difference in execution order is the reason behind the observed variations in microtask sequencing.

setImmediate(() => {
   process.nextTick(()=>{
      console.log('nextTick');
   });
  
   Promise.resolve().then(()=>{
      console.log('promise resolve');
   });
  
   console.log('console.log');
});

If you execute the same functions within a single macrotask, you will observe that the sequence remains as expected.

The program’s execution order is contingent upon the position of the pointer, which queue it resides in, and the task phase it is currently in. Consequently, the program’s execution order can be subject to change, which accounts for the observed differences.

In CommonJS we can load ES modules. Note that it returns a promise.
Now, I assume you understand that when it comes to ES modules, the pointer is positioned at the top of the Promise queue. This is the reason why, during program startup, promise queues are given higher priority than nextTick.

libuv

In OS, operations can be blocking or non-blocking. Blocking operations require a separate thread to enable concurrent execution of different operations. However, non-blocking operations allow for simultaneous execution without additional threads.

File and DNS operations are blocking, meaning they block the thread until completion. On the other hand, network operations are non-blocking, enabling multiple requests to be sent from a single at the same time.

Different operating systems provide notification mechanisms for Non-Blocking I/O. In Linux it’s called epoll, in Windows, it’s called IOCP, and so on. These notification mechanisms allow us to add handlers and wait for operations to complete, they will notify us when a specific operation will be finished.

Libuv uses those mechanisms to allow us to work with the Network I/O asynchronously. But with the Blocking I/O operations, it’s different.

Think about it, Node.js operates on a single thread with an EventLoop, which runs in a semi-infinite loop. However, when it comes to Blocking I/O operations, Libuv cannot handle them within the same thread.

So for that reason, Libuv uses a Thread Pool.

CPU-intensive tasks and Blocking I/O operations pose challenges because we can’t handle them asynchronously. But fortunately, Libuv got a solution for that. Baisicly it utilizes threads to tackle such situations effectively.

With a default thread pool size of 4, Libuv handles file read operations by executing them within one of those threads. Once the operation is completed, the thread is released and Libuv delivers the corresponding response. This enables efficient handling of Blocking I/O operations in an asynchronous manner using the thread pool.

If you attempt to perform 10 file read operations, only 4 of them will initiate the process while the remaining 6 operations will wait until threads become available for execution.

If we want to perform numerous Blocking I/O operations and find that the default thread pool size of 4 is insufficient, we can easily increase the thread pool size.

For that, we need to use this ENV variable.

UV_THREADPOOL_SIZE=64 node script.js

So in this case Libuv will create a Thread Pool with 64 threads.

Please note that having an excessive number of threads in the thread pool can lead to performance issues. This is because maintaining numerous threads requires significant resources. Therefore, it is important to carefully consider the implications before working with this environment variable.

You may wonder why there are 71 threads instead of 64. The additional threads are utilized by V8 and other components for tasks such as garbage collection and code optimization. These operations require resources, which is why the thread count surpasses the expected 64 threads.

Note that if you don’t use any Blocking I/O operations, the thread pool will not be initialized. You will only observe multiple threads if the pool is initialized, which can be done by executing a single Blocking I/O operation.

require('fs').readFile(__filename, () => {}); // Blocking I/O

setInterval(() => {}, 3000);

In my example, I’ve used this piece of code.
Simply remove the first line and observe that the thread count is noticeably reduced.

The reason for this is that the interval operation is not a Blocking I/O operation, which means the Thread Pool is not initialized.

DNS is problematic in Node.js

In Node.js, the dns.lookup function is a Blocking I/O operation when resolving hostnames. If you specify a hostname in your request, the DNS lookup process will introduce a blocking operation, as the underlying implementation may rely on synchronous operations.

However, if you work with IP addresses or utilize your DNS lookup mechanism, you have the opportunity to make the process fully asynchronous. So you can eliminate potential blocking and ensure a fully non-blocking execution flow in your Node.js application.

http.get("https://github.com", {
    lookup: yourCustomLookupFunction
});

This is a pretty interesting topic, so I would suggest you read this article.

Custom Promises - Bluebird

Why not natives?

You may have noticed that people often utilize custom-written promises, such as Bluebird.js. However, Bluebird.js offers much more than just a set of useful methods. It distinguishes itself by providing advanced features, enhancing promise performance.

This is a straightforward visualization of how native Promises work. Essentially, each Microtask has its corresponding callback.

In Bluebird, you can customize the behaviour of promises, which can result in improved performance depending on various situations.

By default, Bluebird combines all resolved promises and executes them within a single task, known as a Macrotask. In my visual example, it occurs within the setImmediate phase, but by default in Bluebird, it’s setTimeout. This approach allows us to prevent thread blocking when dealing with many promise calls.

Basically, in the queue, we will have one event, and the corresponding callback will include as many callbacks as the number of resolved promises we have.

Something like this.

setImmediate(() => {
  promiseResolve1();
  promiseResolve2();
  promiseResolve3();
  promiseResolve4();
});

By the way, we can also configure Bluebird to make promises run in a different phase.

Promise.setScheduler(function(fn) {
    process.nextTick(fn);
});

In this case, promises will have the highest priority.

Bluebird uses setTimeout(fn, 0) as a default scheduler. This means that the Promises will be run in the timers phase.

Just try it yourself and you will see how interesting it works.

Summary

Node.js is constantly evolving, with new updates and features being released regularly. Therefore, it’s important to stay updated by following the Node.js changelog. By doing so, you can stay informed about the latest changes and advancements, enabling you to have an up-to-date diagram of the Node.js architecture and its functionalities in your mind.

Node.js

It’s a JS runtime which allows us to build server-side applications which can work with OS.

Libuv was initially developed for Node.js. It is a powerful library that serves as the foundation for the Event Loop and offers additional functionalities. It is designed to facilitate Asynchronous I/O operations across different platforms such as Windows, Linux, and others, providing ample opportunities for efficient and Non-Blocking I/O handling.

Node.js incorporates a wide range of libraries and essential components that are critical for various processes and operations. These components greatly enhance the functionality and capabilities of Node.js.

EventLoop (Macrotasks and Microtasks)

The Event Loop, the core of Node.js, is implemented in C and C++. It serves as a fundamental mechanism that manages the execution of JavaScript code. It provides multiple queues, known as macrotasks, which correspond to different operations within Node.js. These queues ensure that tasks are executed in the appropriate order and efficiently handle events, I/O operations, and other asynchronous tasks.

In addition to the Event Loop, Node.js also introduces the concept of microtasks, which exist at the JavaScript/Node.js level. Microtasks encompass promises and nextTicks, and they provide a way to execute callbacks asynchronously and with higher priority. Microtasks are processed within the Event Loop after each Macrotask, allowing for finer-grained control and handling of asynchronous operations in Node.js.

A challenging interview question

In a recent job interview, I was asked a nice question about the Node.js Event Loop. It was interesting and made me think. However, I couldn’t give the best answer during the interview because it wasn’t clear to me at first. If I had more time, I think I could have figured it out better.

const http = require('http');

// Create a basic HTTP server
const server = http.createServer((req, res) => {
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.end('Hello, this is your Node.js server!');
});

server.listen(3000);

// Blocking operation
async function block() {
  for (let i = 0; i < 100; i++) {
    Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 100);
    console.log(`Waited ${i + 1} times`);

    // YOU CAN ONLY ADD CODE, YOU SHOULDN'T CHANGE ANYTHING
  }
}

block();

We have a simple HTTP server. When we start the server, we also execute a blocking operation. You can skip the Atomics.wait function; it's just a straightforward blocking operation that pauses the JavaScript thread for 100 ms.

The thing is that when you run this, you will see logs for each iteration of the loop. If you try to access your HTTP server via curl, the request will hang. This is because the event loop is busy and can’t handle the HTTP request.

When the for loop is over, the HTTP request will be processed.

There is a comment for you in the code.

// YOU CAN ONLY ADD CODE, YOU SHOULDN'T CHANGE ANYTHING

You need to add a solution that prevents the blocking operation from impacting the event loop, allowing it to handle HTTP requests. Simultaneously, it should function as expected, blocking the thread for 100 iterations, each lasting 100 ms.

Don’t overthink, don’t use worker_threads.

The solution is simple, but the idea for me is brilliant.

You just need to add some pieces of code. Now think about it.

Hint

In case you need a hint, notice that the blocking function is asynchronous.

Solution

Ta da da dam. Here it comes!

await new Promise(r => setTimeout(r, 0));

Previously, our for loop wasn’t dependent on macrotasks; Event Loop didn’t have a chance to check the queues.
With this approach, we introduce a minimal dependency on macrotasks, specifically on setTimeout. Our asynchronous function awaits a promise that resolves in a manner facilitating the Event Loop to complete one cycle. Within that cycle, it manages to handle the HTTP request.

I hope this was interesting for you. And thank you to the person who created this interview question.

I would like to express my sincere gratitude to those enthusiasts who have contributed their time and expertise to translate this article.

Below, you will find the translated versions of this article, along with the names of the respective translators:

• Non Conosci l’EventLoop di Node.js (Italian) — Andrea Di Blasi

If you’d like to make this article in your preferred language, we encourage you to do so and refer back to the original source. I believe that translations play a crucial role in making knowledge accessible to a global audience.

Thank you for taking the time to read this comprehensive article. I hope you found it informative and gained valuable insights from it.
Feel free to ask any questions or tweet me @nairihar

Also follow my JavaScript newsletter on Telegram: @javascript

• How to Scale Node.js Socket Server with Nginx and Redis
• Graceful Shutdown in NodeJS

You Don’t Know Node.js EventLoop was originally published in Bits and Pieces on Medium, where people are continuing the conversation by highlighting and responding to this story.

Despite being one of the most widely-used programming languages on the web, JavaScript has a bit of a bad reputation. But the truth is, JavaScript can be an incredibly powerful tool for creating client-side and server-side applications.

JavaScript’s flexibility and ease of use have made it a popular programming language, but these benefits can also have drawbacks. Without strict typing and with many processes already abstracted, it can be easy to make errors in JavaScript without realizing it.

It’s as if you’re walking in a forest where it’s easy to take the wrong path without any warning signs or guidance. On the other hand, in other programming languages, the language itself acts as a guide, leading you to learn algorithms, data structures, and memory management, making it more difficult to lose your way.

Let’s dive deeper into the details.

Dynamic typing

JavaScript doesn’t have types. We can do whatever we want.

let value = 2023;

value = [ 1, 2, 3 ];
value.push({ firstname: 'John' });

value[2] = true;

value[3].firstname = function() {};

We can create and manage a variable however we like. Well, JavaScript allows us to do that, but it’s not something which will be beneficial for us or for the engine. This approach is problematic because when we use different types of values for a variable, we may lose track of its intended value over time. Additionally, the engine may not perform efficiently when constantly changing its understanding of the variable.

When learning JavaScript, it’s important to focus on avoiding this type of code. It’s not worth spending too much time trying to understand it since you’re unlikely to encounter it in real-world projects. Even if you do understand it, you may forget why it produces a certain result in the future.

{} + 12 + [] + true 

Later point in case you would like to have types like in any other programming language then you can use TypeScript.

let age: number = 18;

let fruits: Array<string> = [ 'Apple', 'Orange', 'Banana' ];

// ...

So it’s a myth and misunderstanding that in JavaScript we don’t have types and etc …

Built-in data structures

People who start learning programming via Java, C# or similar programming languages, tend to know about many things related to data structures, algorithms, strict typings and etc. Java offers built-in data structures like LinkedList, Stack, and Queue, while JavaScript does not.

It’s rare for instructors to begin teaching JavaScript with data structures and algorithms, as it’s often taught alongside HTML and CSS. As a result, some developers may lack fundamental programming skills when they start their careers, leading to negative perceptions of JavaScript in the industry.

Success hinges on choosing the right path.

It would be better to take a quick course about fundamental computers since and learn about basic data structures and algorithms.

Learning data structures and algorithms can be effectively done in JavaScript by implementing them through the use of Arrays and Objects. This approach can enhance our understanding of these concepts and allow us to practice them in real-world scenarios using JavaScript.

It’s not only for web

Although JavaScript was originally developed for use in web browsers, it has evolved over the years and can now be utilized in a variety of contexts. Today, it is widely used not only for client-side web development, but also for server-side programming, IoT, and more.

This is the key point which makes JavaScript so unique!

Multiple ways to achieve the same thing

If you are a developer who learned programming using other languages, you may find yourself wondering why there are so many similar tools available when working with JavaScript.

function -> function, arrow function, generator function
variable -> var, let, const
class    -> function, class

It’s true that there are many different tools available to achieve the same thing in JavaScript. This can be attributed to the language’s history and standardization process, which allows for continual improvement year after year.

Initially, JavaScript was a fairly simple language that was primarily used in web development. As it became standardized under the name ECMAScript, the community began making proposals for improvements and new tools each year.

If you want to gain a comprehensive understanding of JavaScript and become a skilled engineer, it’s important to learn the language’s history and become familiar with its tools in a systematic order.

Summary

JavaScript is a truly powerful programming language that has been consistently improving every year. Its large and active community is a priceless asset. Although it may seem like some problems in JavaScript have complex solutions, the reality is that there is usually a simple answer that can be found by delving deeper into the language. However, due to its non-traditional nature, learning JavaScript may present some difficulties. In fact, at times, understanding the entire concept of the language can be challenging.

Thank you, feel free to ask any questions or tweet me @nairihar

Also follow my “JavaScript Universe” newsletter on Telegram: @javascript

• ECMAScript 2023 (ES14) — what to expect
• Global NPM/Node.js executables
• How to Scale Node.js Socket Server with Nginx and Redis

More content at PlainEnglish.io.

Sign up for our free weekly newsletter. Follow us on Twitter, LinkedIn, YouTube, and Discord.

Quick Tips for Learning Javascript was originally published in JavaScript in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.