Lawrence Pan

Should you grind Leetcode?

Tue, 28 Jul 2020 00:00:00 +0000

I have always hated the technical interview process, mostly because I don’t enjoy practicing Leetcode problems.

It is one of those dumb things in college that if you don’t do it, others will get ahead of you. If you want to secure those “prestigious internships”, you have to grind Leetcode.

Convincing myself to grind Leetcode was hard. It was hard to accept that life is unfair. Why do I need to do something I don’t like to get something I want? In one of my technical interviews during my freshman year, as an icebreaker, the interviewer asked if I like algorithms. I told him to his face that I hated algorithms. I also made sure to mention that I believed all technical interviews were a waste of time. I didn’t get the job. Not too surprising!

Today, I am a fourth-year student about to graduate. Not too long ago, I was on the phone chatting with a younger friend who was worried about his upcoming internship search. He asked me for advice because he thinks I am an expert at internship hunting. And of course, being extremely flattered, I told him to grind Leetcode.

Pretty ironic right? Not to mention that the young lad also hates practicing Leetcode questions. He prefers building side projects, code that makes a real-life impact.

I told him, like how the seniors told me back in the days: life is hard; there is no shortcut; now go home and grind Leetcode.

Did I change my mind? Not really. I still think grinding Leetcode is dumb. But it is a necessary evil until companies realize it is dumb. The good news is that many companies have already revamped their SWE hiring process.

Coding is about translating one’s thought process into code. In the real world, software engineers translate business logic (what they want the computer to do) into maintainable and performant code. The purpose of whiteboard interviews is to ensure that the candidate knows how to do this translation. They give you a specification in the form of a programming puzzle. Then, you translate it into code.

Here is an example of a successful whiteboard interview from the candidate’s perspective

Understand the programming puzzle
Come up with an algorithm to solve the puzzle
Implement the algorithm (translate thoughts into code)

In my opinion, only the third step somewhat reflects one’s real-world programming skills. Grinding Leetcode will help you to be good at all three steps. However, being good at the first two steps is not very useful for most software engineering roles.

So, should you grind Leetcode?

I might be wrong, but here are my two cents. I believe that every student interested in pursuing a software engineering career should do at least one internship at a “big shop” (Facebook, Google, Microsoft, Amazon, Apple, etc). At these companies, you will see what good looks like: world-class developer tools, mature product development process, rigorous performance review framework, etc. If those companies conduct Leetcode-style interviews, you should grind Leetcode. As of now, both Facebook and Google ask Leetcode-style questions in their software engineering internship interviews. If you want to work there, you should grind Leetcode.

My first stab at Climate

Tue, 30 Jun 2020 00:00:00 +0000

For years, I have been hearing people talking about the problem of climate change. However, I never got the chance to dig into it, until recently I read Ryan Orbuch’s post on Carbon removal. Orbuch’s post briefly mentioned this book called “Carbon Capture”. It caught my attention because of the credibility of the author (MIT researcher spent 30 years in this field). This book is also my first stab at the climate tech rabbit hole.

Climate change is happening because we have emitted too much CO2. There are many ways to address it: mitigation (don’t emit as much CO2), adaption (just change our lifestyle to deal with this rise of temperature), Carbon Dioxide removal (trying to capture CO2 in the atomsphere), etc. The author suggests that mitigation would be the most effective way in the long run.

There are many mitigation techniques. The two most popular ones are (1) moving towards renewable energy sources such as Hydro, Nuclear, Wind, Solar; and (2) lowering the CO2 emission of existing fossil fuel energy plants using the technique of Carbon Capture.

The book primarily focuses on Carbon Capture. The first half of the book provides an overview of the history as well as the current state (as of 2018) of the Carbon Capture technology. I learned the different ways of capturing CO2 from the flue gas produced by the coal plants, as well as how we would store these captured CO2.

The book also briefly touches on NETs (negative emission technologies). The author suggests that currently NETs are not scalable enough. CO2 is extremely diluted in the atmosphere, and capturing it would be very costly. The most popular NET is planting trees, but it is still orders of magnitude away from the target.

The second half of the book discusses the current political climate around addressing climate change. There are two ways for policy makers to address this issue: “market pull” and “technology push”. Market pull is about implementing carbon tax. The author thinks it is the most effective approach because we could just let the market decide what would be the best technology to address climate change. However, this strategy is not favoured by politicians because carbon tax might impact their reelection. For example, they might lose votes from people who have long commutes by car. Also it is not fair since many states (eg. Washington) have geographical advantages to build hydro plants, but many states do not have this advantage so they have to rely on fossil fuels (eg. West Virginia). On the other hand, technology push is about government giving out grants to fund R&D that would address climate change. However, it would be up to the government to pick who would receive the grant, which could be biased.

The author also argues that we should not soley rely on renewable energy, since solar and wind plants are not very reliable; not every region can build hydro plants; and nuclear is controversial. In addition, since the majority of the power plants are fossil-fuel-based, implementing Carbon Capture is significantly cheaper than migrating them to renewable energy plants.

Overall, I really enjoyed reading this book. It fed my curiosity and I learned a lot about the current landscape of addressing climate change. However, I suspect that the author might be biased towards “Carbon Capture” over other mitigation technologies. I would recommend anyone who wants to learn more about state-of-the-art climate technologies to give it a read.

I also learned that the climate challenge is extremely political. The technology seems to be ready. But the bottleneck is public policies, which are influenced by the economy and the culture (how the voters perceive this). I wonder what we can do to contribute to this problem.

[Reading Reflection] Zero to One

Sun, 28 Jun 2020 00:00:00 +0000

The first half of the book presents Thiel’s view on entrepreneurship and innovation. For example, contrary to popular belief, he condemns competition and praises monopoly.

Throughout this book, Thiel has been stressing the importance of thinking for yourself. Our society has transitioned from “definite optimism” to “indefinite optimism”. Many extremely smart people, who could have created these “0 to 1” changes, ended up getting sucked into lucrative industries such as investment banking, simply because these professions are considered “safe”. As how Thiel puts it in the book,

What could be a more appropriate reward for two decades of resume-building than a seemingly elite, process-oriented career that promises to “keep options open”.

This is actually coherent with his “Thiel Fellowship” program. He is not encouraging people to not go to college. Instead, he wants those who are capable to drop out of this “conventional path”, so that they could focus on making “0 to 1” changes to our society, to make it better.

The second half of the book discusses the methodology to create such “0 to 1” ventures. For example, he encourages early-stage entrepreneurs to focus on small niche markets, and “monopolizing” on small things.

I really enjoyed reading this book, especially learning about Thiel’s philosophy on entrepreneurship. How he contrasts “definite optimism” and “indefinite optimism” is also very eyeopening. Most importantly, this book made me realize that, as embarassing as it sounds, I haven’t been thinking for myself, which consequently motivated me to write this blog post.

I encourage everyone, especially university students studying CS, to give this book a read.

The next big thing

Thu, 25 Jun 2020 00:00:00 +0000

I have always been very proud of this blog. So proud that I even put it on my resume. In fact, showcasing this blog on my resume, has proven to be very effective. I managed to secure an internship offer at Datadog two years ago, beating many fourth-year students when I was just a confused sophomore, simply because the interviewer really liked one of my blog posts.

The blog post that impressed the hiring manager at Datadog is still on my resume today, despite it having been written almost three years ago. Recently, a founder of a venture-backed start-up reached out to me and asked if I want to join his 15-person engineering team. During our chat, he told me that he was particularly impressed with my blog. He claimed that he has read all of my blog posts, and he said that he could tell, just from these posts, that I am “one of them”. According to him, I am a builder, someone who is curious and willing to take risks. It also seems that the blog has been circulated within that company. When I was chatting with a few other engineers on the team as part of the interviewing process, many of them also expressed how they liked my blog. The interview went well and I ended up getting an offer, with a very generous package for my experience level.

I have no intention to boast here. In fact, I will probably be doing the opposite in this blog post. I am no longer the person depicted in this blog, the builder, the risk-taker, the self-proclaimed entrepreneur. I changed. Here is a quote from the first post published on this blog, “Hello World”.

I am not satisfied to be just an engineer. I want to bring real revolutionary changes to our society and make it a better place.

I changed. How ironic. It has been more than a year and a half since I wrote my last blog post. It has been, I don’t know how long, since I revisited my first blog post, “Hello World”. When I was reading the quote above just earlier today, I was shocked. It might sound stupid but it almost made me cry. I was no longer the adventurous, risk-taking, and ambitious me.

I first got into coding in 2015, purely because I wanted to build things. My high school at the time didn’t offer any Computer Science courses, and I received zero pressure from my parents. I taught myself HTML and CSS just because I wanted to build websites. I built a lot of them, and even launched some of them in the wild. I didn’t care about making any money. I would be satisfied just by watching the total number of users reaching 3 digits on the Google Analytics dashboard.

Another quality that I had, and that fortunately still sticks with me today, is the willingness to dig deep into things. When I learned HTML and CSS in 2015 just because I wanted to build websites, I quickly became aware of the industry-standard tools, thanks to the Internet and a few mentors I met on the way. I learned about the then-popular movement of SPAs (single page application). I learned that people use Vim to edit code, and use Git to “manage” it. I taught myself Angular JS, and then React. I got myself a Twitter account so I could follow Dan Abramov.

I eventually managed to find myself a Web Development internship, earning 20 CAD an hour, in the summer of 2016 right before I started University. I had zero connection then. I just wanted a job, and preferably something that involves coding. So I searched on Craigslist and Indeed, and applied to at least 100 postings. Luckily, two got back to me. One was for an internship position at a local software agency, who turned me down after learning that I was only 18 when I showed up in person. The other was for an QA position at a hip local internet start-up called Plotly. I managed to impress the interviewer with how much I know about NPM and Webpack that he just decided to hire me on the spot and put me on the front-end team.

My story is far less glamorous than the success stories of these teenage outliers———young entrepreneurs who teach themselves coding at 13 and sell their start-ups to Facebook at 17. But I believe my story is truly special. It was about a boy who is passionate about building things that would create value, and make our world a bit better. He learned to code, decided to pursue tech, and went to the University of Waterloo to study Software Engineering. It all started in the summer of 2016. Now, four years later, what is he doing now?

I am here behind my four-year old laptop typing up this blog post (I bought this computer with the money I made from the job at Plotly right before I started University in September 2016). In the past four years, I did five other internships at five different companies working on completely different things.

Worked with PM and designers to build user-facing features at Universe
Contributed to the low-latency time-series ingestion pipeline at Datadog
Built tooling to manage PlanGrid’s Kubernetes infrastructure
Added features to Twitter’s fault-tolerant stream processing engine
Designed API and evolved complex data models at Stripe

I can say that I am a much better engineer than the old me from four years ago. Thanks to these internships, I got to experience pretty much every software engineering role at a typical SaaS company: doing front-end web development at Universe; designing back-end API in Ruby at Stripe; writing and scaling Golang micro-services at Datadog; developing infra tooling and tweaking YAML files at PlanGrid; maintaining a critical but legacy infrastructure system in Java and C++ at Twitter. There is still a lot for me to explore, but I believe that I now have a pretty good sense of what I should expect from a software engineering career. I have also met many extremely intelligent and inspiring people during these internships who still serve as my mentors even to this day.

I also went through the rigorous CS education at the University of Waterloo. They made us write a compiler that could take a subset of Scala to output MIPS machine code. As part of the Operation Systems course’s group project, my friends and I wrote a minimal kernel that could run on an ARM board. I think they just made us write a lot of code, especially C++. We also had to take many math/engineering courses such as Control Systems and Graph Theory, on top of Calculus, Stats, Algebra, and many other first-year courses.

Back home in Montreal, I was pretty much the only kid who was interested in tech. Everyone else wanted to get into med school. However, at Waterloo, everyone seems to be into the same thing as I am. We shared a collective goal, and we motivated each other, implicitly via the impostor syndrome, to keep us on the grind.

I am way more capable than what I was four years ago. This is not limited to my technical abilities. I also learned how to break things down logically so I could make better decisions, as well as take responsibility for them. I learned how to think clearly. I have also become more self-aware. I learned how to cook myself nutritious meals. I developed a routine and I have become more fit, both physically and mentally.

However, comparing to the me before University, I found that I have become passive, occupied with things that are not initiated by me. I felt that I was pushed to follow train tracks laid by others. For example, chasing prestigious internships and getting a full-time job offer with the highest possible compensation package.

I don’t regret chasing these prestigious internships———forcing myself to read these technical books (shout out to “Designing Data-Intensive Applications”), grinding Leetcode, and preparing for interviews. Without these efforts, I wouldn’t have such a diverse work experience, and I wouldn’t meet my mentors. However, one thing to note is that I wasn’t being deliberate. I went on this path because everyone else was doing it.

For the past few months, I felt slightly unhappy. I wasn’t satisfied with my current personal development trajectory, even though I was still proud of myself, my discipline, and hard work.

I have been doing a lot of journaling and reflection lately. I believe my discontent was because of the lack of deliberation, that I have been following “train tracks” laid by others. I have still yet to figure out my own “train track”, my next big thing. But I know for sure that it is the time for me to start thinking for myself, and doing things for myself.

I am certainly no longer the same person as four years ago. But one thing that remains is that I am still excited about building things, to make our world a bit better, as what I stated in “Hello World” four years ago:

I am not satisfied to be just an engineer. I want to bring real revolutionary changes to our society and make it a better place.

My one-liner Linux Dropbox client

Sat, 22 Dec 2018 00:00:00 +0000

In this blog post, I want to discuss one of my recent attempts to create a simple one-liner Linux Dropbox client using only free and open-source components, including rclone, entr, and systemd.

Context

Recently, the proprietary Dropbox Linux client dropped support for all Linux file systems except unencrypted ext4. And, my home directory is “unfortunately” encrypted.

In early December, the proprietary Dropbox Linux client I have been using stopped working. It logged me out, and prompted me to choose a different sync folder on a “supported file system”.

By the way, I run Ubuntu bionic on a two-year-old Thinkpad t460s.

How I use Dropbox

I am a heavy Org mode user. I take notes in plain text, and my use case of Dropbox is simply “continuously backing up my notes while I type”.

If you are also into data infrastructure, my use case is very similar to “asynchronous single-master replication”. All writes go through my Thinkpad, the master. The remote Dropbox folder is just a read-only follower that I occasionally “issue read-only queries” to, or used as a backup to construct a new master when the current master fails or gets stolen.

Nevertheless, this replication setup has saved my life multiple times. I still remember, very vividly, that my Thinkpad couldn’t boot during the exam season at the end of my sophomore year. Since I continuously replicated all my notes to Dropbox, I didn’t lose any data and I was able to view my latest notes on my mom’s Macbook. Thanks mom!

My failed attempts

When the Dropbox client stopped working, my main focus was to find another similarly feature-rich remote storage client for Linux. I also wouldn’t mind migrating to another storage back-end, such as Google Drive or AWS S3. Some of the other options I considered include overGrive and insync.

However, I concluded that these solutions are way too feature-rich, and not well-suited for my use case.

For example, these clients are modelled as mounting a remote file system onto your file system. They try really hard to abstract the remote file systems away by making them look and feel like local file systems. They typically implement two-way syncing, automatically mapping remote file types to Linux file types, etc.

I don’t need this level of abstraction. I just need something simple that allows me to back up my notes to the cloud continuously while I type. In addition, the abstraction also makes it harder to set up and debug. Not to mention that most of these “feature-rich” clients are proprietary.

rclone

I ran into rclone and realized that it is exactly what I was looking for. rclone is simple but powerful. It is very similar to the rsync tool, but for “cloud storage”.

For example, it not only takes care of fault tolerance (integrity checks), efficient synchronization algorithms, etc., but also provides a simple CRUD interface to interact with popular cloud storage services, including Amazon S3, Google Drive, and Dropbox.

The following command will make the remote directory org identical with the local directory /home/lpan/org

ORG_DIR=/home/lpan/org
REMOTE=dropbox

rclone sync $ORG_DIR $REMOTE:org

entr

entr is a command runner that leverages the inotify API. It basically allows you to run a command triggered by file changes, without polling the file system.

One of its common usages is to rebuild the project when any of the source file changes.

entr takes a list of absolute paths from stdin, and then it will execute a command, passed in as an ARG, when any of the watched file changes.

WORKDIR=/path/to/myproject
find $WORKDIR | grep "\.cpp$" | entr make

The one-liner script

Now we learned about rclone and entr. The final script turned out to be very simple. To remind you about my use case of Dropbox, all I want to do is to continuously replicate my local Org file changes to Dropbox. Therefore, we just need to use entr to watch the files that we want to replicate, and use rclone to “sync” them to the remote storage.

The final script (/home/lpan/sync_dropbox.sh) looks like the following:

#!/bin/bash

ORG_DIR=/home/lpan/org
REMOTE=dropbox

find $ORG_DIR | entr -r rclone sync -v $ORG_DIR $REMOTE:org

Edit 2020-05-13: If you want to watch new files being added to the directory, you can use the -d flag provided by entr with an infinite loop, as shown below:

#!/bin/bash

ORG_DIR=/home/lpan/org
REMOTE=dropbox

while true; do
  find $ORG_DIR | entr -rd rclone sync -v $ORG_DIR $REMOTE:org
end

Make the script into a Daemon

A Daemon is “just a computer program that runs as a background process”. We make this script into a “background process” because we want it to continuously sync local file changes to the remote file system in the background.

systemd provides an interface for us to manage daemon processes.

I created a Dropbox Service in ~/.config/systemd/user/dropbox.service.

[Unit]
Description=Dropbox Daemon

[Service]
ExecStart=/home/lpan/sync_dropbox.sh
Restart=always

[Install]
WantedBy=default.target

Then, you can manage the daemon with the following commands

# reload the service file
systemctl --user daemon-reload

# start the daemon
systemctl --user start dropbox.service

# start the daemon on login
systemctl --user enable dropbox.service

# inspect the status of the daemon
systemctl --user status dropbox.service

Conclusion

In this post, I discussed how I apply the UNIX philosophy and use a set of free and open-source tools to replace the proprietary & deprecated Dropbox client. We talked about rclone and entr. I also showed you how I make this process a Daemon and manage it using systemd.

I want to remind you that the key idea of this post is simplicity. We want to solve our simple problems with simple solutions. My use case of Dropbox is very simple. And this is why one simple line of shell script would be a better solution than using a feature-rich, and possibly proprietary remote storage client.

Thanks so much for reading! I really hope you enjoy this post. Got a better way to do the same job, or know how to extend the script for another use case? Let me know in the comment section below!

Edit 2018-12-24: This article has been discussed on Hacker News and r/linux.

All aboard the gRPC train

Tue, 22 May 2018 00:00:00 +0000

Introduction

During my internship at Datadog this past winter, I took ownership of a critical service which handles around 40,000 requests per second as of April 2018.

To improve its reliability and fault tolerance, I migrated the service’s inter-process communication (IPC) layer from Datadog’s homegrown remote procedure call (RPC) framework to gRPC, an open-source RPC framework made by Google.

Context

The service is deployed on more than 200 nodes and it is consumed by approximately 500 other instances. It comes with a client library. Consumers of this service can simply use this client library to issue RPCs without worrying about service discovery, (client-side) load balancing, generating stub, error handling, etc.

My task was to deprecate this client library, and to migrate its consumers to use gRPC to communicate with the service instead.

However, deprecating a legacy package is never easy. There are always undocumented edge cases. Blindly replacing an old system with a new one may result in regressions (old bugs reappear).

Tests and CI are in fact designed to mitigate this problem. However, in a large-scale distributed system, certain tests (for example, load tests) are not practically possible to be included in the CI pipeline.

To fully understand the old system and to ensure that there is no regression after the migration, I had to go through the service’s outage docs, reinforce the monitoring system (metrics, logs, tracing), and design chaos testing (iptables/pkill -9 server instances).

In this post, I won’t go into details about how I carried out the migration (maybe another post!). Instead, I would like to discuss why it was worth the effort to migrate to gRPC.

The Old Framework

Originally, consumers communicate with the service using an in-house RPC framework. It was built on top of the Redis protocol to take advantage its rich ecosystem and mature tooling. This made sense years ago when Datadog’s infrastructure was not as complex or extensive.

However, the framework was not optimized for large-scale distributed systems. In a distributed system, network is never reliable and nodes can fail unpredictably and independently due to hardware faults or human errors. Reliable software tolerates faults. The in-house RPC framework, however, does not provide any mechanism to make the inter-process communication more reliable.

Unreliable Health Check

A service registry is essentially an eventually consistent snapshot of the current state of the system. It should absolutely not be regarded as the only source of truth.

For example, when a node suddenly becomes unresponsive (the process gets pkill -9‘ed, node is running out of file descriptors, etc), it will take seconds or maybe even minutes for the service registry (health check service) to realize that the node is dead. And it will take more time to propagate this (sad) news to every node in the system.

Therefore, when a consumer of a service retrieves a list of “healthy” nodes from the registry, the list could contain false-positive, already-dead nodes. The question then arises: how to ensure RPCs to only be sent to actually healthy nodes?

Transient Errors vs Fatal Errors

This is where the legacy RPC framework falls short. The framework is unable to identify the type of fault that causes an RPC to fail.

A fault could be transient, which means that if the client waits for a moment and retries (resends the previously failed RPC) to the same server node, it will eventually succeed.

On the other hand, an RPC failure can also be caused by a fatal error. This may imply that the destination node is dead. When a fatal error is identified, the client should send the RPC to another node, instead of retrying on the same node.

Poor Error Handling

Now, let’s go back to the old framework. Since it was not built for distributed systems, being able to identify the cause of a RPC failure and handle it accordingly was not possible. (That’s why people say programming distributed systems is hard! This problem would be trivial if everything happens in memory)

As a result, the legacy RPC framework simply sends the failed RPC to a different node returned from the service registry.

This approach can lead to many problems. For example, clients can run out of instance-wide retry budget quickly. Assume RPCs have locality (eg. client-side load balancing with consistent hashing). If a server node fails, all the RPCs that are mapped to that node will fail on the first attempt, and then retry.

In addition, if there are more than one false-positive nodes returned from the service registry (eg. when doing a rolling restart), the second attempt (the retry) may fail as well.

Fault Tolerance

Let’s say the service registry returns n “healthy” nodes, and among them, there are x false-positive nodes. Assume we use the old retry-on-a-different-node error handling technique and assume that we balance the load randomly. The probability of the client to send a RPC to a bad node is x / m.

The client will then retry on a different node. Now, the probability to hit another bad node becomes (x - 1) / (m - 1). On kth retry, the probability becomes (x - k) / (m - k).

If you are good at math, you will quickly see that if our retry budget is greater than x (assume x < m), we can guarantee that all the RPCs will eventually succeed.

In other words, the system’s fault tolerance is linearly dependent on the retry budget––the total number of failed nodes the system can tolerate literally is equalled to the retry budget. This is clearly not scalable because retries are expensive.

Having an aggressive retry strategy can result in server resource exhaustion which can then leads to cascading failures. On the other hand, conservative retries can exhaust client’s resource (because of back-offs and jitters) which limit the throughput per instance.

A fault tolerant system does not only minimize errors but also minimizes retries.

Node Outage

Node outage is also a tricky problem in distributed systems. For example, after the TCP handshake has been completed and the client has sent the request payload to the server, if the server node suddenly fails before it can respond to the RPC, the client may hang indefinitely. Although the server is dead, from the client’s perspective, the server may be processing a time-consuming query. In other words, there is no way for the client to identify the cause of an unresponsive node.

One of the common solutions is to set a RPC deadline based on the expected response time, which is typically estimated from the size of the request payload. If the deadline is exceeded, we can be confident to assume that the server is faulty. This distinguishes infrastructure faults from application faults. However, it is impossible to implement this functionality in the old RPC framework.

gRPC Comes to the Rescue

gRPC is a remote procedure call framework developed at Google. It has been adopted at many organizations including Netflix and Square.

Similar to the homegrown RPC framework, it exposes an API to generate stubs, integrate with the existing service discovery system, etc. But different from the legacy framework, gRPC is optimized for large-scale distributed systems. It has a huge emphasis on reliability and fault tolerance.

Connection State Management

Instead of treating the service registry as the absolute source of truth, the gRPC client library implements a connection state manager to maintain a pool of guaranteed healthy connections (sorta like a circuit breaker).

On initialization, the gRPC client library instantiates a state machine for each node returned from the service registry. The initial state for all the nodes is IDLE. If the gRPC client successfully establishes an connection (completes the TCP handshake with the target node), it will transition the connection state to READY. If the connection request is rejected or the connection is detected to be faulty, the connection state will be transitioned to TRANSIENT FAILURE. When a TRANSIENT_FAILURE connection is reestablished, it will be transitioned back to READY.

Source: gRPC Connectivity Semantics and API

This way, especially with consistent hashing, when the gRPC client detects that a node is in TRANSIENT_FAILURE (even though service registry says it’s healthy, but it is unresponsive in real life), it will simply remove the node from the list of “buckets”. As a result, RPCs which were originally hashed to the failed node will be rehashed to a different but healthy node. This significantly reduces the total number of retries.

RPC Deadlines

Deadlines and timeouts are pretty much the same…

With gRPC, it is also possible to dynamically set a deadline for individual RPCs. Setting a deadline allows the client to specify how long it is willing to wait for the server to process its request. This can effectively distinguish between fatal failures and long running queries.

For example, if an RPC hangs, there are two possible causes:

The query is very expensive and it will take a while.
The server is dead

For the first case, we only need to wait longer. On the other hand, as for the second case, we do not want to wait at all. We should cancel the RPC as soon as possible. RPC timeouts & deadlines helps to distinguish between the two causes.

We can estimate the maximum response time for a particular request with a certain payload size. Then we can derive the timeout from it. Thus, when an RPC fails because the timeout is exceeded (or deadline expired), we can be pretty confident to say that the server node is faulty (outcome #2) and we want to cancel the RPC and retry on a different node.

Application-Level Keepalive

In addition to the configurable RPC deadlines. gRPC also implements a application-wide keepalive protocol, which further increases the fault tolerance to failed or unresponsive nodes.

For example, if the server appears to be unresponsive for a certain length of time, the application-level keepalive, also known as the gRPC keepalive, will be activated. The gRPC client will send a keepalive ping to the server. If the server does not respond on time, the client will terminate the connection and transition the state of that connection to TRANSIENT_FAILURE to remove it from the healthy connection pool.

This distinguishes infrastructure faults from application faults. In the case of a long-running RPC, the server will be able to reply to the keepalive ping so the client may wait longer. On the other hand, if the server is dead, the client can be confident to terminate the TCP connection and retry the RPC on a different node.

Gains

It is hard to measure resiliency gains (can’t really say I increased the resiliency by 34.5% or decrease the number of future outages by 29%). However, there is in fact one notable gain, faster deployment!

Faster Deployment

Deploying a service typically requires all its instances to be restarted with a new binary. However, depending on how stateful is that service, a restart can result its instances to be unavailable for a length of time. For zero downtime deploys, instances are typically restarted in batches, so that at a given time, there are always enough instances that are available to serve requests.

Before gRPC is introduced, the maximum number of node failures the system tolerates could not exceed the RPC retry budget. As a result, to minimize errors, instances had to be restarted one at a time. This is not scalable because a service’s deployment time becomes linearly dependent on the number of instances of that service! As for the service I owned at Datadog, with approximately 200 instances and a 10-second warm-up time, it can take more than 30 minutes to deploy!

With the migration to gRPC and the increase in fault tolerance, instances can be restarted in larger batches, which significantly speeds up the deployment time.

More Reliable Performance

The response time of an RPC is the time between when the client initializes the request and when the client actually receives the response. If an RPC fails and the client decides to retry, the retry back-off time also contributes to the response time of the RPC.

Therefore, retries can directly impact a system’s performance. Previously with the old RPC framework, an unresponsive node can result in a large number retries or hanging RPCs. A rolling restart or independent node outages can potentially affect the throughput of the system.

The migration to gRPC greatly mitigated this problem. Thanks to gRPC’s per RPC timeout, we are completely immune to hanging RPCs. The number of retries is also minimized with the connection state management feature.

Conclusion

In this post, I talked about the problems with distributed systems and how gRPC improves fault tolerance by mitigating these problems. This is also my first distributed systems blog post. Thanks so much for reading!

Recently I have been reading Martin Kleppmann’s Designing Data-Intensive Applications and I was fascinated by the distributed data section. I find replication algorithms, consistency models, consensus protocols, etc., to be very interesting. Hopefully I can work on another distributed system for my next internship and gain experience in working with distributed data.

Probability and Types

Fri, 13 Oct 2017 00:00:00 +0000

Introduction

In this blog post, I will explain the concept of a Random Variable from a software developer’s perspective.

Definitions

Before we dive in, I want to define a few terminologies, for those of you who are new to probability and statistics. Feel free to skip this section if you think you are too cool for this.

Experiment

An experiment is an experiment… What would you do if you are asked to test if a coin is fair? I would flip the coin 1000 times and see if the counts of head and tails are equal. If head occurs only 300 times and tail occurs 700 times, you probably can conclude that the coin is not fair!

So what is an experiment? In the context of the example described above, flipping the coin 1000 times is an Experiment.

Trial

A trial is one coin flip. The coin-flip experiment above consists of 1000 trials!

Outcome

An outcome is one possible outcome of a trial. As for the coin-flip experiment, the outcome of the trial is either head or tail.

Sample space

A sample space is a set of all possible outcomes of the experiment. Assume we only perform 3 trials for our coin-flip experiment. The sample space will be { HHH, HHT, HTH, HTT, THH, THT, TTH, TTT }, where H denotes head and T denotes tail.

Cool stuffs

So, I learnt about a thing called Random Variable this week in an introductory statistics course I am taking at the moment. According to the lecture note, a random variable is both a function and a variable. It can be assigned to a value and it also defines the mapping of an outcome in the sample space to a value.

For example, let’s consider the coin-flip example. We can define a Random Variable X to be a function that takes an outcome of the experiment and returns the count of heads.

eg.

X(HHH) = 3
X(HTH) = 2
X(THH) = 2
X(TTT) = 0

Now, let’s define another term called Range. The Range of a Random Variable function is a set that contains all possible outputs of the function. Let’s go back to our coin-flip experiment that consists of only three trials. The Range of the Random Variable X we defined above will be 0, 1, 2, 3. It cannot be larger than three since there are only 3 trials and it cannot be negative because monkeys like bananas.

If you understand static typing, the following snippet may be helpful. If you don’t, TOO BAD!

type Outcome = Head | Tail;
type ExperimentOutcome: (Outcome, Outcome, Outcome)
type Range = 0 | 1 | 2 | 3 | 4
type myRandomVar = (Outcome) => (Range)
val sampleSpace: Array[ExperimentOutcome] = { ... }

What if you want to select a subset of the sample space that satisfies a value of our random variable?

For example, how do you select a subset of the sample space such that the subset contains only outcomes that contain exactly two heads.

You can just do

sampleSpace.filter (event) => X(event) == 2

Isn’t it neat?

Thanks for reading LOL

What I learnt from coding a text editor in C

Sat, 19 Aug 2017 00:00:00 +0000

Background

As a modern JavaScript developer, I have worked on numerous single page applications with various technology stacks. I know enough of those technologies to be productive. I learned the best practices and learned when to apply them. I tried to stay on the cutting-edge and adopt new programming patterns early. However, I was never confident to say that I understand [insert state mangement library name here]. I know how to use them but I did not know why to use them. Until I started to work on viw.

For those of you who have never heard of viw, make sure to check it out, github.com/lpan/viw. It is a VI-like, terminal-based text editor written in C. I implemented Undo & Redo with an event-sourcing inspired algorithm and I applied the data-driven programming pattern. Feel free to read the source. A star would be greatly appreciated as well. :)

In this blog post, I want to discuss the lessons I learnt from implementing viw and how are they related to modern front end development. I also want to briefly talk about the Unidirectional UI pattern.

Introduction

It is all about working with constraints. In C, it is hard to find recipes for stuff that you want to do. Libraries and frameworks like Redux that force you to employ a particular design pattern simply do not exist. As a result, when I was working on viw, I was forced to make many seemingly trivial decisions on my own: when do I trigger an update to the UI?, how should I categorize those functions?, how should I name this file?. As I am adding more features to the project, I have to constantly refactor, change internal APIs and move stuff around.

Here are two big refactorings I have done:

Interestingly, as the project grows bigger and as I make more incremental adjustments, the application architecture ends up becoming something that is very similar to the modern unidirectional UI architecture. The app has an infinite loop that captures all the keyboard events. Depending on the current state of the editor, these keyboard events are mapped to a series of functions (I call them mutations) that make changes to the application state—the single source of truth. After all the mutations are done, It recalculates all the computed properties (eg. cursor position) based on the new state, and pass them to ncurses to render the new output on the terminal. Then, the application waits for the next keyboard event. Recently I added Undo & Redo functionality. Inspired by event sourcing, I refactored mutations such that each of them is modelled as a command that can be stored in a log. I can pop the log to accomplish undo and re-add the command back to the log to accomplish redo (the actual implementation is slightly more complex).

viw helped me understand what UI development is really about. It is not like programming a compiler which “simply” takes an input and spits out an output. When you are programming an UI, your app has to react to events. The events can be initiated from a user, a websocket subscription or a returning AJAX request. Then, according to the event as well as the current state of the UI, your app will produce a series of resulting effects to address the incoming event. In other words, UI programming is about mapping incoming events to a series of effects.

A Simple Example

Sounds confusing? Let’s walk through a concrete example.

Consider a simple Todo application. Our user is able to see a list of all the active todo entries as well as their total count. In addition, she is able to add new entries.

As front end developers, the first thing we should do when given a problem like this is to identify what the incoming events are.

User clicks on the “add todo” button.
User enters a character into the input field

Now we have identified all the incoming events, what’s next?

Remember

UI programming is about mapping incoming events to a series of effects.

To implement this simple Todo app, our goal is to map #1 and #2 to a series of effects that responds to them!

According to the specifications of our todo app, #1 should be mapped to

if the body of the input field is not empty

increment the total todo count.
draw the new todo entry on the UI.

if the body of the input field is empty

draw “PLEASE AT LEAST ENTER SOMETHING” with an angry emoji in red right below the input box.

See the conditional statement ;)? This is why I said

According to the event as well as the current state of the UI…

#2 should be mapped to

if the key pressed is a backspace and the input field is not empty

erase the last character in the input field

if the key pressed is a backspace and the input field is empty

do nothing

if the key pressed is a valid character

draw the character on the input field

As long as you have those two mappings implemented, you will get a working todo application.

What About the Unidirectional UI Pattern?

If you want to implement the todo app with imperative programming, you will model each of the programmatic effects as an impure function. In pseudo code, it will look something like this:

function handleAddTodo(todoText: String) {
  if (!todoText) {
    renderError!
  }
  
  else {
    incrementCounter!
    renderNewTodo!(todoText)
  }
}

function handleKeyboardEvent(c: char) {
  if (c == backspace) {
    deleteChar!
  }

  else {
    renderChar!(c)
  }
}

This is bad because:

Multiple sources of truth => will result in invalid UI states.
Hard to implement computed properties.
Hard to test (functions are not pure).

Unidirectional UI pattern is an attempt of data-driven programming in the UI world. Instead of making a series of effects right away in respond to the incoming events, we do data transformation first. Then we emit all the effects based on the newly transformed data (push effects to the edges). In other words, instead of do incrementCounter! and then do renderNewTodo!, we “mutate” a data structure (let’s call it the application state), then according to the new state, we emit all the effects.

// without a persistent data structure

state = {
  todos: ['eat pizza', 'drink water']
  error: nil
  newTodoField: 'I am a new todo'
}

function handleAddTodo(state) {
  if (!newTodoField) {
    state.error = 'Empty field!'
  }

  state.todos.push(state.newTodoField)
  state.newTodoField = nil

  return state;
}

function handleKeyboardEvent(state, c: char) {
  if (c == backspace) {
    if (!state.newTodoField) {
      return
    }
    state.newTodoField.pop()
  }

  else {
    state.newTodoField.push(c)
  }

  return state;
}

// effects
renderTodo!(state) {
  renderCount!(state.todos.length)
  renderTodos!(state.todos) // react is gonna take care of it LOL
  renderError!(state.error)
}

while (true) {
  event = getEvent // blocking
  if (event.type == AddTodoButtonClicked) {
    state = handleAddTodo(state)
  }

  else if (event.type == KeyPressed) {
    state = handleKeyboardEvent(state, event.payload)
  }

  renderTodo!(state)
}

With this pattern, we get the benefits of functional programming:

Pure functions (with persistent data structures) => easy unit tests.
Predictable states.
And more!

Conclusion

Unidirectional UI is an attempt to bring data-driven programming to the UI world. Disagree with me? Feel free to leave a comment below!

Edit 2017-08-29: This article has been discussed on Hacker News and r/programming. Thanks to @mxstbr and @wowamit for submitting it. :)

Thoughts on React and friends

Sat, 08 Jul 2017 00:00:00 +0000

Introduction

React truly revolutionized modern UI development. Instead of imperially mutating the View based on the newly arrived data, React enables us to declare the View as a function of state V = f(state). In React terms, we call those functions components and thanks to the nature of functions, we are able to compose them and eventually structure our complex UI as a Tree. Additionally, we can do neat things such as high order functions on our components (remember, components are functions!!), react-redux’s connect is a perfect example.

React

React claims itself to be a library but not a framework. I agree with this claim as all it does is providing an unopinionated API so that we can declare our View as as a function of State. However, it also comes with a state management API setState which, in my opinion, is an opinionated way of managing the application state.

It is totally feasible to build a relatively complex application without a state management library in React.

Let’s say we are making a task management application. We have a list of users and each user has a list of tasks. Using the above approach, we will store the entire state (root state) in the root component. Then, we pass the lists of tasks along with users to User components. Afterwards, we pass each individual task data to Task components. Task components will be the leaf nodes in our UI tree in this case.

There are a few problems with this approach. First of all, due to React’s functional nature, data flows in one direction. The root component needs to pass mutation functions down to their children. This will result in boilerplate and obfuscated code. Most importantly, the shape of your state tree has to match the shape of your UI tree. This constraint makes it difficult to model your application state. But our state is not always a tree, it is most likely a graph in real life. As for our task management application, let’s say we want to introduce an admin user who can delete other users’ tasks. We want to render a x next to a task if the current signed-in user is an admin. With React’s state management API, you will end up passing isAdmin down to every single Task component!

Redux

The biggest problem Redux (and many Flux libraries) solved is that you can model your application state tree independently from your UI tree. With mapStateToProps and selectors, you can connect a node or multiple nodes of the state tree to a node on your UI tree. In addition to that, Redux enforces a few rules which introduces a bunch of boilerplate but makes your app easier to reason about. To name a few,

Managed mutations: All mutations must be requested and it is applied one at a time. -> consistent
Reducers: New state is produced by a pure factory function. -> Predictable state -> no invalid state.
Immutability -> cool development tools

Relay/Apollo-client

Redux is cool but it does not provide a solution to handle the data fetching/caching layer. Developers usually have to develop a custom solution. It is pretty fun to check out how different people implement data fetching with React and Redux.

Graphql not only enables us to write descriptive queries but also unlocks the ability to compose smaller queries (through fragments). This allows individual components to declare the data it needs and Relay will automatically aggregate them into a single query so that we can fetch the data with one single network request. This seems to be unrelated to the state management problems Redux is trying to solve. But if you look at it closely, Relay allows us to not declare our state data tree explicitly. Relay is gonna figure out what does your state tree look like according to the queries declared by each of your components. That is pretty neat!

Conclusion

Single page apps make our users happy but our lives hard. I hope you enjoyed my two cents on the React ecosystem in 2017. Currently, I am fiddling with Datomic and om-next. IMO om-next’s query language is much simpler than Graphql (familiar data structure vs a DSL). I am planning to build a side project using those two technologies and hopefully I will be able to write a blog article about them in the near future.

A front-end developer's fantasy

Sun, 11 Jun 2017 00:00:00 +0000

A bit of background

It has been a while since I posted my last blog article. Work, gym and side projects were keeping me busy.

Anyway, I was walking back home earlier today and for some reason (possibly divine intervention), I started to ponder what is the proper way to design a front end application so that when it scales, it does not break conceptually (fancy phrase for ‘tech debt’ :wink:).

Real stuffs

You want your users to have a good experience so you decided to build (or migrate to) a SPA instead of rendering HTML pages on the server. As a trade off, you now have to deal with data fetching and caching (inb4 graphql), bundle size optimization (code splitting), SEO optimization, state management and more. Since many other peeps are scratching their head the same way as you are, many good, benevolent and not creepy people on the Internet developed cool libraries to make everyone’s life easier. As someone who is sane, you (and many others) started to use those libraries in your code base.

As a consequence, a typical front end stack went from simply jQuery and CSS to React (map data to DOM with minimum effort), Redux (‘finite’ state machine), Webpack (module bundler, so you can actually test your code), Babel (JavaScript transpiler thanks to IE and Windows 7 on your grandma’s computer), Yarn (inb4 npm 5.0) and many many more.

With the overwhelming number of libraries, it is hard to find experienced developers (you can’t find someone with 5 years of React.js experience when React.js was released 6 months ago) which makes it extremely easy to make design mistakes.

Here is the ‘ideal’ way of structuring/designing a front end application that is scalable and testable in my dream.

React

Functional components only.
For local UI state, use withState from recompose. One state at a time
Do not derive new props in render, everything should be done with selectors
Take advantage of the high order components pattern

Redux

Redux is a state management lib not a rest api cache management lib!
Avoid nested reducers.
Maximize the use of selectors

Reselect

Use Reselect plez so you won’t design terrible looking reducers
Group the selectors according to the reducers they are involved
- eg. selectors in the user directory does not mess with todoReducer
- If a selector touches multiple reducers, think hard. Then put it in rootReducers directory

Data fetching

I am still reading up on graphql :P, please don’t judge. I actually did a small experiment on graphql yesterday.

Keep track of the state of a request
Make sure each entity has an ID you can refer to
Relationships
- REST: join the entities then throw it into the cache
- Graphql: huehuehue you just throw it into the cache I guess

Conclusion

Thanks for reading! I hope you learnt something and had fun. On a side note, for those of you who studied electrical/computer engineering, I feel a front end SPA is literally a digital circuit. Side effects from the DOM that triggers state transitions are the inputs, Redux is the finite state machine, selectors/getters + React are the combinational logic that produces the output (UI) according to the current state.