Different organizations take different approaches to software development. And depending on which phase the organization is at, one approach might outweigh the other.

Let’s begin with a story.

Recently I came across a project that was at the crossroad, where we needed to choose between Economy of Scale or Speed to Market. I decided to pursue leveraging the internal team’s offerings to meet the long-term organizational goals and minimize technical debt.

Well… That failed and it came back to bite me and my team. Irrespective of the outcome, as I reflect on this project, I walked away with a few thoughts on assessing this very topic that I hope would help you (and my future self) would benefit when choosing between the economies of scale and the build-it-yourself approach.

The economies of scale allow teams to specialize in a specific domain.

What got you here isn’t going to get you there. As an organization scales, the domain expertise that a team needs to retain grows. This is the fundamental reason why teams get split and missions get formed: to solve atomic problems.

Specialization in a domain enables the team to go deep into an area and own the code, operation, and security aspects of the service. Expecting engineers to understand the entire organization and to be able to deal with everything often leads to inefficiencies.

Expecting the entire company to understand all the compliances that come with managing data such as PCI/PII/HIPAA can be extremely costly. This is where the specialized team has an edge; they become the experts in this area, and their customers just leverage their offerings and trust that these are handled accordingly.

All this doesn’t come for free, of course. The hidden costs can sway the pendulum to choosing the option of building software yourself.

As you start to work and depend on other teams, the alignment of priorities becomes crucial. What is important to you isn’t always important to them. You will need to convince them to also be invested. Existing platforms and services do not always work based on their documentation (if any), there is always a high likelihood that the APIs seem to work but don’t. Be ready and buffer for these.

The release schedule is no longer self-dependent. You cannot release a product where your critical dependency isn’t going live. More coordination is required and this is where the involvement of strong leadership becomes critical.

Bugs and defects can be problematic as well and it can lead to “tossing the hot potato” instead of dealing with the customer’s pain points and complaints.

Learnings

At a macro level, I believe that leveraging internal teams that specialize in a domain outweighs reinventing the wheel. This creates long-term leverage. A slight delay in a project in exchange to build a mechanism to work yields a far better outcome in the long run.

However…

There are still a few things to assess:

1. Does your organization value doing it right or doing it fast?
2. Does the team understand the benefit of economies of scale?
3. Is the culture ready to shift to this particular model?

If you don’t have a strong conviction for all the questions above, it would be an indication that the organization is not yet ready to fully adopt, irrespective of how the organization is shaped.

One of the parting thoughts that I want to share is to have the courage of cutting your losses. In the example above, we should’ve cut the losses and built it ourselves, as the organization was clearly not ready. While the project failed, we succeeded in identifying the maturity level of the organization.

I hope this was helpful in providing some of my own thoughts on how to evaluate tradeoffs.

Let’s begin with a story. A story that truly depicts what infrastructure teams have to work with, and one that hopefully illustrates how much of a rollercoaster ride the journey itself presents.

A true story

On one of what-seems-to-be-a-normal-evening, right before Christmas of 2020, an on-call engineer on my team was paged due to an increase in latency on their production service. It was almost 8PM EST. As always, we attempted to dig through the reported issue and saw nothing out of the norm that would indicate a failure of our network stack or underlying infrastructure. In conjunction, the customer re-attempted to deploy their application, and the latency reduced (for now at least…).

In less than one hour, the symptom itself resurfaced, only this time, with more services being impacted.

I hopped on a call with my team and we began to slice and dice the logs and metrics we have on DataDog and we realized a particular service that was misbehaving, taking up all resources within the node (EC2 instance) and thus making all workloads deployed on it to be unresponsive (hence latency increase).

Now that we know the culprit, we paged that team that owned the suspected service and asked them to hop on the bridge. However, due to the late night, they were unable to push a fix out soon with the level of confidence that they needed. We (as the infra engineers) were stumped. What were we going to do? How are we going to tell our customers that their services are likely to experience a network failure due to another team that has, in most cases, nothing to do with them? From a customer standpoint, the impacted customers have deployed their services onto our infrastructure and they were experiencing a network degradation that impacts our end-users. This was definitely unacceptable.

Taking the matter into our own hands

This particular incident delineates a clear goal for us to be able to own our own destiny. We need to be able to control the overall health of our infrastructure regardless of who deployed what kind of workloads onto it without affecting one another.

The need to manage workloads across various tenants creates the need for multi-tenancy. Perhaps, you need to differentiate the access level for frontend applications vs backend applications; perhaps, the need is to grant teams with the elevated access to their own deployments and resources and theirs only.

However, multi-tenancy alone does not solve our incident. After all, as mentioned before, the underlying resources are shared (although this can be achieved via Resource Quotas). What we need is to have workloads to be dynamically moved from one node group to another, thus reducing the blast radius.

Kubernetes offers multiple declarative options for your deployments to deploy applications as a multi-tenant to multi-node group. This can be achieved via the combination of the following:

• namespaces
• taint and toleration
• node affinity and anti-affinity.

All these options do work, and they do work well. But they share a common downfall: they all need the owners of the Kubernetes manifest to be aware of the underlying infrastructure (they may depend on the creation of the namespace, or be aware of the node labels that are available to them etc.)

This also introduces an extreme infrastructure overhead for upgrading and managing the nodes as there the requirement to update the services’ Kubernetes manifests and wait for them to be deployed by the service teams.

Enter Kyverno

Kyverno is a configuration based policy engine that is built for Kubernetes. Based on the policies specified, it can mutate the Kubernetes resources on creation and updates.

Why is this so powerful?

The main premise that make this so desirable is the ability to have the policy created and maintained by us, the infrastructure engineers. We offer an execution substrate that works and that is always up, and we strongly believe that customers do not need to be involved. We should be able to migrate the workloads from one node group to another under the hood or we should be able to specify the types of EC2 instances that workloads should be deployed onto (CPU optimized vs memory optimized vs general purpose ones) etc.

Resolution to the incident

Luckily we were already in the process of adopting Kyverno, so we immediately provisioned a new set of node groups to our clusters. We then created a policy that would deploy the problematic service onto that newly created node group. The mutation of the Kubernetes configuration actually happens under the hood, on the deployment event. Once the configuration was applied on our production clusters, all we had to do is add an label to the deployments of the problematic app and it was migrated off of the main node group. As soon as this was done, we have seen a drop in network latency.

One of the core traits for an infrastructure engineer is to be able to develop a set of toolkits within the toolbox. And this is one of them. Being able to decouple the infrastructure changes from the services allows a great flexibility and agility in making the infrastructure robust.

Over my career, I have always viewed myself as someone who would never consider the option to work from home. The thought of working remotely, distant from everyone, and the possibility of isolation, never came through to me as a viable option in a work setting. I need to see people. I need to talk to them. I need to collaborate with them.

This has been deeply ingrained in my mindset when I was an individual contributor, and it has further solidified as I transitioned into an Engineering Manager role. After all, isn’t management about people after all? How would I be able to be an effective leader if I worked distant from my team?

There have been multiple opportunities where I was able to jump-start on the Work from Home train a few years ago, and, at every crossroad, I chose to fall back onto the path of corporate offices. I was accustomed to it. It felt natural.

Fast forward to 2020, the year of COVID-19, most technology companies were forced to operate remotely. There I was, alone, working out of a very small studio apartment in Manhattan NY.

At the time, I believed that it was a temporary measure; I believed that we would’ve been back in the office in no time. But things worsened. New York entered a lock-down, and when I heard the border between the US and Canada was about to be closed, I embarked on a plane to make it back home, to Vancouver BC where I can be with my family.

I consider myself fortunate. I work in the technology space, where working from home is an actual option during the pandemic. As long as I had a laptop with electricity and internet, I would be able to perform my job effectively. That being said, working from home, especially in the long run, isn’t the same as working from home occasionally. Many challenges arise.

Challenges

Timezone Differences

As I came back to the west coast, I was distant from the team members who were (at the time) all on the east coast. To minimize the impact on them, I began working on the east coast hours, where I started my days at 7 AM PT and end at 3 PM PT.

This was not a smooth transition as one would probably imagine. Waking up early was not my forte, especially not when the alarm rings at 6:30 AM sharp. However, after a year of undergoing this particular schedule, I have actually grown to be someone who appreciates the quietness of the morning and eventually became a morning person. By noon (pacific time), 3/4 of my workday has elapsed; and having the day end between 2 PM and 3 PM felt awesome. This gave me the time to run errands, exercise, take walks and spend more quality time with my family.

Office Space

Working from home full time requires a comfortable table and chair. While nothing prevents anyone from working out of your couch, working long hours in a non-ergonomic setting does impact your overall health.

To set up my workspace for the long run, I purchased a monitor to function as my default screen, giving more screen real estate to do my work, split-screen etc.

I also purchased a standing desk so that I can strike a good balance between sitting and standing. Sitting all day long has long been a source of unhealthy living which leads to unproductivity.

Communication

As we became remote, the number of meetings began to increase, largely to compensate for the traditional in-person conversations. The “watercooler” discussions were replaced by running daily “virtual coffee” meetings, where we would just find 15–30 mins to relax and chat about non-work-related topics. Of course, we pivoted and adjusted as people got more accustomed to the work from the home situation.

At Compass, and much like other companies, we leverage Slack as the primary IM tool to communicate with one another. Instead of being able to turn around and clarify your thoughts with your peer, you now are required to communicate your thoughts more effectively through writing. We however do hop on calls for discussions and regular 1–1s; but we ensure that any decisions are documented via writing.

Technical designs have been forced to be reviewed asynchronously, thus requiring them to be more concise and digestible to a wider audience. This has truly elevated the depth of our documentations, and we have defaulted as a team to communicate more effectively through writing.

Light along the path

Working from home isn’t necessarily for everyone. It definitely wasn’t something that I would ever think to enjoy. However, I have grown to appreciate the positive aspects that come along with it.

• Being able to be close to family
• Become a better communicator through writing.
• Transformed into a morning person.

The ability to do the work that you enjoy with the leisure of being comfortable at home, and not having to travel to work, dramatically improved my efficiency and productivity. While what’s happening in the world is not necessarily worth cheering for, I am seeing and appreciate the opportunity that Covid has forcibly introduced to my life.

I now secretly wish we never return to the office, at least not full time.

Today, close to 97% of Compass’s applications and services are now running on our Kubernetes infrastructure. With Kubernetes, we have been able to achieve an exponential increase in the deployment frequency and successfully reduced the suspense that comes with deployments to production.

Truth be told, the journey itself has not been as smooth as one would hope for. I mean, after all, what can go wrong when we use managed Kubernetes (EKS)? Turns out, a lot can go wrong.

Our backend gRPC services leverage Istio as the service mesh. Hence, each one of our running pods would have an istio-proxy sidecar injected into it. This is achieved via the istio-sidecar-injector component.

One of the most vivid incidents that we have encountered (fortunately on our test environment) was when an application deployed onto Kubernetes took our pre-production environment down.

On October 14, 2020, we received reports that deployments to the Kubernetes backend cluster were failing and pods were no longer responding to incoming gRPC requests.

Immediately, the team began the investigation in an attempt to resolve it. With a simple diagnosis, we concluded that the deployments failed because the istio-sidecar-injector ended in a CrashLoop state. We manually restarted the pod, which forced the pod to be recreated in a different node, and the incident seemingly recovered. This was done to give us sufficient breathing room for proper investigation whilst unblocking our customers.

However, in less than an hour (or so), the issue itself regressed. This time, impacting multiple services and latency in our test environment spiked up. After deeper investigation, we realized that one of our nodes entered a NotReady state. We immediately cordoned the node to make it no longer schedulable, and we decided to SSH into the instances to see what was happening.

After looking at the kubelet logs, we noticed that there was an error emitted.

Oct 15 00:08:36 ip-00–01–2–345.ec2.internal kubelet[5195]: runtime: failed to create new OS thread (have 2 already; errno=11)

Oct 15 00:08:36 ip-00–01–2–345.ec2.internal kubelet[5195]: runtime: may need to increase max user processes (ulimit -u)

How can we run out of threads at the OS level? That made no sense to us.

Since we use DataDog for monitoring, we began to slice and dice potential data that can provide leads, from memory to CPU to load, and we noticed an increase in the docker.thread.count metric for one of the services.

After debugging the code for the suspected service, we realized that the service itself had a dependency on Kafka which leverages AWS Security Group for permission. This particular service was recently migrated to this cluster, and the service team forgot to update the Kafka Security Group. However, what we found out was that, instead of handling the connection error gracefully, this service, in turn, repeatedly spawns additional threads to reconnect, hence resulting in exhaustion of threads available on the node.

The immediate resolution was to terminate the pods for this service (and route the traffic back to the original cluster), and we saw an immediate recovery. We later worked with the team to add the new Security Group to the Kafka instance.

Learnings

Never assume that because applications are running in their own containers and deployed independently they would not interfere with one another. After all, they share the same underlying resources (CPU/Memory, thread pool etc.), and as an infrastructure engineer, we need to have the right tooling in our toolbox to react to those external variables.

We need to have observability in the entire stack of our infrastructure. Shortly after this incident, we built our Prometheus monitoring stack that now exposes the metrics via Grafana for Kubelet, Kube Proxy, CNI and the node itself.

As an action item from this incident’s postmortem, we began to incorporate this particular scenario as part of our test suite. We built a test app that reproduces the same behaviour as the problematic service, and we updated our configuration for Kubelet with the following parameter:

--pod-max-pids=5000

The Pod max PIDs limit allows the definition for the maximum number of process IDs that a single pod can use. This was eventually patched on each of the EC2 instances and we verified that this issue would no longer occur.

As one of the fastest-growing technology companies, Compass is on a mission to build a modern end-to-end real estate platform that integrates agents, buyers, and sellers through technology.

If you are interested in joining us to making huge impacts, and you are excited by the challenges of infrastructure, we’d love for you to join us.

How an application took down our Kubernetes cluster. was originally published in Compass True North on Medium, where people are continuing the conversation by highlighting and responding to this story.

The way our brain learns and adapts is generally through a series of events that happen throughout our journey, from burning ourselves touching something hot to standing back up once we fall. After each event, we seldom remember the cause (burnt) but rather remember the underlying effect and impact (not to touch hot material again).

However, during interviews, we often put emphasis on the following question:

What is your biggest failure and what have you learned from it?

This is one of my most disliked questions during an interview.

Biggest? In what terms? There can be small failures that have yielded deep learning, whilst the opposite is equally true. I am truly skeptical that this question on its own would be able to allow any interview panel to form an objective opinion on hiring, not to mention that this question can easily be rehearsed prior to the interview, which furthermore obfuscates and clouds our judgements.

No situations are ever identical. Therefore, I have found it way more valuable to ask the following question:

How would you handle this particular situation?

We, as humans, are built on top of prior experiences. By giving a situation/behavioural question, candidates would be able to showcase their toolkit that they have acquired and engage in a much more meaningful, on the spot, conversation with the interview panel.

I fundamentally believe that people that retain the learnings from failures don’t dwell on the failure itself. They would convert the learning into one of the many tools that can be re-leverage for future situations. The cause itself is not truly worth remembering, while the outcome, however, is.

Forgetting is an Art. Being able to look back on how you’ve gone your way of becoming a stronger and well-rounded individual, in my opinion, is the true indication that someone has gown and taken the learning with them.

After living for over 20 years in Canada, I recently decided to relocate from Vancouver to New York, USA.

Many of my friends sounded surprised as I announced to them that my wife and I were going to experience a new adventure, something that we haven’t really done before.

A little back story. In 2016, my wife and I travelled to the other side of the world to visit Australia. Over the course of the three weeks of travel, we fell in love with the country. Before returning to Canada, I asked my wife a question that I have never asked before: “Would this be a place that you can see us living?”.

About a year ago, I stumbled upon an opportunity at a company that is actually headquartered in Sydney Australia. Unfortunately, I declined it. I came up with excuses: my wife had just changed jobs, my family is going to be far, compensation wasn’t great, etc.

The truth is that I was scared. I was scared of a change in my life. I became complacent about my life. Too comfortable.

Though, since I have rejected the offer, I often thought about how would our lives have changed if we had decided to relocate. The constant feeling of “what-if” left me hanging and imagining as to how my life would’ve been otherwise.

Fast forward to 2019, a few recruiters reached out to me for different opportunities in Vancouver and externally to Vancouver. Truth be told, I haven’t felt too excited with the Tech vibe in Vancouver (another article specifically about this).

As I was discussing with various companies that were down in Silicon Valley, a recruiter that is based out of New York reached out to me about an opportunity at a company named Compass. As we were discussing the role, company and embarking in the interview process, we eventually ended in the final stage of the on-site interview.

During this interview, my wife and I came to New York for the first time and we spend time to explore the city from a no-so-tourist lense, envisioning if we could see ourselves living here for the next couple of years.

During the short, but eventful weekend, we learned lots about Manhattan. We saw the sheer amount of people that were walking on the streets; we understood why this city is considered a city that never sleeps; above all, we felt a sense of vibe within this city. A vibe that I have not felt back home. This energic vibe is what ultimately drove us into this city.

Is this the right choice? Honestly, it’s too early to determine that. Whether the choice itself is right or wrong is beyond the point. The outcome is that this experience is invaluable: the ability for us to take a leap of faith and experiencing something new is, on its own, rewarding. So yes, ultimately, the decision will be right.

In the commencement speech at Stanford University, Steve Jobs talks about connecting the dots. This quote has since been close to my heart. Taking a leap of faith, trusting that dots will connect when looking backwards, has been a tool that allows me to get a sense of distress, trusting that things will eventually work out.

About two years ago, I transitioned from a Senior Engineer to an Engineering Manager role. I was excited and I was ready. I was ready to make a big impact, and be a different manager.

Or so I thought.

I still remembered that, as I looked at the announcement that my manager sent out about my promotion, I expected to get some sort of training or a guide on the dos and don’ts as a manager. None of that happened. I felt that I was left hanging.

The feeling of excitement quickly faded away and was replaced with fear and anxiety. Fear to be caught that I wasn’t the right person; fear to let my manager disappointed about having promoted me; fear to let my peers down. I got what I later learnt as imposter syndrome.

I eventually overcame the imposter syndrome by ignoring the daunting thought as to what a manager should do but rather focusing on what I can deliver to the team.

Focusing on what I can deliver to the team and the organization is how I have allowed myself to start growing and learning again, now as a manager. Being a beginner again.

Things I have learned

Your peers will look at you differently.

Once I got promoted, I quickly noticed that people looked at me differently. Everything that you have done, everything that you are doing and everything that you will do are now going to be viewed differently.

You cannot continue to be an Individual Contributor.

When I started this journey, I envisioned that I would be different, that I would be able to keep coding while still doing my job as an engineering manager. It didn’t take long to realize that I was spread too thin.

Meetings creep in and I had no time to properly test the code that I wrote. I had to let coding go in order to give myself breathing room and give the team space and accountability.

You need to trust and rely on your team.

As a manager, you might want to do everything and often feel that you should know everything. However, this urge does not help when you want to scale out. You need to trust and rely on your team and not be afraid to ask for help. Being able to trust your team to deliver is key and takes time as a new engineering manager.

Reflecting back on the two years, I have learned tremendously. I am thankful to my manager who took a leap of faith in me and who believed that I would excel in this role.

To those that share the same imposter syndrome, I leave with this thought: choose to believe and trust yourself. You are in the role because you belong there.