An API success rate of 100% at the backend API gateway doesn’t mean anything if the customer success rate as perceived when interacting with an app is 0%.
Said another way, the only success rate that matters is the success rate measured where the user experience actually is, all the way at the mobile edge. I cannot count how many times in my career a backend API has returned an HTTP 200 OK only to have the response crash the app!
The same reasons that we’re obsessed with observability for servers obviously apply (perhaps even moreso) to mobile. And given the clear importance of mobile observability in operating reliable customer experiences for our applications, it seems like the topic should be widely discussed and widely applied, yet it’s not. For the rest of this post I’m going to talk about why this is the case (teaser: it’s very difficult!), and what we can do as an industry to prioritize its implementation in a cost effective manner that ultimately improves customer experiences.
Before diving into mobile observability, let’s start by talking about why mobile development is difficult:
Unlike server where most engineers are used to being able to deploy nearly instantaneously to their entire fleet, mobile deployment is a very lengthy process that starts with submitting new app versions to the app store, waiting for it to (maybe) be approved, and then waiting for gradual rollout. More established apps also likely have processes for alpha, beta, etc. before starting production rollout. All of this means that it might take 4 weeks or more before code changes can actually wind up in users’ hands, leading to excruciatingly slow iteration cycles. Additionally, because there is typically a long tail of slow upgrades, we are forced to support old app versions for a very long time.
While new smartphones have quite a bit of power, they still have a finite amount of CPU, RAM, and disk that is being shared across all apps on the device. The smartphone OSs provide very limited control around when apps get killed due to excessive CPU/RAM usage (where the definition of “excessive” can change at any point due to low battery, “doze” mode, the CPU overheating, etc.), and apps can be interrupted at any point due to receiving a call/text, user initiated context switch, etc. Finally, the OS makes no guarantees about disk availability which means that apps have to be very defensive about any assumed local storage. With server, you own the device and can control everything that’s happening on it; with mobile, you don’t.
Related to the previous point, mobile OSs make no guarantee about whether an app can run in the background. This means that an app may be shutdown and not run again for days or weeks depending on when the user next interacts with it explicitly.
Unlike server where we rarely think about the underlying hardware and Linux version (other than perhaps whether we need to support ARM), apps must routinely support old OS versions and old phone models where supported APIs and capabilities may materially differ. This can lead to complex branching and feature enablement depending on detected capabilities.
Both Apple and Google are continuously evolving their posture on privacy and security. This means that apps need to support all branches of requested permissions and not all models and OS versions support all capabilities.
Users are likely to have different languages, locales, time zones, calendars, etc.
While the internet is fantastically reliable (all things considered), mobile apps have to contend with a wide range of poor networking conditions including:
Additionally, even when things are working, bandwidth is limited and shared across the entire device. Furthermore, not all users have unlimited data (this is vastly more common in developing countries) and will care about how much data apps are using because they are being charged by the byte.
Clearly, mobile development has many unique challenges that are completely foreign to the average backend engineer, even when the backend engineer works on a distributed system that ultimately is in support of a widely deployed mobile first product! Not surprisingly, these challenges directly affect the implementation of observability in the mobile environment.
In the vast majority of cases server engineers assume that connectivity between distributed system nodes is constant and largely consistent and reliable. This is not to say that server engineers do not have to handle sporadic failure in distributed system networking (they clearly do). However, for the most part, modern observability systems attempt to push metric/log/trace data through the pipeline with a limited amount of retries and local buffering in either RAM or on disk. The assumption is that the data will probably get where it needs to go. And if the retries or local buffering are not sufficient during sporadic failure, the data is dropped. Because server networks are generally extremely reliable, sporadic drops are rarely considered to be a large issue. To summarize, in the server observability world we assume:
Not surprisingly, the high level assumptions we make in the server observability world are at odds with the challenges inherent in mobile development and implementing mobile observability.
Due to the epicly long release cycles inherent in mobile, the simple act of adding a log or analytic event can take weeks for initial rollout and months/years to reach the entire population of app users. This means that all things being equal mobile engineers would like to add as much logging/analytics as possible ahead of time in case the data is needed. However this is unfortunately at odds with the fact that:
Users that are on metered data plans care deeply about how much data an app is using, and will notice if an app is using “too much.” Furthermore, even if users have unlimited data, slow networks and sporadic connectivity means that observability data is sharing available bandwidth with critical application traffic. I.e., sending too many logs can materially impact the performance of the main application. Note that nearly every large mobile app has measured that perceived application performance (time to first interaction, delay when moving screens and clicking buttons, etc.) impacts conversion and core business metrics so this is not an academic concern.
Unlike on server, sporadic or poor connectivity on mobile must be considered a normal event and not an outlier. This means that observability systems must plan ahead for sophisticated batching, local storage, etc. or enough data will be lost to impact the reliability of the observation system.
Because apps can be terminated at any time and may not be able to run in the background, mobile observability systems must plan for this case, similar to how sporadic connectivity might be dealt with.
Depending on the app, a relatively small server infrastructure may support hundreds of thousands or even millions of monthly active mobile users. Storing individual telemetry for each active user is very costly so aggregation and summarization systems are typically needed for mobile observability.
Some interesting mobile telemetry may require explicit permission opt-in (for example location tracking). App developers need to think carefully about whether they want to request permissions from a user and either way have to handle the case where a user denied the permission request. This makes data analysis tricky as the analysis has to account for data that a user refused to send.
While mobile apps can be multithreaded, in general a single “main” thread is used for drawing the UI and handling user input events. Care must be taken to not block the main thread unnecessarily or the app’s frame rate and general responsiveness may suffer. This is particularly important for local storage of observability data which needs to be durably stored due to several of the previously described issues. Thus, the observability implementation needs to carefully consider how logging data is processed and stored so as to not adversely impact app performance.
I will also mention that the idea that logging in production in mobile apps is “bad” (due to network utilization, blocking the main thread, etc.) is pervasive. Some examples include:
There are clearly a lot of challenges stacked against mobile observability that we need to solve in order to get it widely deployed so that we can accurately measure real user experience.
Given how important mobile observability is for measuring and debugging customer experiences where they ultimately matter, it is in our best interest as an industry to invest more heavily in fundamental technologies that solve for all of the challenges described above.
In a previous post I argued for not storing any telemetry by default, adding a control plane, and adding sophisticated local storage. The goal of this system is to reduce cost overall, but still allow for highly detailed telemetry to be accessed when needed in order to debug customer issues: 1000x the telemetry at 0.01x the cost. Is this type of observability system a potential path forward for mobile observability? I think it is. Adding a control plane and distributed local storage solves the challenge of mobile observability in multiple ways:
If anything, the technical and cost challenges inherent in effective mobile observability most obviously necessitate a drastic change in how we think about observing systems in general via adding a control plane and distributed local storage. (Perhaps not surprisingly this is why bitdrift’s initial focus has been on mobile observability, though I think adding a control plane and local storage is broadly applicable to the entire observability ecosystem.)
I look forward to the day when identifying and diagnosing mobile-centric issues — like a server change causing crashes in older app versions due to malformed JSON in HTTP 200 responses, becomes as simple as it is with the server-side observability systems of today. If as an industry we prioritize the importance and implementation of effective mobile observability (the “final frontier” as it were), this day may be just around the corner. Onward!
]]>In this post, I’m going to take you through a journey where we explore a world in which observability is flipped on its head: we store nothing by default, but have access to anything we would ever want on the fly. What does life look like when you can get any telemetry you want, any time, without thinking about cost or storage? What would it mean for our ability to debug and operate systems?
The sad, or maybe just ugly, reality is that the vast majority of telemetry in traditional systems is never read, either by human or machine. Even as storage, compute, and network have gotten cheaper over the years, the complexity of our systems (microservices, CaaS, FaaS, etc.) and their ability to generate large volumes of telemetry data has kept pace, leading to egregious waste, high relative costs, and unhappy operators.
In my previous post, I argued that we can apply a dynamic control plane, plus use local storage, to intelligently decide what telemetry to store and use. If we conservatively estimate that overall monthly telemetry volumes and costs can be decreased by 90% using a fully dynamic control plane and local storage (I think the real volume decrease could approach 99% or more in some systems), we now have slack to reuse some of this excess capacity for more telemetry when it matters. Effectively we are trading a steady state of dubiously useful telemetry for bursts of extremely high fidelity telemetry that are targeted towards solving specific problems at hand.
For the rest of this post I am going to go into some specific examples across both mobile and server observability where getting bursts of 1000x the telemetry will fundamentally change how engineers debug and operate systems. These examples are by no means exhaustive and I fully expect that once dynamic telemetry systems are widely deployed we will continue to see new use cases, limited only by the imagination of engineers around the world.
Before talking about different examples of what 1000x the data can do, let’s first discuss the definition of cost in the telemetry context.
As an aside, on the topic of overhead I recall one particular bug I was chasing early in my career inside a mobile phone cellular stack – if we turned on all logging the bug wouldn’t repro – we had to add a single log line at a time and spend hours trying to repro the issue! The problem ended up being a super rare race condition deep in the telephony stack which was tickled by a particular phone model on a particular network carrier. What a fun bug!
For the remainder of the post when I talk about cost I am talking about all four: the financial cost, the performance overhead of telemetry collection, the cognitive cost of having too much irrelevant data to sift through, and the opportunity cost of not being able to get all of the data we need. Adding a control plane and local storage allows us to amortize the cost of extremely high fidelity telemetry collection across short collection intervals, large populations, and very specific collection triggers (finite state machine matchers sent from control plane to data plane), thus making all four costs low.
Nearly every logging/event framework in existence has a way to categorize logs into relative levels of severity. For example, error, warn, info, debug, and trace. While there is no standard on how to categorize logs, and no standard on which severity level of logs to emit by default to traditional observability systems, most engineers have some rubric that they follow, typically based on personal experience and/or company culture.
In order to reduce volume and cost, most organizations emit info and higher severity logs to be stored by a traditional log storage system. Because of this, most engineers have been trained to categorize the severity of their logs, and are generally told to be judicious about info and higher severity, while debug/trace have no restrictions due to not being sent anywhere by default.
Thus, it is an extremely common occurrence that info logs do not have sufficient data to fully understand the path of execution that a program took prior to arriving at the info log in question. The number of times in my career that I have wished to easily see debug/trace logs on a production system are innumerable!
Adding a control plane and local storage on the other hand allows us to dynamically enable debug and trace logs for a limited period of time (and in response to a specific sequence of events) and get access to the entire volume of data that is invaluable in understanding the full set of events that led up to a particular application event.
While metrics are largely considered to be more cost efficient than logs, they still can create enormous bills in large organizations. The largest organizations typically perform metric aggregation, essentially taking a set of related metrics and collapsing them into a single aggregate time series, in order to both reduce cost and increase query performance (since many queries typically end up asking for an aggregate average, max, etc. on the many underlying time series). A concrete example of this approach would be taking a counter generated by all Kubernetes pods in a service, dropping the pod label, and aggregating the remainder into a per-service count.
Using this technique can drop overall metric volume by 1-2 orders of magnitude depending on the deployment. Most users don’t notice any difference because as I mentioned above, queries tend to ask for aggregates anyway of the per-pod data.
However, there are cases in which having per-instance (per-pod) metrics is useful during debugging. For example, looking for outliers that are easy to lose in a large batch of aggregate data. A control plane driven observability architecture makes it trivial to temporarily turn on per-instance metrics, and even stream them to a live viewer if live debugging is all that is needed. This avoids permanently storing orders of magnitude more metrics when they are rarely needed.
Networking is pervasive in modern systems, both within server-side infrastructure as well as spanning the internet out to large fleets of mobile/IoT/web devices. Due to the highly concurrent and inherently fallible nature of networking, debugging problems that crop up can be very difficult without detailed tracing. Example low level protocols used in modern distributed systems include: IP, TCP, UDP, QUIC, and TLS. High level protocols include HTTP, gRPC, and many others specific to individual databases, caches, and so on.
Similar to logging severity levels discussed in the previous section, it is possible to emit variable levels of network protocol tracing, up to and including “trace” logging which might include the actual network payloads to greatly aid debugging.
Clearly, continuously sending extremely detailed network payload information is infeasible due to overhead and cost, but if it could be accessed on-demand when needed imagine how much simpler it would be to debug hard to understand issues!
Having worked in the application networking world for many years, I cannot count the number of times I have been asked to help capture HTTP REST/gRPC request and response bodies to aid debugging. A control plane and local storage makes this easily possible at a reasonable cost.
As an aside, collecting deeply detailed telemetry, especially of this nature, has substantial security implications. For example, network payloads are very likely to have PII, credit card numbers, passwords, etc. Future highly dynamic observability systems will have to very carefully consider data access security and encryption of telemetry data. One nice property of the control plane / data plane split is that some cases can be solved by matching only within the data plane and not capturing any data at all with its inherent security implications (for example emitting a synthetic metric when a match condition happens). This will be the topic of a future post!
Using software technologies like eBPF or functionality built in to hardware itself, it’s possible to produce extremely high volume tracing data to aid in problem analysis. Like networking data, it’s not practical to send this level of data all the time, but when required for in-depth analysis of hard to reproduce problems, being able to dynamically enable hardware level tracing for a short period of time can be invaluable in root cause analysis. Note that this category applies to all types of high volume hardware signals including instruction traces, accelerometer readings, GPS/location readings, virtual reality camera/sensor recordings, etc.
Similar to hardware instruction/sensor tracing, in-depth profiling using tools like Linux perf can be invaluable in understanding performance bottlenecks during application execution. However, collecting performance profiles has non-zero overhead and also generates large volumes of data which are impractical to collect. Modern continuous profiling tools get around this limitation by using sampling which is a fantastic approach but can make it difficult to drill down into very specific cases and get accurate real-time data specific to a particular program sequence. By allowing the observability control plane to engage the profiler when a specific sequence of events occurs, very accurate performance data can be retrieved on demand and at low overall cost.
Session replay is generally defined as recording a user’s journey through a mobile or web application. Some variant of session replay is included in many different Real User Monitoring (RUM) and general observability tools. Typical session replays capture some variant of the following UX data:
This is in addition to general observability data capture including logs, networking events, etc.
The important thing to understand about session replay is that cost increases linearly along with the amount of data stored (the fidelity). For example, recording screen wire-frames is far more efficient (and privacy conscious) than doing pixel perfect recordings, but has less overall fidelity. Similarly, the more additional UX data is captured (up to and including phone gyroscope readings) the higher the cost both in terms of data volume and capture overhead.
Because of this, especially on mobile, very high fidelity session replay is rarely deployed to production due to overall cost. However, when using a control plane and local storage it becomes possible to on-demand capture very high fidelity data when a specific sequence of events occurs, making it substantially easier to understand real user journeys even for applications that are deployed to millions of active users.
In this post I’ve covered a small number of cases where using a control plane and local storage to dynamically enable very high fidelity telemetry can aid in root cause analysis of customer issues. Fundamentally, this type of system provides a massively better ROI on telemetry cost (both financial, overhead, and cognitive), because the generated data is highly detailed when needed and absent when not. 1000x telemetry at 0.01x the cost sounds too good to be true, but in the future I firmly believe that we will look back on traditional observability systems and wonder how we were ever able to debug anything!
At bitdrift we are just beginning to scratch the surface of what is possible when reimagining observability with a control plane / data plane split, and I can’t wait to see what other systems and use cases the industry comes up with collectively over the coming years!
]]>In this post I’m going to talk specifically about the perceived need of storing all possible telemetry data. Is this a real need? Or is this an idea that has been drilled into us for so long that we think there’s no other possible way of doing it? If we can break away from the disconcerting idea that all possible telemetry is not stored by default, what new paradigms and cost models for observing and debugging systems become possible?
Before diving into the question of whether we really need to store telemetry, it’s first important to break down the reasons why we do store telemetry. Telemetry emission can be generally placed into three discrete categories:
For the purpose of the remainder of this post I will only consider the first category. Telemetry required for compliance, auditing, or business intelligence is very important though out of scope due to its required storage and is thus exempt from the thought exercise: can we disable everything by default?
When presented with the idea of disabling all telemetry by default, the most common skeptical response is: “What if I need to debug a problem that just happened?” This is a perfectly reasonable response and I admit that the idea of disabling all telemetry by default is likely to be deeply disconcerting to some.
Let’s start by breaking down how telemetry helps with observability of systems:
In my personal experience as an engineer working on very large scale systems for nearly 25 years, I understand the fear of disabling telemetry by default, but I also believe that there is no problem that happens once and never happens again. This is the critical point. If problems are not truly unique and are likely to happen again, can’t we take steps to catch the issue the next time around with targeted telemetry specific to that problem? Said another way, once a problem is known, can we instruct the system to collect additional information and provide it to us the next time the problem occurs?
One might then naturally ask, “if I disable all telemetry by default how would I know about problems in the first place?” While the terms APM, monitoring, observability, etc. have become muddled due to vendor marketing, it is important to differentiate the act of “monitoring” a system from the act of debugging. The ideas presented in this post do not aim to change the methods of “monitoring” for problems: we are notified of problems via customer complaints, alert conditions on metrics or event counts (that are sent all the time so that we can run alert queries), in newer systems possibly anomaly detection, and so on. A different way of thinking about “off by default” is that all signals become opt-in vs. opt-out. I.e., the data needed to run an alert query would be opted-in by default. This post is talking about what happens after we are notified of a problem.
I admit that the scale of the systems that I have worked on have strongly shaped my views on this topic. Between working on high performance computing in AWS EC2, to building Twitter’s edge proxy, to creating Envoy at Lyft, the order of magnitude of requests per second across the entire system and on each node has always meant that capturing detailed logging by default was never practical purely from an overhead perspective. (By default, Envoy logs at Info level and at that level emits nearly no logs after startup, a pattern I have come to use on all systems that I build, for better or worse.)
Instead, I have relied on metrics for large scale debugging and monitoring of these systems. While this has been a successful approach overall (Envoy after all is run by thousands of organizations and is considered a highly reliable piece of software), it’s not without issues. Taking Envoy as specific example (even though the same idea applies to other systems I have worked on):
On the topic of the prolonged debugging process, what does that look like using traditional telemetry production techniques?
Thankfully (3) above doesn’t happen often, but it does happen, necessitating painfully involved debugging experiences.
Building these types of systems over many years and debugging them has led me to think that there must be a more efficient way. And if there is a more efficient way for debugging and observing systems like Envoy, can those ideas be applied to systems at large in such a way to both improve the debugging and observation experience but also reduce cost along the way?
For 30 years how telemetry is produced has not changed: we define all of the data points that we need ahead of time and ship them out of the origin process, typically at large expense. If we apply the control plane / data plane split to observability telemetry production we can fundamentally change the status quo for the first time in three decades via getting real-time access to the data we need to debug without having to store and pay for all of it:
Beyond simply enabling/disabling/filtering at the source, we can add intelligence to our telemetry production points and do so much more. We can have the control plane send instructions on a specific sequence of events to match on (effectively a finite state machine sent from control plane to data plane), followed by a set of actions to take when that sequence of events occurs. Actions could be to dump extra information, take profiles, take screenshots on mobile apps, and so on.
Continuing with the Envoy example in the previous section, instead of having to painfully make code changes and deploy them over a long period of time, if all of the built-in metrics, debug, and trace logs could be controlled dynamically with very specific match conditions and actions, think about how much shorter the debugging cycle would be. If we assume that problems are never unique and will happen again, we can start doing iterative debugging in real-time against a fleet of running instances, and likely vastly reduce the time required to root cause an issue.
Another way of thinking about it is that this model makes debug information available with immediacy relative to the criticality of the issue as defined by the number of occurrences. Meaning, during an incident the wait time for recurrence will be close to zero, while the wait time might be longer for very uncommon things that are likely to be considered less critical.
This approach is powerful across multiple dimensions:
As long as operators are able to accept the idea that problems will occur again, with the right system they can catch them the next time around!
Along with real-time control, we can also add local storage of telemetry data in an efficient circular buffer. Typically, local storage is cheap and underutilized, allowing for “free” storage of a finite amount of historical data, that wraps automatically. Local storage provides the ability to “time travel” when a particular event is hit. How many times have we logged an error only to realize that it’s impossible to figure out the real source of the error without the preceding debug/trace logs?
A possibly obvious tradeoff of this system is that the lookback period is bounded, with the number of seconds of data available a function of the buffer size and the data rate. I still think that the benefits of the circular buffer in terms of cost efficiency outweigh the downsides of limited historical retention.
When coupled with the matching and actions described in the previous section, dumping the local buffer becomes a specific action to take, along with many other possible actions.
I will note that this is not a new idea: the Apollo spacecraft guidance computer had a system to store recently executed instructions in a circular buffer and dump them when a problem occurred to ease debugging. Similar debugging tools have been implemented for many years in embedded devices and other systems. Circular buffers have also been used in modern observability systems as part of centralized trace aggregation systems. The key difference is moving local storage all the way to the edge where it is easier to scale and coupling it with a control plane, which unlocks the ability to deploy dynamic queries across many targets that can result in history being dumped to ease debugging.
Imagining the combination of local storage and real-time control being used to debug Envoy is what started me down this entire path in the first place!
The paradigms used to emit telemetry have remained unchanged for many years. To this end, engineers are very used to sending data, and expecting it to be there in case they might need it.
Adding a control plane, local storage, and not sending any data by default is a drastic change to how engineers think about observability. Some engineers find these ideas deeply disconcerting, and fear that the data might be needed so it should be sent no matter what.
By starting from a default of no exported telemetry, engineers have the ability to “change the game” in multiple ways:
It’s true that changing how engineers and operators think about telemetry production and consumption is a tall order, but in my opinion the benefits of this change far outweigh the challenges of relearning fundamental techniques: namely, vastly decreased cost and vastly increased fidelity for helping to understand and fix product issues that are actively problematic.
I founded bitdrift last year with the singular purpose of moving away from 30 year old telemetry production paradigms and tackling this problem head on. Is it indeed possible to not send any telemetry by default and still successfully observe large scale production systems?
Since we launched Capture publicly a few months ago it’s been super interesting to start talking to potential customers; the responses we have received vary widely, ranging from those that immediately buy into the idea of off by default to those that are inherently skeptical, asking the same (completely reasonable!) question posed above: “But what if I really need to look at some of that data after the fact?”
I continue to be extremely excited by the idea of what we can build if we start from the position of accepting that bugs will happen again, not sending any telemetry by default, and proceeding from there.
So, say it with me: “No, I don’t really need to store all of that telemetry!” Or, “Just say no to the logging industrial complex!”
]]>Meanwhile vendors are tripping over each other trying to come up with the next great incremental innovation in pricing models and overall cost reduction techniques for storing logs, metrics, and traces. To be clear, there is some innovative and exciting work happening in areas like:
These techniques will possibly lead to better customer experience and lower bills over time. Yet, they are still a small iteration on the basic formula that we have been using for distributed system monitoring and observability for 30 years:
The critical thing to understand is that because the fundamental origin of observability data has not changed (engineers must determine what might be needed ahead of time and send it), there is only so much that can be done to bend the overall cost curve. Said another way, there is no free lunch in computing. Once data is emitted, every layer it passes through and its eventual storage incurs significant incremental cost.
The remainder of this post covers a (very) brief history of modern observability, how we arrived at the current observability cost crisis, and offers a new way to think about observability costs and pricing models: how would the computing world change if we turn 30 years of status quo on its head and by default never send any observability data?
Though engineers and operators have been debugging computer systems since “the beginning,” the origin of modern distributed systems observability can be traced to the 90s with the creation of the first major internet services such as Yahoo!, Amazon, and Google. The big difference with these systems versus what came before was both the complexity of the underlying distributed architecture as well as the expectation of 24/7 high reliability operation.
Engineers at these organizations pioneered (often independently) observability systems that look not all that different from what many of us are used to using today:
During the 00s as the internet and large scale distributed systems proliferated, other organizations such as Facebook, Twitter, Uber, etc. had to solve the same set of basic observability and monitoring problems, and developed similar parallel systems. The late 00s brought AWS and the beginning of the cloud era, and along with it came a new cohort of infrastructure software providers, a portion of which operated in the APM/observability/monitoring space, including pioneers like Splunk, New Relic, AppDynamics, and Datadog. Organizations implementing large internet systems found themselves no longer needing to reimplement basic observability primitives and could focus on other more urgent business concerns.
The 10s brought two major changes that directly lead to the observability cost crisis covered in the following section:
Another important innovation of the 10s was the commoditization of infrastructure building blocks via open source and open protocols. Prometheus, OpenCensus, OpenTracing, OpenTelemetry, Elasticsearch/Kibana, and Grafana were all introduced during this time period. As it became easier and easier to build large scale distributed systems via IaaS, PaaS, CaaS, and FaaS, the amount of telemetry emitted both by default from underlying systems and directly by engineers, and without regard to cost, increased exponentially.
By now it is hopefully clear what the real root cause of the current cost crisis in observability tooling is:
Observability vendors will gladly identify any number of other causes such as “It’s important and should be 30% of one’s budget!” and “Stop using metrics and just use events!”, but the fundamental reality as was said previously is that there is simply no free lunch: massive production of telemetry at the source, without regard for the usefulness of the data, will inevitably lead to high costs and unhappy end-users. Someone has to pay for all of that transport and storage.
I should mention that OpenTelemetry, while a great advance for end-users that decouples data production from transport and storage, does not in any way solve this problem. If anything it exaggerates the existing crisis by making it even easier to emit large amounts of pre-defined telemetry data.
So much about building large distributed systems has changed in the last 30 years. And while there is no dispute that observability tooling has gotten more feature rich and prettier to look at, the fundamentals have really not changed at all. We pre-define all of the data we might ever need, and then pay for the privilege of emitting the data and storing it. Even worse, because we are not completely ignorant of the cost or the performance implications of emitting large volumes of telemetry, we often still do not have what we need to debug issues and must make changes and deploy, sometimes multiple times.
But perhaps there is a better way.
Another major infrastructure innovation of the 10s is what might be called the “control plane data plane split.” Spearheaded by projects like Envoy, infrastructure concerns began to be split into two discrete components:
Historically, data plane configuration was notoriously hard to update, so it was largely static. Operators would define a configuration and then when updates were required go through a laborious and lengthy deployment process. With the rise of the control plane driven architecture, it became possible to update data plane configuration in near real-time, opening up a new age of dynamically adaptable infrastructure.
What if we were to apply the control plane / data plane split to observability? And taken to its extreme, what if the data plane is the original source of telemetry before moving outside the process to be filtered, transformed, and stored by sometimes many proxy layers, at substantial cost? What if by default we never send any telemetry at all?
I firmly believe that the end of ZIRP excess and the rise of control plane driven infrastructure is going to lead to a new golden age of large scale observability that spans both backend systems and the massive mobile/IoT/web networks they service. We are already beginning to see this in practice via more dynamic observability pipeline offerings (both open source and vendor driven), AI/ML driven observability control, and observability products built from the ground up starting with fully dynamic control over production (full disclosure I started one such company).
I am personally very excited to work on this problem and leave behind 30 year old observability data production paradigms. I can’t wait to see what we as an industry come up with next. Onward!
]]>During my 8 (!) years at Lyft we built some very impactful systems that enabled Lyft to weather its hypergrowth years. During this time we learned a lot of lessons that are broadly applicable across the industry. For the last year and a half I’ve had the opportunity at Lyft to develop a new business unit, aimed at commercializing some of the pieces that we felt would be most impactful, especially given the recent industry-wide renewed focus on infrastructure cost cutting. During this time we have made tremendous progress, including acquiring our first external customers.
By late last year it had become clear that in order for the fledgling opportunity to reach its full potential it would be necessary to spin out into an independent entity, so my co-founders Pete Morelli and Martin Conte Mac Donell and I set out to do that. After months of ups, downs, and close calls (it turns out spinning out a company is hard!), we are an independent entity named bitdrift. We are starting as a team of 9, with 6 folks from Lyft joining us on this journey: Jackson, Miguel, Rafal, Sarvar, Snow, and Vicky. This is a fantastic team, and I could not be more excited for what we and the new company are going to accomplish.
By now you are likely wondering: what does this company do? We will share much more soon, but we are going to stay in stealth a while longer while we work with more design partners and finish our go to market feature set. However, if you want a bit of a tease and a way to sign up to be notified when more info becomes available, check out our website at bitdrift.io.
While it is certainly bittersweet that my time at Lyft has come to an end - I will be forever grateful for the opportunity to build Envoy and share it with the world - I am beyond excited for this new chapter. More soon and onward!
]]>Except for a few small detours, my two decade career in the technology industry has been focused on low-level systems: embedded systems, operating systems, virtualization, filesystems, and most recently distributed system networking. My journey with distributed system networking began at Amazon in early 2010 where I had the fortunate opportunity to help develop the first high performance computing (HPC) EC2 instance types. I learned a tremendous amount about low-level high performance computer networking, though I had only limited exposure to distributed systems concepts.
In 2012 I joined Twitter, where after a few false starts I wound up on the edge networking team. This was my first real exposure to distributed system application networking concepts. I lead the development of a new HTTP edge proxy called the Twitter Streaming Aggregator (TSA), which was first launched in 2013 to scale delivery of Twitter’s “firehose” API (streaming all tweets). In the runup to the 2014 world cup, we decided to launch TSA as a general purpose HTTP/HTTP2/TLS edge proxy in points of presence (POPs) close to the event in Brazil. This was done primarily because it was not possible to deploy the existing resource hungry JVM based edged proxy on the small number of colocation racks that would be available in the POPs. My team delivered a successful and incident free world cup on an extremely compressed schedule. (I can fondly remember a period of time in which I would page myself when the software crashed, no matter the time of day, fix the bug, and redeploy the canary fleet to keep testing.) During my time at Twitter I also had exposure to the way the company performed service-to-service networking with great success via the Finagle library.
Around New Year’s Eve 2015, in what would begin the coda to my time at Twitter, TSA caused millions of Twitter Android users to get logged out via a bug I wrote:
Let me tell you about the time that I (indirectly) logged out about 40M Android users from @Twitter with a single character bug. They couldn't log back on for several hours, and the follow on impact on logged in users was ... not great. https://t.co/HeivgpUimN
— Matt Klein (@mattklein123) April 13, 2018
I left Twitter in the spring of 2015, partly due to the fallout from the logout incident, partly due to frustration about not being promoted, and partly due to a desire to try something new. I followed my boss from Twitter to Lyft, along with a bunch of my other Twitter coworkers.
When I joined Lyft, the company was relatively small (< 100 engineers), and was struggling with a migration from a monolithic to microservices architecture. I have talked about this portion of the Envoy journey many times, so I won’t rehash it again, but the very short summary is that Lyft was having all of the typical microservice migration problems, primarily rooted in networking and observability. Additionally, Lyft was already “polyglot” (using multiple languages and frameworks), so it seemed impractical to use a library based solution to solving these problems. Thus, based on my previous experience building TSA and observing how service-to-service communication worked at Twitter, and due to my immediate credibility via overlapping coworkers, I proposed building a new application networking system called “Lyft proxy.”
After some spirited discussion that included whether the new proxy should be built in Python (yes, really), we agreed on the broad outlines of the project and settled on using C++ as the implementation language. At the time C++ seemed the only reasonable choice. Would I choose C++ today? No. However, today is not early 2015, eons ago in the technology world.
This part of the history would not be complete without the origin of the name “Envoy.” We were setting up the initial devops scaffolding for the project when a forward thinking coworker (Ryan Lane) said that we couldn’t call this new project “Lyft proxy,” we had to pick something better. Always practical, I went to the thesaurus, looked up “proxy,” and settled on Envoy as the new name.
I didn’t start in earnest on the Envoy source code until the summer of 2015. Those months were some of the most fun of my career. Empty source files and no support burden should be treasured while they last, because they don’t last long. I worked long hours toward producing something that would add value to Lyft within a reasonable amount of time (by my definition 3-4 months for a project of this type.). Lyft had given me a tremendous amount of rope to hang myself with, as the saying goes, and I was committed to making sure said hanging did not happen.
Of course, my efficiency can mostly be attributed to being fresh off of the compressed development schedule and many mistakes (mostly my own) that went into TSA at Twitter. I knew what mistakes not to make, what abstractions were needed, what type of testing worked and what didn’t, etc.
The initial version of Envoy that was readied for production in the fall of 2015 contained just a tiny fraction of the functionality and sophistication that the project contains today. It did not support TLS termination, only supported HTTP/1, and had extremely simplistic routing and resilience features. What it did have was the bones of what you see today. There have been very few major refactors in the history of the project, primarily because, as I said previously, I knew what was coming and what abstractions needed to be in place in order to support the functionality. What Envoy did have from the very beginning was top notch observability output, in the form of metrics and logs. In 2021, this type of network observability is table stakes (thanks in large part to the success of Envoy), but it was not so at the time.
Envoy was first rolled out at Lyft as an edge proxy, sitting behind the AWS ELBs that were providing TLS termination. By late fall of 2015 Envoy was serving 100% of Lyft traffic, and the edge dashboards that were produced by the system paid dividends immediately (e.g., providing API call percentile latency histograms, per endpoint success rate and request rate, etc.).
Shortly after the initial launch, another Twitter coworker (Bill Gallagher) joined me on the project and we quickly added features, such as TLS termination, HTTP/2 support, more routing and load balancing functionality, etc.
At the same time, Lyft’s Envoy based “service mesh” started to take shape. First, Envoy was deployed next to the PHP monolith to replace HAProxy and some of its inherent operational issues (at the time HAProxy was still single threaded for example) to aid with MongoDB proxying. It would not be an exaggeration to say that a substantial portion of Envoy’s early development was targeted towards MongoDB stability (load balancing, rate limiting, observability, etc.).
The benefit of direct Envoy based observability between the edge fleet and the monolith was immediately obvious. Shortly after, we deployed Envoy next to some of the high RPS decomposed microservices to aid in troubleshooting networking issues. The value was proven there as well. Over time we expanded beyond an observability focus and added features to aid in system reliability such as direct connection and service discovery (skipping internal ELBs), outlier detection, health checking, retries, circuit breakers, etc. The number of load based major incidents at Lyft slowly decreased from a frequency of every 1-2 weeks to much less. Envoy cannot take credit for all of that decrease, of course, but the network abstractions it provided did help a substantial amount.
In early 2016, we decided to push for a service mesh with 100% coverage. Initially, we thought it was going to be a slog that would require top-down mandates. In practice, teams signed up to do migrations because the benefit they would get was evident. “Carrot” migrations are almost always successful. “Stick” migrations are rarely successful, or if they are, leave behind a trail of tears and anger within the organization.
By mid 2016 Envoy was used for all network communication at Lyft including edge serving, service-to-service communication, databases, external partners, etc. By any measure the project had been a resounding success, helping Lyft complete the microservice migration, increasing overall reliability, and abstracting the network such that most engineers did not need to know anything about the real system topology. Bill had since left the project to work on other things at Lyft, and in his place Roman Dzhabarov and Constance Caramanolis had joined me. Our small team developed and operated Envoy for all of Lyft.
By the summer of 2016 we started to have a serious discussion about open sourcing Envoy. Early Lyft employees had an appreciation for open source and what it had done for the company. It was clear that Envoy was not Lyft’s primary business, so why not put it out there and give back? I will be honest in saying that we all approached the open sourcing process with different goals and expectations, as well as a substantial amount of naivety around what would happen if the project became very successful.
Prior to Envoy, I had used quite a bit of open source, but I had almost no experience with open source contributions and zero maintainer experience. (I did have a single commit in the Linux kernel though!) Open sourcing Envoy seemed like a great opportunity to expand my skill set and learn something new, possibly further my career, and frankly, I had no desire for there to be a TSA v3 at a third company. For Lyft, Envoy was a substantial piece of engineering, and leadership felt that open sourcing would lend credibility to Lyft as an engineering organization and help with recruiting. As I said before, all of us were naive about what it takes to both create successful open source and - more importantly - nurture it if it becomes successful.
But, we decided to give it a shot. We spent a good portion of the summer of 2016 working on documentation (Jose Nino joined the team around this time and his first task was reading and helping improve all of the docs), cleaning up the repository to make it “less embarrassing,” working on a website, a launch blog post, etc. I am truly grateful for my coworkers at Lyft during this time who not only supported us but helped us with myriad tasks including website design, logos, and more. Even at this early time, it was intuitive to us that first impressions matter, and if we were going to make a go of open source we had to make a good first impression via quality documentation, web presence, etc.
During this period we also used our industry connections to meet with some of Lyft’s “peer companies” (“unicorn” bay area internet startups) to show them what we had done with Envoy and get their feedback, thinking that if we managed to get a launch partner before going public it would be a major help to the project. All of these meetings were very friendly, and across the board all of the companies we met with were extremely impressed with what we had accomplished. But, as is obvious in hindsight, all of them said that there was no way they could adopt Envoy right away with their small infrastructure teams. They wished us the best with open sourcing and said they would check back later. We couldn’t help but feel depressed at the outcome of these meetings, but we pushed forward anyway.
In August of 2016, I had my first auspicious meeting with Google. A Lyft coworker (Chris Burnett) had spoken at a gRPC meetup and mentioned Envoy as it related to Envoy’s gRPC bridging support. Unbeknownst to me, Google was preparing for the launch of Istio on top of NGINX when they found out about Envoy. One meeting led to another, and then many more, and before Envoy was open sourced a substantial number of Google employees had already seen the source code and documentation. (More on this later.)
By the beginning of September we were ready, and set the open source day as September 14th. In general I’m a (over?) confident person, but there have been times in my life where I had a significant amount of anxiety about my ability to succeed. The ones that immediately come to mind are: starting high school, starting college, and starting at Microsoft after college. And open sourcing Envoy was one of those times. I remember being petrified about the public reaction. What would people say? Would the feedback be positive or vicious? Although we were a small team by the time we open sourced, I had still written 90% or more of the code, and felt like putting it into the public domain was a reflection on myself and my abilities.
As scheduled, Envoy became open source on September 14th, 2016. I remember celebrating with my wife and saying something along the lines of: “I will be happy if we can get just one other company like Lyft to use Envoy.”
The reaction to the open source release was almost universally positive. Much to our surprise, almost immediately, we started hearing from big companies, not small ones. Within weeks we were talking to Apple, Microsoft, and the conversations with Google kept picking up pace. Large companies had issues with existing solutions, and had large teams of people ready to dive in and work on solving those issues. Ironically (at least in the view of the Twittersphere), C++ was a help here, not a hindrance. The large companies all already had ample C/C++ development resources, existing libraries they wanted to integrate, etc. C++ was a selling point to them.
During this time, not surprisingly we had the most interaction with folks at Google. Initially primarily the teams building Istio, but gradually we spent more time with Anna Berenberg, now a distinguished engineer at Google leading various networking and load balancing efforts. That relationship would turn out to yield the “jet fuel” that really launched the project in early 2017.
By early 2017, it was becoming clear that Envoy was gaining traction quickly. Google committed to replacing NGINX with Envoy for Istio (which eventually launched in the spring of 2017), and much more importantly for the future of the project, Anna’s large team working on GCP cloud load balancing features began their march towards using Envoy for various cloud load balancing products as well as internal use cases (this was all very secret during this time period but is now well known).
I will always remember that period interacting with Google as being one of the most stressful of my career. In all honesty, it felt like an acquisition (inquisition?) process. I remember long meetings and email threads justifying our technical decisions, “interviews” in which Google was trying to determine whether we would be a good OSS partner to work with, etc. It was painfully obvious to us at the time that landing this “acquisition” would put Envoy on a trajectory that we could never achieve on our own, so we did everything in our power to make it a success, which it ultimately was. And, our work with Google truly has been an outstanding partnership over the last 4+ years. Early Google cloud engineers that eventually became maintainers, Harvey Tuch and Alyssa Wilk, brought loads of talent to the project, both technically, as well as leaning into open source and the community. My gratitude to them is immense and the project would not be what it is today without them. The rest of the Google engineers who have contributed to the project over the years (there are now many) have added an immense amount of engineering horsepower that the project would not have otherwise had, in addition to universally being excellent community stewards. I certainly had concerns about the initial Google partnership (technical and philosophical differences, etc.), but I can honestly say that none of those concerns have become a reality.
Apart from ensuring the success of the Google collaboration, both across Istio and GCP teams, we were also spending a considerable amount of time working with and onboarding other companies and maintainers, many of whom have had outsized impact on the project and are still heavily involved today either as maintainers, contributors, or users. The project would not be what it is today without these early community members and I am extremely grateful to them as well for placing their trust in the project.
At the same time, as the project continued to gain traction, I started to receive a substantial amount of investor interest in Envoy. There was a strong desire to get me to leave Lyft and start a company around the project. I wrote about this part of the journey so I won’t rehash it here, other than to say that a lot of time and headspace went towards processing all of these interactions. As the linked post describes, I ultimately decided to stay at Lyft and not start a company in order to support Envoy’s continued success.
Meanwhile, I still worked at Lyft, and as I will discuss further later, I was increasingly working two jobs. My first job was internally leading the networking team and operationally supporting Envoy at Lyft. My second job was being the public face of Envoy including OSS leadership, code reviews, fixing bugs, writing features that would further the project, speaking at conferences, helping other companies adopt and deploy Envoy, etc. I was starting to get spread too thin and showing signs of burnout. However, by mid 2017 there was no denying the fact that Envoy’s trajectory was substantially “up and to the right.” Adoption continued to climb across major corporations, “peer companies,” vertical products and services, etc.
By the fall of 2017, two things were clear:
To address point one, we finally agreed to consider moving Envoy to the CNCF. The CNCF had been courting the project for months, but it never seemed like there was any compelling reason to join. By late 2017, it was clear that CNCF resources would be at least neutral to the project, if not a net benefit. We began the submission process and ended up joining the foundation almost exactly a year after we initially open sourced the project. I am thankful to Alexis Richardson and Chris Aniszczyk for shepherding the project through this process.
Point two was much more complex. Fundamentally, I was working more hours than I had capacity to work, effectively across two different jobs. Furthermore, I was expecting my first child, due in early 2018, which as the arrival date got closer was causing me increasing anxiety. By this time it had become clear that I had not done a good enough job on setting expectations and boundaries on what I was capable of providing to Lyft while still focusing on the continued growth of Envoy from an industry perspective. Increasingly, I was letting things drop at Lyft, getting into interpersonal squabbles, and not meeting the expectations of my level in terms of providing mentorship and leadership to more junior team members.
In short, I was hitting my breaking point and ultimately I chose Envoy over Lyft, to the detriment of my Lyft coworkers. I would like to think that if I had been more transparent with the Lyft leadership about my workload in early to mid 2017 I might have avoided some of the worst outcomes, but the unfortunate reality is that resourcing OSS industry work which is not immediately obviously useful to the employing company is a complex endeavor. It might have gone better and it might not have. In any case, while I regret some of the interpersonal issues that I could have handled better, for better or worse I do not regret focusing on Envoy. My priority was Envoy over Lyft and I did what I thought I had to do at that time to make it succeed.
My first child was born in February of 2018, and Lyft’s extremely generous paternity leave policy provided a natural break and reset for me. I got some space from Lyft, and started to think more deeply about what I wanted and what would be sustainable for me.
When I came back from paternity leave, I was clear with the Lyft leadership that I could no longer participate in the “day to day” of operating Envoy at Lyft. Conversely, the infrastructure team also wanted some separation from me due to some of the fallout from late 2017. Due to this, I stepped back substantially, and actually took a complete hiatus from infrastructure at Lyft to work on writing the firmware networking code in the Lyft Bikes and Scooters initial release in mid to late 2018. This was an amazing team effort to get something shipped on a compressed time scale, and I really enjoyed doing something completely different for a few months.
2018 was also the year in which I aggressively started to figure out what it would look like to “replace myself” within the Envoy OSS community. I spent a substantial amount of time (and continue to spend a substantial amount of time) grooming maintainers, new contributors, organizing the first dedicated EnvoyCon, etc. Any leader should always have a goal of making sure that the organization will continue to function well should that leader step aside one day.
By the end of 2018, my major burnout risk had been sorted out, and I was working reasonable hours again and spending plenty of time with my wife and son, splitting my time roughly 50/50 between Envoy OSS work and general infrastructure leadership at Lyft. To be clear, the privilege that came from Envoy’s success enabled me to carve out this work life balance with Lyft. Over time, as my industry stature increased, my leverage increased in parallel, making it easier to set the terms of my employment as I liked them. Not many have the fortune of being in this situation and I understand how lucky I have been to “break through” to the other side of the burnout wall without having to leave my job.
Since 2019, and through Covid, I have continued roughly the 50/50 split between Lyft infrastructure leadership and OSS leadership that I described above. There have certainly been times of monotony and yearning for something different (historically I am a habitual job changer - 6.5 years is by far the longest I have ever worked on one thing), but overall I have enjoyed seeing Envoy move from being an “upstart” to more of a “teenager.” I’m no longer preoccupied with doing everything my competitive brain can come up with to make Envoy an outsized success, because frankly Envoy is an outsized success, has swept the market, and has changed what users have come to expect from their application load balancing tools. Instead, I’m more focused on project sustainability. We are in this for the long haul, and these days I feel much more like a run of the mill CEO looking at attrition numbers, priorities, budgeting, security issues, etc. It’s not to say this is not useful work; it clearly is, it’s just a different kind of work from the early days which was substantially more technical and fast paced.
As of late 2021, the thing I am most proud of about Envoy is that in my opinion the community has become self-sustaining. We have an incredible group of maintainers, contributors, and users who are passionate about the project’s success and have all played a part in making Envoy what it is today. It is truly a team effort.
The past 5 years have been an epic journey. While I feel I have learned relatively little technically, I have grown and learned so much about leadership, community building, and all of the other non-technical things that go into building a successful enterprise, whether corporate or a major open source success story. What follows are short summaries of some of my main learnings.
Perhaps controversially, I think that if one has a goal to create an extremely successful OSS project they need to think of it like starting a business. There are lots of factors involved in starting a business beyond the core technology:
Intuitively, I knew this going into the initial open source effort for Envoy, and I aggressively pursued all of the above areas as I worked to grow the project from where it started to what it is today. Everything in the above list is critical and a project is unlikely to succeed without all of them, especially if the technology area is crowded with well financed corporate competitors.
I strongly encourage those contemplating a large open source effort to invest in the above areas ahead of time to make the best possible impression on day one. Additionally, new open source projects should be prepared to invest more heavily in the above areas if the project grows and starts to see adoption.
Not surprisingly, I do relatively little coding on Envoy these days. My time on the project is primarily managing all of the non-technical aspects of the project (everything in the above list and more!) and making sure things are on track. Most of the coding items I do take on are “janitorial” background projects that are good for the project, but are not very much fun and are not likely to inspire other contributors (which I of course have no say over what they do on a day to day basis and am incentivized to keep them happy as much as possible so they don’t leave).
These days a lot of “big OSS,” especially in the infrastructure space, is financed by large corporations and venture backed startups. I won’t detour into a discussion about the difficult economics of OSS as I already wrote about it. I will say that I strongly believe that end-user OSS has a substantial advantage over corporate and venture backed OSS: an initial captive customer that is almost certainly deriving value from the software, otherwise the software wouldn’t be funded. This virtuous cycle of building something alongside a customer is powerful. It almost universally leads to better outcomes: software that is more reliable, more focused, and with less feature bloat. There are many examples of end-user driven OSS that then goes on to achieve substantial commercial success. This is not surprising to me given the solid foundation and inbuilt product market fit. I would love to see more end-user driven OSS than we do today, though I recognize the economics are difficult. For those who have the opportunity, lean into the structural advantage that this type of software has!
This is perhaps a corollary to “successful OSS is like starting a business” and “end-user driven OSS is a structural advantage,” but I can’t stress how critical it is to focus relentlessly on what customers actually want versus what the hype cycle thinks they might want. For example, over the years, there has been an endless amount of fun poked at Envoy for being written in C++. Do I like C++? No, not really. Did it get the job done in 2015 and appeal to the initial set of main users? Definitely. This is an example of focusing on the customer and the market and not giving into hype with no real “business” impact. If one treats OSS like a business, it becomes immediately clear that being customer and market focused is the only way to achieve massive success. With Envoy I have spent a substantial amount of time arguing for the end user, to make sure we are building things in a way that benefits everyone, and not just a small set of niche users.
Following the customer can often lead to customer requests that may not fit cleanly into a project’s architecture. From an OSS perspective, losing focus of the primary goal of the project can lead to feature sprawl, unmaintainable software, and overburdened maintainers. At the same time, saying “no” is a guaranteed way to lose a potential user.
With Envoy I wanted to make sure that we could always at least say “yes, but…” in the form of offering a robust extensibility model that would allow users to fulfill their needs without having every change and feature need to be pushed upstream. This strategy has paid dividends many times over by reducing maintainer burden, allowing users to solve their own problems and, more importantly, pushing Envoy into use cases that I never would have imagined when the software was initially designed.
Extensibility, especially for OSS building blocks, is critical.
A further corollary to following the customer is that quality really does matter. Users want software that is easy to operate, is relatively free of bugs, cares about security, etc. At times it can seem that because OSS is “free,” quality is not guaranteed. This is maybe true in spirit, but practically users will not converge on a piece of software in large numbers until it’s clear that a project takes software quality seriously. Because acquiring users is a flywheel that acquires further users (especially when moving from early to late adopters) it’s even more critical to make sure that time is budgeted for overall software quality.
With respect to Envoy, I have always had a “zero crash” philosophy. Any crash is investigated and fixed, no matter how infrequent. This kind of attention to stability and quality does not go unnoticed.
It’s obvious, but I will say it anyway: community is the only way to scale OSS. This is a community of maintainers, contributors, and users. Furthermore, the tone of the community is set at the very beginning of the project and is extremely hard to change. Humans tend to follow norms. Once norms are set, outliers to those norms tend to be shunned, no matter what the norms are. Thus, the initial public tone of the project is extremely critical to setting its long term community trajectory.
When we made Envoy OSS, I put in a huge amount of effort into working with people on GitHub, using constructive and welcoming language. In general I did everything I possibly could to make Envoy a welcoming place where people wanted to come and contribute, whether that be maintenance, the occasional contribution, or users helping other users.
Of all the different types of success that Envoy has had, the part that gives me the most personal gratification, by far, is that I have been told by a non-trivial number of people that they had sworn off OSS, and especially infrastructure OSS, because they felt people in most projects were awful to each other. Conversely, they love contributing to Envoy because the community is so respectful and welcoming to each other. It required a lot of hard work and discipline, especially early on in the project, to achieve this outcome and it has paid off in multitudes.
Do not underestimate the compounding effect of setting a project’s culture and tone from the very beginning.
There has been much written about the difficult economics of OSS (including my own article which I referenced above). Suffice to say that trying to mix commercial success and open source success is very difficult, primarily because the successes can often be at odds with each other.
I do believe that Envoy threaded this needle via both its robust API and extensibility system. Essentially, Envoy became a tool that is now consumed by a large variety of vertical products and services. This has yielded a community that is filled with companies who have chosen to work together on a common substrate, even while shipping higher layer products that compete with each other, via innovating at the extension/API/control plane/UI/UX layer.
Any successful open source project will see substantial commercial/investor interest. If the goal of a project is to maintain a vibrant community while still allowing commercial success (which I would argue is required for overall project success as the money has to come from somewhere), it’s extremely important to think up front about how to split the core from the commercial layers. The practicality and strategy of doing this will differ depending on the project and technology, but I believe that focusing on a robust API/extensibility split is a fruitful strategy.
There is much discussion in the modern open source discourse about the role of foundations. I’m not going to comment heavily on this topic, but my main piece of advice is not to get distracted by foundations and the theoretical benefit they might provide. Instead, focus aggressively on product market fit, shipping quality software, and providing value to users. The rest will come naturally if these things are achieved.
For very successful projects, foundations, and more specifically neutral trademark holding grounds, are useful constructs, so I would definitely consider one at that point. The value that Envoy has derived from being part of the CNCF has increased over time as the project has matured. CNCF employs OSS lawyers, marketers, public relations personnel, a top notch event staff, and more. These extra resources are invaluable when it comes to “running the business.”
Every time I see another OSS community leadership blowup, I can't help but think that BDFL is in practice the lowest drama and most practical way of running large OSS projects over the long term.
— Matt Klein (@mattklein123) September 13, 2021
I wish this were not the case but it certainly seems that it is.
Open source governance is extremely hard. By its very nature, open source is anarchic, without a clear leadership structure. There is no one size fits all approach to project governance, and each project has to find its way forward, either via a “BDFL”/CEO type model, a steering committee, an Apache PMC like process, etc. All governance models have pros and cons and have different failure modes.
What’s most important is to think hard about governance up front, before the project becomes large and successful. Write down a set of rules and norms and especially take time to document the project’s conflict resolution process.
Also realize that per my comment above about how community norms are set early, early project maintainers will have an outsized impact on the overall style of dialog and conflict resolution, much like early employees at a company have an outsized impact on the company’s culture.
We have been extremely fortunate within Envoy to not have had any major disagreement that I can remember that was not quickly resolved amicably. We have never in the history of the project needed to invoke the maintainer voting process for conflict resolution. This is in my opinion a substantial achievement, and a testament to the quality and professionalism of all of the maintainers, especially given how popular the project has become and all of the commercial interests that surround it.
I alluded to this above, but much of my own burnout stemmed from a poor job of setting reasonable expectations with my employer on the amount of time I needed to spend managing Envoy’s open source growth. I’m not going to lie and say that having such a conversation will magically make an employer carve out lots of time for someone to work on OSS, especially items that may not be directly applicable to one’s day job. With that said, I do believe it’s very important for all involved to have open and honest expectations about the open source process. The following are reasonable questions to ask either before open sourcing a project or before starting to work in an open source capacity:
Mismatched expectations between employers and employees is a recipe for future resentment and burnout.
To some, it may seem like the underlying network proxy mechanics that Envoy provides is the complicated part of the project. As it turns out, the proxy bits are (in my opinion) relatively simple compared to what it has taken to evolve a stable API ecosystem for Envoy. The mechanics of balancing API ergonomics for both human and computer consumption, maintaining stability across versions, growing the API to support other clients such as gRPC, specifying protocol semantics so that Envoy can speak to hundreds (possibly thousands?) of different management servers, etc. are mindboggingly complicated. I’m proud of what the team has achieved in this area (and a special shout out to Harvey who has driven much of this work), even with some mistakes along the way (such as the forced migration from v2 of the API to v3).
If a piece of software offers an API, and more importantly wants this API to be a critical building block for other systems, don’t underestimate the cost and complexity of offering a stable and ergonomic API surface. The flipside of this is that robust APIs are a strong part of an ecosystem’s flywheel of more products and users yielding further products and users, so in my opinion it’s well worth the effort.
I’m not one to believe that a good work life balance is achievable 100% of the time if one wants to accomplish big things. The reality is that any success is a mix of existing privilege/opportunity, a good idea, good execution, and a whole lot of luck including being in the right place at the right time. All of these things came into play with Envoy, and I’m not going to pretend that I didn’t work myself almost into the ground, especially in 2017. I would also do 2017 all over again, because from my perspective I did what I had to do to make the project a success. (Sometimes I wonder whether Envoy would have happened at all if I had already had children. I’m not sure that it would have, but that is a subject of a much longer conversation!)
With all of that said, the type of epic pushes that I describe in 2017 can only go on for so long until a person breaks. I encourage everyone to be reflective on an ongoing basis about their work life balance, and figure out a sustainable path forward for themselves. All situations are different, and I can’t offer any one piece of advice for avoiding burnout, but I think being reflective is a good start, and something that I have had to work on quite a bit for myself.
Working on Envoy for the past 6.5 years, 5 of those as open source software, has been the highlight of my career. The project’s success has truly been a team effort that I never could have accomplished on my own, and I am so proud of what all of us (maintainers, contributors, and users) have accomplished together. The maintainers and contributors who work on the project are the best group of engineers I have ever worked with; too talented a group to ever find themselves all at the same company or in the same geographic location - truly the theoretical potential of open source played out in practice. As a team we have had worldwide impact, changed what users expect from their software load balancing systems, and also built a vibrant and welcoming community. In my wildest dreams I never would have imagined that the project would become what it is today.
What the future holds for me is less clear. As I said above, my focus has shifted to sustainability. I want to make sure that should I leave one day, the project will remain healthy. With that said, that day is not yet here, and I look forward to helping to lead the project forward for the foreseeable future, to hopefully even greater success and adoption. Onward!
]]>Although I am continuously talking to Envoy users, our now yearly EnvoyCon/KubeCon has become an important check-in point for me as I try to wrap my head around the types of problems that are being faced deploying Envoy “on the front lines.” Although I have access to this type of information via Lyft’s deployment, having more data points allows me to get a better idea of the most important things the project can be working on to better serve the needs of the broadest possible set of users.
After attending the most recent EnvoyCon/KubeCon and having some time to think about what I learned, a few high level points have become very clear to me:
In the following sections I will touch on each of the above points in more detail and then discuss what I think we can do as a project to pragmatically help the greatest number of large scale Envoy users in the near term.
Much of the current cloud native ecosystem is a shiny place that mostly assumes greenfield deployment. In this context, greenfield means starting from scratch with a set of cloud native technologies that have been designed and tested to work well together. Although as an industry we certainly have a long way to go to make greenfield cloud native systems easy to use (I’m looking at you massive tangles of YAML), we have made large strides in making the building blocks stable. On the other hand, large organizations already deploying legacy technology at very large scale have a different and more complicated problem entirely: attempting to bridge the old with the new without forcing users to undergo massive migrations, instability, and pain.
To use Lyft as a concrete example, our “legacy” compute stack is a bespoke build/deploy system that uses raw EC2 virtual machines, auto scaling groups, etc. Our original Envoy-based service mesh and API gateway grew up tightly integrated into this system and all of its inherent assumptions. Over the last couple of years, Lyft has undertaken a migration to Kubernetes. Setting aside the discussion of whether this migration has yielded positive ROI, the technical requirements of such a migration are massive. We have built a mechanism to bridge our legacy system and new system into a unified compute platform, thus allowing services to be run simultaneously and transparently across both to allow for safe compute migrations. It’s almost inconceivable to think that one of the “out of the box” Envoy service mesh solutions could have been used for this purpose without massive modifications. (For more information on how Lyft adapted its service mesh to Kubernetes see this excellent EnvoyCon talk by Lita Cho and Tom Wanielista.)
One of the hardest problems we face in the cloud native space is balancing configurability with complexity and usability. The more opinionated a solution is, the less configuration it needs, and the less complex it is (at least theoretically). In this regard, greenfield solutions, and especially PaaS and FaaS, have a major leg up. They can make a variety of assumptions about compute stack, deploy tooling, networking, etc. that allow them to significantly reduce needed configuration, thus decreasing complexity and increasing usability.
As a project, Envoy has never tried to simplify configuration as a primary goal. We assume that Envoy is a tool that is used in a large variety of deployments and vertical products. This assumption works against us in the form of sometimes bewildered users that come to our documentation and APIs and don’t know where to start. At the same time, Envoy’s rich API has been one of the driving forces behind its rapid adoption within the industry, making this a fruitful tradeoff in my opinion. The main point is that configurability versus complexity is a sliding scale, and it’s impossible to make every customer happy, thus necessitating a layered approach to composing systems.
In this regard, the various service meshes and API gateways built on top of Envoy are approaching layering in a pragmatic way: they are using Envoy as a building block and creating a “simpler” and more opinionated platform on top. However, as previously explained, simplification is at odds with configurability, making it significantly more complicated (if not impossible) for these solutions to work in more complex brownfield deployments such as Lyft’s.
Lyft’s story is not uncommon and is representative of why Envoy is extensively deployed across the industry most often with a proprietary custom control plane written in-house: each of these organizations have independently decided that for their brownfield deployment it is more efficient to have full control over the Envoy xDS API and the control plane code that drives it.
Will most organizations still be writing custom Envoy control planes in 3-5 years? I doubt it. In that timeframe we will have seen more migration to standard cloud native technologies and many of the current greenfield systems will have since become large scale deployments. In such a future world, I would expect the majority of deployments to use one of the vertical solutions that are being developed today. In the interim however, things are messy. The realities of large scale brownfield deployments are pushing users to develop custom Envoy control plane solutions, and all of them are facing similar problems at scale. These problems include:
Immutable infrastructure compute systems such as Kubernetes typically drive very high rates of system changes, as containers are created and terminated rapidly due to autoscaling, deploys, batch jobs, etc. Our experience at Lyft has been that the rate of change can be an order of magnitude or more beyond that which occurred with our legacy system. A high rate of change in the compute/network topology puts a substantial amount of pressure on the control plane; if not careful, a naive implementation is easily susceptible to runaway topology recomputations.
An Envoy control plane has to implement a large number of standard distributed system best practices. This is partly due to immutable infrastructure leading to high rates of system change and partly due to the realities of scale in general. Such best practices include:
At large enough system scale, it becomes untenable to send every endpoint for a service to every downstream client for the following reasons:
The solutions to the previous two problems are twofold:
Each of the previous two solutions are far from easy to implement in practice.
The debugging and observability requirements for operating a control plane at scale are not significantly different from operating Envoy at scale. It’s critical to understand the state of the control plane and its caches, whether the configuration has converged, what the current configuration is, etc. Although implementing such debugging and observability features are not complicated, each organization deploying Envoy at scale has duplicated this effort in a slightly different way.
If we assume, as this post outlines, that deploying Envoy is going to be messy for the next 3-5 years as organizations eventually converge on vertical solutions and products, what can we do as a community to make deploying Envoy control planes easier for everyone currently writing custom solutions?
The go-control-plane project has existed for quite some time as a library that can be used to speed up development of Go-based Envoy control planes. The library provides the outline of things that a control plane needs to support including caching, configuration delivery, etc. It also handles the unglamorous task of compiling Envoy’s protobuf API to Go code.
However, go-control-plane is a very thin library and does not currently provide any of the distributed systems best practices functionality described above, nor does it implement subsetting or incremental/delta xDS.
I’m often asked if Lyft will ever open source our control plane. The answer is no, primarily because of the brownfield tangle that I described earlier in the post. There are simply too many assumptions about Lyft’s legacy infrastructure and business logic built into the control plane. Not surprisingly, every organization of Lyft’s size or larger using Envoy says a similar thing.
What I think both Lyft and the industry can do is move more functionality into go-control-plane such that the library itself becomes more useful. This includes things like rate limiting, batching, back pressure, enhanced caching, a reference Kubernetes and Service Mesh Interface (SMI) implementation, etc. This obviously won’t help organizations that want to build their control plane in a different language, but with Go (like it or not) becoming the lingua franca of cloud native development this seems a reasonable compromise. Moving more functionality into the library will not produce a new service mesh product, but it will produce a more reliable base that we can all collaborate on, find bugs in, and harden.
In addition to enhancing the out-of-box capabilities of go-control-plane, we are kicking off a project called xds-relay (design document). This easiest way to think about xds-relay is that it is a CDN for Envoy configuration. We believe that it is possible to create a self-contained server (built itself on go-control-plane) that can be deployed in front of any xDS compliant control plane “origin” server. The relay will be a scale out component that helps large Envoy deployments achieve high availability by implementing all of the distributed systems best practices outlined in this post in one open source place.
Additionally, once the xds-relay MVP (simple caching and scale out) is complete, there is a huge amount of potential for community collaboration on additional features such as:
By moving much of the complexity of building Envoy control planes into open source, we can all collaborate together, find bugs together, and harden the implementation together, thus making the proprietary/internal pieces much simpler to reason about. This is a win/win for everyone. If you are interested in helping with relay development please reach out and join us!
The rapid industry uptake of Envoy has seen the creation of many vertical products and services, but also the creation of hundreds of custom bespoke control planes. It has become clear that many organizations with a custom control plane are not going to give it up any time soon in favor of shrink wrapped service mesh and API gateway solutions, but they are also redundantly solving the same set of distributed systems and scaling problems. Over the next year, through projects such as go-control-plane enhancement and xds-relay, we can improve the status quo for the average Envoy user as well as increase the stability of the vertical products and services that will ultimately utilize these projects as building blocks.
]]>Because ~50% of my job is engaging with the industry via Envoy, CNCF, etc. I thought it wouldbe fruitful to set aside some time each week to have office hours open to anyone. This time can beused to talk about Envoy, CNCF, cloud native technologies, or just get general career advice. I’m open to anything! For more information on how to sign up please see this page. I look forward to chatting with you!
]]>The last time I did any substantial web development was in the 90s. A lot has changed since then!
I decided that I need a personal website. I haven't done any web work since the 90s. What you can build today in an hour \\\\\*for free\\\\\* with @Netlify, @letsencrypt, and static site generators is absolutely incredible.
— Matt Klein (@mattklein123) March 4, 2020
We use Netlify for the Envoy website. I have a bit of experience with it so I decided to use it for my personal site. I registered my domain name using Google Domains and then switched the name servers over to Netlify. Netlify automatically takes care of getting a TLS cert via Let’s Encrypt. For the static site generator I decided to use Hexo based on a coworker recommendation. After browsing through a bunch of Hexo themes I picked a simple one called Cactus. With 1-2 hours of tweaking I had the fully functioning website you see now. Amazing! I even set up Netlify CMS so I can write posts directly in any browser. Netlify takes care of all of the authentication and authorization via GitHub, and even opens PRs for in-progress draft posts.
The fact that I could create this level of sophistication in such a short period of time, without using a database, and for free absolutely blows my mind. Granted my experience in this area is 20+ years out of date but it’s nice to see so much progress in the state of the art.
In the future I plan on adding comment support, Google analytics, and a few other small features.
Up until now I have done all of my writing on Medium. Although I know that Medium is a frequent source of complaints, it has worked well for me. I haven’t decided yet whether I will transfer my medium posts here, only write new posts here, or potentially keep using Medium for long form writing. Stay tuned.
]]>