Introduction

I left Airbnb a few weeks ago, a decade after joining in early 2016.

Taking inspiration from Reflections on Palantir and Reflections on OpenAI, I want to write down some reflections on my time there while the memory is still fresh. This post is part about what I think makes Airbnb unique, and part a recollection of lessons learned over the years. All in all, I left the company with a lot of gratitude, and that’s the spirit I’m writing this in.

Company

Business

When I joined in early 2016, the company was in full growth mode, and I spent my first few years growing the host side of the marketplace. That’s where I learned that running a two-sided marketplace is mostly about designing incentives and rules that don’t overly favor one side or the other. Take Instant Book, where guests can book without host approval. It’s convenient for guests, and it also means hosts have less control over who stays in their home. Or House Rules, useful for setting expectations. Pile up too many of them and you’ve created a hassle for guests. Growing a marketplace means the market design matters just as much as the growth curve.

After several years of high growth, Airbnb announced its intention to go public in late 2019. Then COVID hit: by April 2020, gross nights booked were down 72% year over year, and net bookings turned negative, an unprecedented situation, as cancellations skyrocketed. After raising capital and cutting costs everywhere else, Airbnb made the hard decision to lay off about 25% of its workforce, roughly 1,900 people, which remains the largest layoff in Airbnb’s history.

As the world slowly adapted to life with COVID, business recovery came faster than it did for most of our peers, as travel demand shifted toward domestic trips and rural destinations. Having supply across 100,000 cities and towns and 220 countries and regions meant that wherever demand reappeared, we already had listings there. In a miraculous turn of events, Airbnb went public that December, ringing the bell with hosts around the world. After the year everyone had just been through, it was an emotional moment.

Airbnb has spent the years since perfecting the core product, and it’s now expanding beyond it. Services, Experiences, and a push into boutique hotels are among the newest bets. It’s also exploring what AI can do for the product and the business. Most of these are early and unproven, but a new chapter has begun.

Culture

For a company whose business is predicated on 99% of humans being good (after all, you’re staying at a stranger’s home), Airbnb looks for many of those same qualities in hiring. Beyond the standard interview process, Airbnb runs culture interviews to assess a candidate’s fit with its core values. In mine, I talked about a trip I’d helped plan to Peru. I was told later that the stories I shared embodied the core value of “Be a Host.”

It’s no surprise, then, that my favorite core value is “Be a Host,” and it shows up in small but telling ways. On my first day at Airbnb’s HQ, people were opening and holding doors for me and for each other. That kind of behavior extended beyond small daily etiquette into how people actually worked together. My coworkers were brilliant, and brilliant assholes were rare. People were competitive, rarely combative. It was, on the whole, a place of low drama, where people treated each other with a lot of respect.

On the flip side, this desire to be a great host can sometimes get overextended into playing nice, or even conflict avoidance. This shows up often in consensus building, which can be a slow and sometimes painful process. It often comes as a shock to those used to a culture of direct, blunt feedback.

I do think the cofounders don’t want to fuck up the culture, and they’ve tried to hold onto the early spirit every time the company outgrows itself. That said, it’s hard not to miss some of the early days, when the company was smaller and less formal. Every Friday we had nerds@, where engineers shared what they’d learned or built that week. At world@, product leads frequently showed early snippets of what we were about to launch. Some of that faded as we grew, and I wish we’d kept more of it.

Operating Model

Given that Brian Chesky, co-founder and CEO, is a designer by training, it’s no surprise that he applies that same design mindset to building the company itself, and he’s never shy about learning from iconic companies.

On the fourth floor of Airbnb HQ, frames hang on the walls describing the host and guest journey, inspired by how Walt Disney designed the storyline of Snow White. In the years leading up to COVID, Brian drew heavily on Amazon, with the ambition to grow Airbnb into a Trip platform that went beyond just accommodation. More recently, he’s turned to Apple, reshaping how we do product marketing, launches, and roadmap planning. Each era brought its own ambitions and its own way of working, often under new leadership too.

Every change, though, came with its own growing pains. Pre-COVID, that meant unfocused sprawl: Airbnb was simultaneously pursuing experiences, magazine, business travel, hotels, and even flights, spread across four largely independent business units. Post-COVID, the pendulum swung toward bi-annual, big-bang releases: high-stakes, all-or-nothing launches that made it harder to isolate what was working. And Brian’s founder mode talk spun up its own debates, internally and externally. Despite this thrash, I appreciate that Brian has the ambition to build an iconic company and isn’t afraid to adapt and experiment, treating the company itself as an iterative design problem.

One more operating model choice worth mentioning: Airbnb is one of the few companies that genuinely committed to remote-first since COVID. I’ve benefited from the flexibility, but I miss standing at a whiteboard with a coworker and working a problem out. Keeping that kind of collaboration alive is a hard problem. We’ve hired strong people in places we couldn’t have reached otherwise, though it’s harder to screen for candidates who want the job rather than the flexibility. The remote-versus-office question is challenging, and Airbnb is still working hard to find the right balance.

Data at Airbnb

Teams

In the early days, the entire data team, known internally as the A-team, was small enough to fit in a single room. It skewed toward early-career PhDs from a mix of backgrounds: economics, statistics, operations research, and the social sciences. All of them were sharp thinkers, deeply data-driven, and product-focused.

Organizationally, individual contributors owned specific domains and became experts in them, but they all reported up to a Head of Data. In more recent years, they report to various leads in engineering in a decentralized fashion. By the time I left, Airbnb had hired its first VP of Data Science, so perhaps the pendulum will swing back toward a more centralized data organization. These swings are not uncommon: LinkedIn and Facebook went through similar evolutions.

Airbnb’s relationship with Data Engineering has been a complicated one. The investment started early: many of the founding data engineers came from Facebook, and our early warehouse showed it. The medallion architecture and core data were heavily inspired by what they’d built there. Around 2018, the org dismantled the Data Engineering team, and in my opinion it was one of the costliest mistakes data leadership at Airbnb made.

Luckily, we retained some of the strongest engineers, many of whom moved to Data Platform. Under new leadership, Airbnb reinvested in data engineering hiring in 2019, and the community is strong again today. Airbnb was also one of the first companies at this scale to create an Analytics Engineering organization, thanks in part to its investment in tooling like Minerva, our semantic layer.

Overall, the roles are increasingly specialized: data scientists and analysts focus on product work, analytics and data engineers build company-wide datasets, and software engineers build the underlying platform.

Platform

Airbnb historically leaned “build” over “buy,” and several successful open-source projects were born here, most notably Airflow and Superset.

For offline data, everything lives in a lakehouse: data on S3, stored as Parquet files in Iceberg tables. Data is typically date-partitioned, and we compute incrementally wherever possible, though some late-arriving data (think cancellations or alterations) forces a full history rewrite. Spark is the main engine for batch, Flink for streaming, and Trino for interactive queries. For orchestration, Airbnb runs one of the largest Airflow deployments in the world, often at a scale the open source community isn’t equipped for.

Airbnb leans heavily on config-driven frameworks, so much so that some joke that our data contracts are entirely built on fragile YAML files. The frameworks built this way are still widely adopted: ML feature platform Chronon, Minerva as our semantic layer, and an experimentation platform called ERF, to name a few. Python is the primary language for building these data frameworks.

We invest just as heavily on the consumption side. Dataportal is a catalog and UI that helps people find the right data, and a unified metadata service sits underneath it, storing the metadata every other data tool depends on, such as ownership, landing times, and asset tagging. More recently we built an internal data agent, and it has shown real promise, largely because the semantic layer and metadata service were already there for it to stand on.

I’m biased here. I think Airbnb’s data ecosystem is sophisticated and underrated compared to peer companies. Investment in data was one of the reasons I joined, and it held up.

Semantic Layer

For seven of my ten years at Airbnb, I worked on the company’s semantic layer, Minerva. People at other companies often ask how we scaled it across the entire organization. There wasn’t one standout strategy. It came down to aligning with company initiatives, finding champions, and a relentless ownership mindset.

As early as mid-2018, we were already thinking about consolidating definitions across business metrics and experimentation metrics under a single source of truth. The real catalyst came around 2019, as Airbnb prepared to go public and data quality, or the lack of it, became an existential problem. Years earlier, we’d made the mistake of dismantling our Data Engineering team, and we’d been paying for it ever since. Different teams built their own versions of “bookings,” “active listings,” and “revenue.” When Brian asked for last week’s bookings number, he’d get a different answer depending on who he asked. For a company about to report to public markets, that was untenable. Our CTO was so concerned he declared “data bankruptcy.”

Out of that urgency came three efforts. First, a company-wide data quality initiative to rebuild the most business-critical data models from the ground up. Rebuilding models once wasn’t enough to keep them trustworthy, so we created a certification process called MIDAS to hold data to a consistent bar of quality. Finally, we invested in infrastructure: a way for data producers to define a single source of truth for business metrics and dimensions that could actually be certified. Minerva became the natural home for that.

IPO-readiness created the urgency. MIDAS gave us the program and process to act on it, and Minerva ended up being the paved path that came out of it. We worked closely with key leaders early on to position it that way, then expanded team by team as wins built momentum. It took about two years before Minerva became the standard tool for Analytics. In some ways, the analytics engineering role at Airbnb exists because we had this central piece of technology sitting in the middle.

Work Lessons

On Building Software

Working at the intersection of software and data engineering, I got to learn from several exceptional software and data engineers who taught me how to think about the craft.

My first big takeaway is that “software engineering is programming integrated over time,” a definition popularized by the book Software Engineering at Google. Design patterns and abstractions are tools for managing complexity, more than ends in themselves. When a group of people with different levels of understanding and mental models work on the same codebase, having these at our disposal is a proven way to evolve the software while keeping everyone’s understanding aligned.

Speaking of design patterns, I learned that they are a useful vocabulary for identifying common problems and possible solutions. Several came up again and again: Adapter pattern, which let us wrap different backend databases behind a consistent interface; Bridge pattern, which we used to decouple the Airflow operator from the unit of work it carries (what we call Step); and Strategy pattern, which became the backbone of our audit framework for checking data quality.

On abstraction: I started off writing procedural code to implement our write-audit-publish (WAP) pattern, then watched more experienced engineers replace it with abstractions that simplified the code I’d written and made writing new code easier. In another example, we introduced a Spell abstraction that lets developers add codemod capabilities, invoked by both developers and CI for config validation. It replaced a series of very fragile shell scripts strung together over the years. Nowadays, if I find myself writing similar code with unnecessary implementation detail for the task at hand more than once, I pause and ask whether there’s a useful abstraction we can introduce to hide that complexity.

I’ve also picked up several other technical lessons along the way: domain modeling, why choosing boring technology is usually the wiser choice, composition over inheritance, dependency injection and inversion, and snapshot testing. Each one probably deserves its own blog post. The common thread is that all of these took time to internalize. I only appreciated them after working in the same codebase long enough to see why they mattered.

On Maintaining Software

There was a period where we were heads down building and re-architecting. There are also periods that call for reliability and stability over agility and speed of change. Navigating different phases of a software lifecycle requires shifting how you think and where you focus, and for me, it took a few painful incidents before I really internalized that.

For a while we treated every dataset roughly with the same priority. Once the platform was adopted company-wide, it became clear that financial reporting and executive dashboards were far more critical than someone’s ad-hoc report. We introduced a data tiering system that lets us prioritize compute resources according to each dataset’s tier and SLA requirements. We also invested in observability. For a long time we didn’t know what “healthy” looked like for most of our systems, so we built layered observability: real-time, intraday, daily, and defined “good” for each tier. To keep low-signal alerts from burying the real issues, we routed alerts to the right owners and tuned thresholds until the ones that fired were worth acting on.

To harden our system, we learned from every incident for corrective action. One recurring lesson: big-bang releases are risky because the blast radius is enormous. We leaned into gradual rollouts, feature flags, and staged deployments, and over time deployments got calmer. We also surveyed the team on which parts of the job were intolerably repetitive and automated as much of it as we could. Restarting failed jobs, validating releases, and clearing alerts by hand were the biggest offenders, and we automated most of them over time.

At times, we needed to make much bigger changes. The accumulated weight of new use cases and tech debt made the architecture itself unmaintainable, and the only real fix was to rebuild the foundation. We did that twice with Minerva over seven years, and I expect it will happen again as requirements and use cases continue to evolve.

On Ownership Mindset

Having worked on Airbnb’s Data Platform for many years, I’ve come to believe that one of the keys separating a successful project from a mediocre one is the level of care contributors bring, the ownership mindset that makes you take pride in your work.

For us, that mindset showed up in everyday choices. While it was often tempting to build the intellectually interesting thing, we pushed back on our own over-designed proposals, and more than once killed them, in favor of something simpler that unblocked users. We cleaned up technical debt when the opportunity came, not because we treated debt as something to avoid at all costs, but because we knew it would help other developers down the line, even if users never noticed. And when users ran into issues, we took pride in unblocking them fast. Our on-call rotation was where we learned what was broken, not just a chore to rotate through.

Ownership mindset mattered even more during difficult times. There were periods when reliability problems hurt our team’s reputation, voluntary attrition left us severely under-resourced, and we couldn’t prioritize the work users were asking for. In moments like those, we called out the problems honestly and took steps to address them before they turned into fires. When the difficulty was structural, organizational misalignment rather than a technical gap, we weren’t shy about rallying leadership to make bigger changes.

I was lucky to work on a team where everyone showed a high level of agency and ownership. That’s part of why Minerva stuck with Airbnb’s data community for as long as it did.

Personal Lessons

Switching Roles

Some of the defining moments of my time at Airbnb came when I stepped outside my existing role and started over from scratch.

From 2016–2019, I worked as a data scientist helping grow our host community across several marketplace tiers (remember Airbnb Plus?). When the company dismantled its data engineering org, I was one of the few data scientists who dove into the wreckage and rebuilt from there, and found I enjoyed the work. Building pipelines and producing high-quality datasets taught me how much leverage comes from the right tooling, so from 2019 to 2021, I pivoted into product management to help scale Airbnb’s semantic layer from 0.5 to 100. From 2021 on, I worked as a SWE on the Minerva team to rebuild our stack from the ground up.

A shift in your interests doesn’t automatically earn the organization’s trust that you can execute in a new role. Every stretch into something new is a bet the org makes on you, and it’s on you to make it pay off. My playbook stayed the same each time: do great work, earn a reputation that precedes you, find an adjacent area that interests you, identify a sponsor, then pivot if that move propels growth.

Switching functions re-accelerates your growth. It isn’t free, either. It can slow your climb up any single ladder, and the expertise you’ve built fades into the background, at least for a while. You have to be willing to sit with that discomfort and be honest about your own learning curve (more on that in the next section).

Looking back, this is the main reason I stayed at Airbnb as long as I did. It felt like three different jobs packed into one decade, each giving me a distinct experience and its own growth opportunities. In the age of AI, I think the people who can operate beyond any single role are the ones best positioned to thrive, and I’m curious where that takes me next.

Growing Pains

With each transition come growing pains. Two stories from my time at Airbnb still stick with me. In one, I leaned in immediately. In the other, I flinched for months before turning it around.

In my first week (yes, first!) as the new product manager for the Minerva team, we had the largest incident in the team’s history, internally known as CIM-198. It was big enough that we had to put a moratorium on Minerva, blocking users from contributing new semantics to our platform. My immediate job was to communicate the scope of the incident and how we’d fix it. My broader job was to work with engineers to find the gaps in our system and build both short- and long-term fixes so it couldn’t happen again.

The whole thing felt like an extended interview, testing product management skills I barely had yet. Our users were patient, leadership gave us room to work, and I partnered closely with engineering to build a solid plan. We shipped several new features, including offline backfill, which still plays a key role in the architecture today. The incident was memorable enough that we eventually printed T-shirts reading “I survived CIM-198,” a badge of honor I hope we never have to award again. I had fond memories of that first week and thought I handled it well.

The second story is from when I first transitioned to software engineering. I worked with a mentor who was extremely capable and had a high bar. His style and presence could be dominating, which made me feel insecure. Unlike the CIM-198 crisis, there was no deadline compelling me to act here, so instead of leaning in, I did the opposite. I worried constantly about people’s perception of me, afraid of wasting his time, asking naive questions, and looking dumb. I’d charge ahead on implementations without checking in, only to surface a PR too late for meaningful feedback. It got bad enough that one day he asked me, point blank, “Are you avoiding me?” That was a rude awakening.

Something he told me afterward has stuck with me since: even if my judgment was 99% bad starting out, as long as I was willing to reflect on why, my taste would improve over time. That was the actual path to growth. It took some psychological work, but I realized a strong re-start meant embracing the growing pains and letting go of my ego. I eventually leaned into the discomfort and changed how I conduct myself. I started pair programming more, asking questions earlier, and letting myself look unsure. Six months later, that same mentor told me another respected engineer on the team considered me one of the most reliable people he worked with.

The difference between the two stories was how quickly I let go of my ego and embraced the growing pains. I’d like to think I’ve gotten faster at that over time.

Seeking Impact

In most companies, growth is measured by your level on the career ladder. It’s a useful external scorecard, but orient your whole career around it and you might end up chasing levels and labels instead of the work that energizes you.

I learned this the hard way as a PM scaling Minerva. By most external measures, things were going well. I had real impact, worked with great people, and the product was growing fast. The day-to-day was wearing me down anyway: constant context switching, no time to think deeply, managing up, down, and sideways all at once. I started seriously thinking about leaving the company.

What changed my mind was a conversation with my partner. She listened to me vent, then said: “It seems like you’re still very passionate about the company and the team’s mission. You’re just in the wrong job.” She was right. I still cared about Airbnb, the team, and the mission. What had happened was that I’d drifted into a shape of work that didn’t fit me, and I hadn’t been honest with myself about it. I’d gotten pulled in by the feeling of having impact and stopped asking whether the work itself was right.

That conversation pushed me to advocate for a move back into a more technical role, one built for deep problem solving. It wasn’t a straightforward move on paper, giving up a PM role to start over as an IC in engineering. It was the right call for what I wanted my days to look like, and it set up the most fulfilling chapter of my time at Airbnb.

Nobody will advocate for the work you love more than you will. Your manager has other priorities to juggle, the org has its own gravity, and the career ladder keeps pulling you toward the next level whether or not that’s what you want. Knowing yourself well enough to push back, and having the courage to do it, matters more than any promotion I got.

Parting Thoughts

I never thought I’d stay at Airbnb this long.

Looking back, it felt like working three distinct jobs across three different companies. Careers are non-linear, and staying self-aware is what let me course correct along the way. In the process, I came to understand what I’m good at, what I enjoy, and what I want more of. I made a real impact, and I was lucky to work alongside some of the kindest and most talented people I know, some of whom I now consider dear friends.

I’ll miss Airbnb dearly. I’m excited to see what the next chapter looks like, and this decade has given me what I need to navigate it.

—

Thank you, Jason, for onboarding me to Airbnb, which led to a decade-long friendship. Thank you, Vaughn, for teaching me everything about Airbnb and what it means to do right by the business. Thank you, Aaron and Hamel, for teaching me all things data. Thank you, Ricardo and Cuky, for believing in me as a Data Scientist. Thank you, Jeff, for betting on me to lead the Minerva team when I had no PM experience. Thank you, Shao, Dave, and Vyl, for helping me transition to SWE when I asked for it. Thank you to the whole Minerva team for working alongside me all those years. Thank you, Philip, Krist, and Clark, for being constants on the Minerva team and going through the ups and downs together. Thank you, Toby, Chris, Barak, and Ginter, for showing me what it means to be an outstanding engineer.

Recently, a few incidents prompted my team to re-examine our approach to reliability. As a result, we invested deeply in tooling, observability, alerting, and deployment safety. Along the way, we uncovered several important lessons — not just for our team, but for any team operating complex, business-critical data infrastructure.

Here are five of the short lessons.

1. Not All Data Is Equal — Tier Accordingly

One of the most important realizations was that we were treating all datasets more or less the same — even though some were clearly more critical. For example, financial reporting data, executive dashboards, and launch metrics have much higher business impact than internal experimentation logs or ad hoc data.

We introduced a tiering system to classify datasets based on business importance. This helped us allocate engineering attention and infrastructure resources more effectively. It also paved the way for differentiated alerting, recovery priorities, and SLA expectations.

If you’re running a data platform, ask yourself: Do you know which data is truly critical to the business? If not, it’s time to map that out — and start treating your top-tier assets with the extra care they deserve.

2. You Can’t Improve What You Don’t Measure

As cliche as it might sound, we realized we lacked a reliable definition of platform health. We had dashboards, but they weren’t comprehensive. We had metrics, but not always the right ones. And without a clear picture, it was hard to know where to focus or whether we were making progress.

We tackled this by building layered observability views — real-time, intraday, and daily dashboards that gave us a cockpit-style look at system performance. We also worked with stakeholders to define what “healthy” looked like, and used that to formalize incident criteria and drive alerting improvements.

This effort underscored a core principle: visibility is the foundation of reliability. Without good measurement, you’re flying blind.

3. If Alerts Aren’t Actionable, They’re Just Noise

Another root cause of incidents — was alert noise and fatigue. We had too many low-signal alerts, often triggered by datasets that weren’t even business-critical. Engineers were getting paged, but not always with clear next steps or ownership.

We addressed this by routing dataset-level alerts to the appropriate owners, fine-tuning thresholds, and improving our on-call runbook with clear triage steps. The result was fewer false positives, less fatigue, and faster response to real issues.

Alerting hygiene doesn’t sound glamorous, but it’s essential. Audit your alert surface regularly, kill flaky or redundant signals, and make sure every alert includes enough context for someone to act.

4. Avoid Big-bang deployment, roll out gradually

Several incidents were caused — or worsened — by risky changes rolled out all at once. We had good testing practices, but we lacked safety mechanisms to gradually roll out new behavior or catch issues before they hit production.

So we leaned hard into gradual rollouts. We started using feature flags for high-risk changes, implemented staged deployment pipelines, and made sure every release was thoroughly baked in staging before going live.

This shift transformed our deployment culture. Releases became calmer, confidence went up, and reversions were much faster when needed. If your deployments feel risky or stressful, consider what parts of the rollout you can decouple, stage, or flag off until proven stable.

5. Manual Operations Don’t Scale

Finally, we looked at all the places where engineers were doing repetitive manual work: restarting DAGs, tagging releases, validating deployments, clearing alerts. Not only were these steps tedious — they were also prone to human error.

We invested in automation: Airflow utilities for bulk operations, scripts to validate release readiness, and tooling to reduce friction in our library release process. The result was a more reliable platform and happier engineers.

A good first step for any team: track how often you’re doing the same task by hand, and ask whether a script or tool could do it instead. Small automations add up — and often unlock disproportionate leverage.

Looking Ahead

The exercise to improve reliability wasn’t easy — but it was necessary. It forced us to ask hard questions, build better tools, and recommit to the discipline of operational excellence. Today, our infrastructure is more observable, safer to operate, and better aligned with the needs of the business.

If you’re building a data platform or operating critical analytics infrastructure, I hope these takeaways help you strengthen your own reliability practices.

Introduction

I joined Airbnb in early 2016 and recently celebrated my ninth anniversary at the company. Many, including myself, are surprised by how long I’ve stayed with a single company. In Silicon Valley, where the average employee tenure is usually no more than four years, nine years at one company feels like an eternity.

When people ask why I’ve stayed so long, I often explain that rather than sticking to the same job for nine years, I’ve explored three different roles across distinct functions. Over the years, several colleagues have reached out for advice on navigating similar transitions, since I’ve successfully made this shift not once, but twice.

While luck certainly played a big role, in this post, I would like to share a few things that I did that increases my odds of expanding my skill sets and career paths.

My Path

I started at Airbnb as a data scientist in early 2016, focusing on growing Airbnb’s supply. Over the years, I contributed to key projects and was always fascinated by some of the internal tools we built at Airbnb. From 2017 to 2019, Airbnb’s Data Science team faced a challenging time. We no longer had a dedicated Data Engineering team, which meant data scientists, including myself, had to take on engineering responsibilities. While some found this shift frustrating, I was passionate about the field and embraced the opportunity to learn as much as I could.

In 2019, Airbnb revamped its data strategy, and an opening to lead the Metric Infrastructure team came up. Despite my lack of formal product management experience, the hiring manager took a chance on me. From 2019 to 2021, I worked with a fantastic team of engineers to scale Minerva, Airbnb’s metrics platform, from a handful of use cases to a core component of the company’s data quality initiatives. Minerva transformed how analytics is done at Airbnb and remains the primary tool for creating trustworthy datasets.

By 2021, Minerva had become highly successful, but some of its early technical decisions were showing limitations. We decided to rebuild Minerva from the ground up to serve users for the long term. Seeing this as an exciting opportunity, I transitioned into engineering, with support from my engineering partner and my director. Since then, I’ve been a software engineer on the same team I once managed as a PM. By early 2025, we successfully migrated the entire stack to a new, more powerful system.

How to Improve Your Odds

While my path might seem straightforward, each transition involved strategic exploration, seeking sponsorship, and advocating for myself. While luck plays a huge role, it’s something beyond our control. Instead, I want to focus on the actions you can take before formally considering a transition.

Do Good Work

This should be obvious: if you can’t excel in your current role, it’s hard to convince others that you’re ready for more responsibility or something new. On the other hand, showing excellence makes it easier for others to envision you succeeding in different areas.

When Airbnb’s Data Science team needed to build critical data pipelines, I became one of the lead contributors on the Host team. I collaborated closely with data platform partners, learning a lot from them. I was also frequently on call for data issues and proactively proposed new ways to improve reliability and enhance our data tools. By taking ownership and rapidly improving my data engineering skills, I stood out during this time.

As Charlie Munger once said:

“To get what you want, you have to deserve what you want.

Establish a Good Reputation

Around the same time, I worked closely with the Data Platform team, particularly the Machine Learning Infrastructure group. As an early adopter of Zipline, our feature store, I faced numerous usability issues. Rather than getting frustrated, I turned these into constructive feedback and detailed documentation to improve the developer experience.

Witnessing the power of these new tools firsthand, I proactively advocated for them on behalf of the platform team and encouraged my fellow data scientists to give them a try. I partnered with individual users, led brown bag sessions, and conducted training workshops — each crucial to driving the adoption of these tools.

These efforts helped me build trust with my Data Platform peers, earning a reputation as someone who understood user workflows and could translate pain points into actionable requirements. I also became known as a natural developer advocate who could evangelize impartially. Little did I know, these would be key traits for a Product Manager.

Identify Adjacent Opportunities

While the transition from Data Science to Product Management might seem like a huge shift, it felt more like a natural progression for me. The role involved leading an internal product in the data space, rather than a highly visible product like the Airbnb app.

Given my background, I had deep empathy for how users might want to use the tools. Plus, my proven ability to collaborate with Data Platform engineers as an early adopter of Zipline was a huge asset. The hiring manager valued this experience to offset my lack of formal PM experience.

Similarly, when transitioning from PM to Engineering, I didn’t switch to a completely different domain. I focused on honing my engineering skills within the same team, which minimized context switching. This decision allowed me to focus my cognitive energy on learning engineering without the added complexity of learning an entirely new domain.

Identifying adjacent opportunities is, to me, a crucial aspect of successfully switching roles. Just as I advocate for learning adjacent skills, I believe the best way to achieve that is by pursuing adjacent opportunities.

Finding Sponsorship

No matter how qualified you are or how much desire you have to try new roles, you’ll need a sponsor — someone who is willing to bet on you. The best approach isn’t to ask for opportunities but to create and propose them. Clearly articulate why being in those roles will help address the sponsor’s challenges.

In fact, I didn’t initially consider the PM opportunity. It was the PM who asked if I knew anyone in my network who would be a good fit. After receiving some guidance on the ideal candidate the hiring manager was looking for, I reached out to a few people, only to realize later that I might be uniquely qualified for the role myself. I engaged with the hiring manager, and we went through several rounds of discussions. It wasn’t until after a few conversations that we both felt comfortable and confident that this would be a safe bet for both of us.

For my transition to engineering, my sponsor was the senior staff engineer who leads the overall team effort. I shared with him my long history and interest in building great data tools and explained what basic engineering skills I had (and those I still lacked). After reviewing some of my earlier work as a data scientist, he believed there was a good chance I could excel in the role and decided to advocate for me in front of our director. The director trusted the senior staff engineer’s judgment, and that’s how I began my engineering trial, which I eventually completed successfully.

Conclusion

Looking back on my time at Airbnb, I’ve realized that career transitions aren’t just about jumping into a new role. They’re about positioning yourself for new opportunities, embracing challenges, and continuously learning. Each transition I made was about taking the next step thoughtfully and strategically, building on what I already knew, and finding new areas to grow. Just as Fortune favors the prepared mind, luck favors the prepared.

Doing great work in your current role is key to proving you’re ready for more. Building a strong reputation for reliability and adaptability helps others see your potential to succeed in new roles. Focusing on adjacent opportunities, where you can leverage existing skills while learning new ones, makes transitions smoother. And remember, having a sponsor — someone who believes in you — is essential to opening doors.

Introduction

I recently came across Sean Goedecke’s post on What Can Strong Engineers Do That Weak Engineers Can’t?, and I found it to be a compelling read. His thesis is that strong engineers can tackle tasks that others simply cannot. While I agree with much of what he describes, translating those observations into actionable advice that helps one become a stronger engineer is not always straightforward.

Among the many skills an engineer can develop, one mindset stands out to me: proactively make the codebase you contribute to and maintain better. In this post, I want to share a few practical tips to make your codebase better every day.

A Shared Codebase Is Like a Shared Campground

Camping at a well-maintained campsite is a wonderful experience. A clean and organized campsite means you can set up your tent and enjoy nature immediately without worrying about leftover mess. In contrast, arriving at a disorganized and dirty site can be frustrating — you have to clean up before you can even start to enjoy your surroundings.

A software codebase functions similarly to a shared campsite. While each engineer can work independently to ship code, the final product is the collective effort of everyone involved — past, present, and future. Software engineering is inherently collaborative, relying on design discussions, code reviews, and shared knowledge to function effectively. This is especially true for any sufficiently large software project.

When you modify a piece of code, chances are that someone else wrote it before you. Your task then becomes understanding what was built, what the author’s intent was, and why certain implementation choices were made. If the original code was not thoughtfully written, future engineers are left guessing about its purpose, how to modify it without introducing bugs, and whether they might trigger hidden landmines in the process.

Strategic Mindset

Some of the best engineers I know approach software development with a long-term mindset. When writing code, they use meaningful variable names, write complete docstrings, and ensure clarity in their implementations. When shipping new features, they aim for minimal, well-structured changes to extend functionality. When introducing major components, they consider not just development costs but also long-term maintenance.

Most importantly, they think about the future state of the codebase — from the perspective of their future selves and teammates. They strive to make the codebase cleaner, even if they don’t refactor everything immediately. They have a keen sense of which parts of the codebase need improvement and take pragmatic steps to address them. They embody the Boy Scout Rule: Always leave the campground cleaner than you found it.

What Strategic Mindset Looks Like

Here’s a non-exhaustive list of actions starting today to make your codebase cleaner everyday, using strategic mindset:

1. Proactively Addressing Issues — Instead of waiting for a bug to escalate into a customer complaint or a system outage, proactive engineers fix problems early. They take responsibility for issues, even if they didn’t create them.
2. Thinking Beyond Immediate Tasks — While a typical engineer might implement a feature as specified, a proactive engineer asks deeper questions: Is this the right approach? How will this scale? What are the potential failure modes?
3. Maintaining a High-Quality Codebase — They don’t just ship code that works; they ensure it’s maintainable, well-documented, and efficient. If they encounter a confusing or outdated module, they take the time to improve it.
4. Caring About the Developer Experience — They recognize that engineering is a team effort and work to improve tooling, documentation, and onboarding processes to make life easier for their colleagues.

What Tactical Mindset Looks Like

On the other hand, engineers who lack a strategic mindset are tactical by nature. They tend to let problems fester, sometimes allowing them to escalate into significant issues. These engineers often adopt a “not my problem” mentality, leaving inefficiencies, technical debt, and small bugs unresolved until they become intractable. This undermines the collective effort of the team, creating friction and slowing down progress.

Signs of a lack of strategic mindset include:

• Neglecting minor issues because they fall outside the scope of immediate tasks, rather than addressing them as opportunities for improvement.
• Writing code that meets the minimum requirements but lacks consideration for future maintainability, scalability, or integration with the broader system
• Avoiding proactive improvements, such as refactoring, updating documentation, or adopting better design practices, even when they know such changes would benefit the team in the long term.
• Failing to migrate legacy systems or communicate necessary changes, leaving teammates in the dark or users unprepared for migration or system upgrades.

A highly skilled engineer with only tactical mindset can still be a liability to the team. They might write impressive code but leave behind technical debt, make short-sighted decisions, or be indifferent to the long-term health of the system.

Conclusion

Every team should encourage strategic mindset when it comes to software development. Otherwise, the maintenance burden of critical infrastructure often falls on a small group of engineers who deeply understand the system. This increases the bus factor and can lead to burnout and frustration.

Building a strong engineering culture and strategic mindset means fostering an environment where engineers take pride in their work, think about long-term maintainability, and actively contribute to the overall health of the codebase. A cleaner, more maintainable codebase benefits everyone — not just today, but for years to come. This mindset is not just psychology; it’s about contributing to a healthier, more sustainable engineering culture.

When engineers take responsibility for the codebase, they create an environment where innovation can thrive, technical debt is minimized, and collaboration is seamless. So next time you touch a piece of code, ask yourself: Am I leaving this better than I found it?

Bonus

I found the book, A Philosophy of Software Design, extremely relevant to this topic. If you don’t have time to read the whole book (though I highly recommend it), here’s a great talk summarizing its philosophy.

By: Amit Pahwa, Cristian Figueroa, Donghan Zhang, Haim Grosman, John Bodley, Jonathan Parks, Maggie Zhu, Philip Weiss, Robert Chang, Shao Xie, Sylvia Tomiyama, Xiaohui Sun

Introduction

At Airbnb, we lean on data to inform our critical decisions. We validate product ideas through randomized controlled experiments, and we track our business performance rigorously to ensure that we maximize values for our stakeholders. To achieve these goals, we needed to build a robust data platform that serves the internal users’ end-to-end needs.

While we have previously shared how we ingest data into our data warehouse and how to enable users to conduct their own analyses with contextual data, we have not yet discussed the middle layer: how to properly model and transform data into accurate, analysis-ready datasets.

In this post, we will share our journey in building Minerva, Airbnb’s metric platform that is used across the company as the single source of truth for analytics, reporting, and experimentation. Specifically, we will set the context on why we built it, describe its core features and the ecosystem of tools it has enabled, and highlight the impact it has had on Airbnb. In upcoming posts, we will deep dive into the technology behind Minerva and share the lessons we learned along the way. By publishing this series, we hope our readers will appreciate the power of a system like Minerva and be inspired to create something similar for their organizations!

A brief history of analytics at Airbnb

Like many data-driven companies, Airbnb had a humble start at the beginning of its data journey. Circa 2010, there was only one full-time analyst at the company working on data, and his laptop was effectively the company’s data warehouse. Queries were often run directly against the production databases, and expensive queries occasionally caused serious incidents and took down Airbnb.com. In spite of the pitfalls, this simple solution helped Airbnb identify many growth opportunities over the years.

As Airbnb’s footprint continued to grow in the early 2010s, more data scientists were brought on to the company and data kept growing both in terms of size and of variety. It was around then that we went through the first phase of changes, upgrading and stabilizing our data infrastructure. We switched from Chronos to our home-grown, now open sourced, Apache Airflow for workflow orchestration and invested in building a set of highly critical data tables called `core_data`.

With `core data` serving as the foundation, analytics at Airbnb began to blossom. First, we brought the culture of A/B testing to Airbnb by building and scaling Airbnb’s experimentation platform. We built an in-house data catalog, Dataportal, to organize and document our data and created, now open sourced, Apache Superset so more users could analyze data independently and interactively. Last but not least, we focused on data education by launching Data University, a program to teach non-data scientists useful skills in an effort to democratize data analysis at Airbnb.

Growing pains

While `core_data` brought several step-function changes to Airbnb’s data capabilities, our success did not come without some significant cost. In fact, the proliferation of data and use cases caused serious growing pains, both for data producers and for data consumers.

First, as `core_data` continued to rise in popularity, more data producers wanted to use it for analytics, forecasting, and experimentation. New tables were created manually on top of `core_data` tables every other day, but there was no way to tell if similar tables already existed. The complexity of our warehouse continued to grow, and data lineage became impossible to track. When a data issue upstream was discovered and fixed, there was no guarantee that the fix would propagate to all downstream jobs. As a result, data scientists and engineers spent countless hours debugging data discrepancies, fighting fires, and often feeling unproductive and defeated.

For data consumption, we heard complaints from decision makers that different teams reported different numbers for very simple business questions, and there was no easy way to know which number was correct. Years ago, when Brian, our CEO, would ask simple questions like which city had the most bookings in the previous week, Data Science and Finance would sometimes provide diverging answers using slightly different tables, metric definitions, and business logic. Over time, even data scientists started to second guess their own data, confidence in data quality fell, and trust from decision makers degraded.

Overcoming our growing pains with Minerva

As these pain points worsened, Airbnb embarked on a multi-year journey to revamp its data warehouse with the goal of drastically improving data quality at the company. As a first step, our data engineering team rebuilt several key business data models from scratch, which resulted in a set of certified, lean, normalized tables that do not use unnecessary joins. These vetted tables now served as the new foundation for our analytics warehouse.

Our work hardly stopped there, however. In order to translate these tables into insights, we needed to be able to programmatically join them together to create analysis-friendly datasets. We needed to be able to backfill data whenever business logic changed. Finally, we needed data to be presented consistently and correctly in different consumption tools.

This is when Minerva — Airbnb’s metric platform — came onto the scene. Minerva takes fact and dimension tables as inputs, performs data denormalization, and serves the aggregated data to downstream applications. The Minerva API bridges the gap between upstream data and downstream consumption, enabling Data Engineering teams the flexibility to modify core tables while maintaining support for various downstream consumers. This API serves a vital role in Airbnb’s next-generation data warehouse architecture.

To date, we have more than 12,000 metrics and 4,000 dimensions in Minerva, with more than 200 data producers spanning across different functions (e.g., Data, Product Management, Finance, Engineering) and teams (e.g., Core Product, Trust, Payments). Most teams now regard Minerva as their preferred framework for analytics, reporting, and experimentation at Airbnb.

Data production in Minerva

From an infrastructure perspective, Minerva is built on top of open-source projects. It uses Airflow for workflow orchestration, Apache Hive and Apache Spark as the compute engine, and Presto and Apache Druid for consumption. From metric creation through computation, serving, consumption, and eventually deprecation, Minerva covers the full life cycle of a metric.

• Metrics definition: Minerva defines key business metrics, dimensions, and other metadata in a centralized Github repository that can be viewed and updated by anyone at the company.
• Validated workflow: The Minerva development flow enforces best data engineering practices such as code review, static validation, and test runs.
• DAG orchestration: Minerva performs data denormalization efficiently by maximizing data reuse and intermediate joined results.
• Computation runtime: Minerva has a sophisticated computation flow that can automatically self-heal after job failures and has built-in checks to ensure data quality.
• Metrics / metadata serving: Minerva provides a unified data API to serve both aggregated and raw metrics on demand.
• Flexible backfills: Minerva version controls data definitions, so major changes to the datasets are automatically tracked and backfilled.
• Data management: Minerva has built-in capabilities such as cost attribution, GDPR selective deletion, data access control, and an auto-deprecation policy.
• Data retention: Minerva establishes usage-based retention and garbage collection, so expensive but infrequently utilized datasets are removed.

The above mentioned features allow us to standardize metric creation, data computation, and data delivering. In the next post, we will deep dive into these features and explain them in more detail!

Data consumption in Minerva

Minerva’s product vision is to allow users to “define metrics once, use them everywhere”. That is, a metric created in Minerva should be easily accessed in company dashboarding tools like Superset, tracked in our A/B testing framework ERF, or processed by our anomaly detection algorithms to spot business anomalies, just to name a few. Over the last few years, we have partnered closely with other teams to create an ecosystem of tools built on top of Minerva.

Data catalog

First, we partnered closely with the Analytics Product team to index all Minerva metrics and dimensions in the Dataportal, Airbnb’s data catalog. When a user interfaces with the Dataportal and searches for a metric, it ranks Minerva metrics at the top of the search results. The Dataportal also surfaces contextual information, such as certification status, ownership, and popularity so that users can gauge the relative importance of metrics. For most non-technical users, the Dataportal is their first entry point to metrics in Minerva.

Data exploration

Upon selecting a metric, users are redirected to Metric Explorer, a component of the Dataportal that enables out-of-the-box data exploration. On a metric page, users can see trends of a metric with additional slicing and drill down options such as `Group By` and `Filter`. Those who wish to dig deeper can click into the Superset view to perform more advanced analytics. Throughout this experience, Metric Explorer surfaces metadata such as metric owners, historical landing time, and metric description to enrich the data context. This design balances the needs of both technical and non-technical users so they can uncover data insights in-place seamlessly.

A/B testing

Historically, Airbnb’s Experimentation Reporting Framework (ERF) had its own experiment metrics repository called “metrics repo”. Experimenters could add any business metric to an experiment and compare the results of the control and treatment group. Unfortunately, the metrics repo couldn’t be used for other use cases beyond experimentation, so we decided to integrate Minerva with ERF so all base events for A/B tests are defined and sourced from Minerva. Using the same source across experimentation and analytics means data scientists can be confident in their understanding of how certain experiments could affect the top line business metrics.

Executive reporting

Long since Airbnb became a public company, we have adopted a practice of reviewing Airbnb’s business performance on weekly, monthly, and quarterly cadences. In these meetings, leaders across different functions meet and discuss the current state of the business. This type of meeting requires executive reports that are high-level and succinct. Data are often aggregated, trends are analyzed and plotted, and metrics movements are presented as running aggregations (e.g., year to date) and time ratio comparison (e.g., year over year).

To enable this type of reporting, we built an eXecutive Reporting Framework (XRF). XRF takes a list of user-specified Minerva metrics and dimensions and turns them into aggregated metric time series that are report-friendly. This framework automates a lot of the manual work and allows us to standardize high-fidelity, business-critical reports by leveraging the same Minerva metrics and dimensions used for analysis and experimentation.

Data analysis

Last but not least, Minerva data is exposed to Airbnb’s custom R and Python clients through Minerva’s API. This allows data scientists to query Minerva data in a notebook environment with ease. Importantly, the data that’s being surfaced in the notebook environment is computed and surfaced exactly the same way as they were in the aforementioned tools, such as Superset and Metric Explorer. This saves enormous amounts of time for data scientists as they can pick and choose the right tool for the job depending on the complexity of the analysis. Notably, this data API encourages lightweight prototyping of internal tooling, which can later be productionalized and shared across the company. For example, data scientists have built a time series analysis tool and an email reporting framework using this API over the last two years.

How we responded to the COVID-19 crisis with Minerva data

As Minerva became a centerpiece of analytics at Airbnb, we saw again and again the power and productivity gain it has brought to the data community at Airbnb. In this last section, we want to give a concrete example of how Minerva aided the business during the COVID-19 crisis.

In March 2020, global travel came to a halt due to COVID-19. Almost overnight, Airbnb bookings plummeted and cancellations skyrocketed. This was a scary moment for us, and it raised many important business questions: How was the coronavirus affecting our nights backlog? How was it affecting our occupancy rate? What was the financial impact of rising cancellations? How had the coronavirus altered travel demand in terms of travel distance? We needed to answer all these questions quickly and correctly.

In response to this influx of inquiries, our data science team gathered the questions and started to brainstorm how we could leverage data to answer them. Crucially, since many important business metrics and dimensions for supply, demand, finance, and customer support were already defined in Minerva, our Central Analytics team was able to wireframe an executive dashboard and roll out the initial version in just under a few days. The COVID-19 dashboard quickly became the single authoritative source of truth and was reviewed closely by our executive team in the midst of the crisis. Since then, it has amassed more than 11,000 views and 1,500 distinct viewers. Not surprisingly, the COVID-19 dashboard was the most viewed Superset dashboard at Airbnb in 2020.

Insights generated from Minerva metrics also allowed the company to confidently pinpoint the rapidly changing landscape. For example, we uncovered market opportunities such as demand shift to local travel and longer-term stays. These findings led us to redesign several important touch points of our product pages to meet the shift in user preference. In moments of crisis, the ability to answer questions and uncover insights is more important than ever. We are able to do this efficiently and effectively, thanks to the single source of truth data in Minerva!

Closing

In this post, we briefly summarized the history of Airbnb’s analytics journey, the growing pains we faced in the last few years, and why we built Minerva, Airbnb’s metric infrastructure. In particular, we covered how data is produced and consumed via Minerva. Toward the end of the post, we also highlighted a recent example of how Minerva helped Airbnb to react to the COVID-19 crisis.

In the next post, we will deep dive into Minerva’s technical architecture, including the design principles, the user development flow, as well as the data computation graph. In the last post of the series, we will introduce Minerva API, which is our single layer of data abstraction that made all the integrations outlined above possible. We will close the series by sharing the lessons that we’ve learned from building Minerva in the hope that these lessons will be helpful for others building similar systems.

Until then, stay tuned for our next post!

Acknowledgments

Minerva is made possible only because of the care and dedication from those who worked on it. We would also like to thank Lauren Chircus, Aaron Keys, Mike Lin, Adrian Kuhn, Krishna Bhupatiraju, Michelle Thomas, Erik Ritter, Serena Jiang, Krist Wongsuphasawat, Chris Williams, Ken Chen, Guang Yang, Jinyang Li, Clark Wright, Vaughn Quoss, Jerry Chu, Pala Muthiah, Kevin Yang, Ellen Huynh, and many more who partnered with us to make Minerva more accessible across the company. Finally, thank you Bill Ulammandakh for creating the beautiful visualization so we can use it as our header image!

Apache®, Apache Airflow, Apache Superset, Apache Hive, Apache Spark and Apache druid are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

Presto is the registered trademark of LF Projects, LLC.

GITHUB® is the exclusive trademark registered in the United States by GitHub, Inc.

All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.

How Airbnb achieved metric consistency at scale was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Habits Are Too Light To Be Felt Until They Are Too Heavy to Be Broken

Recap

In my earlier post, I shared my approach to building self-directed learning projects. I emphasized, before jumping into any project, you should pick carefully and strategically what skills to invest in. In particular, prioritizing skills that are foundational, transferrable, and adjacent to your core competency are some of my criteria for acquiring new professional skills.

I also highlighted the importance of planning — from defining the objectives and key results, decomposing a project into concrete milestones, to surveying and sampling learning materials, these planning exercises can significantly reduce logistical complexity and enable you to deeply focus on learning itself.

In this second post, I will pick up where we left off and describe how you can develop habits and rituals to reliably execute against your plan. Furthermore, We will cover techniques that will further solidify your learnings such as doing drills for procedural learning, building mental models for conceptual learning, and using free recall and spaced repetition to remember facts. By the end of this post, I hope you will be inspired to develop your learning system as many others have.

From Strategy to Execution: Habits and Rituals

While most people agreed with “strategy without execution is useless”, many still fall short when it comes to learning project execution. For example, the chart below shows the view progression for some of the most popular online courses on YouTube. In the aggregate, people tend to drop off quickly just a few lectures into the courses, and many people fail to take their studies to the finish line.

The key to fighting against the odds, in my opinion, lies in the book “Atomic Habit”. The author James Clear argues that all great performers, be them athletes, musicians, and any professionals, have rituals and routines designed to support their high performance. I believe, to get the most out of your projects, you too should design learning habits to support your deliberate practice. These learning habits are the key to finishing a good project.

The Habit Loop

Drawing from a wide variety of research and interviews, Clear develops a framework that describes the mechanics of habit formation. In his words:

The cue triggers a craving, which motivates a response, which provides a reward, which satisfies the craving and, ultimately, becomes associated with the cue. Together, these four steps form a neurological feedback loop — cue, craving, response, reward; cue, craving, response, reward — that ultimately allows you to create automatic habits.

If you analyze some of your habits that are seemingly automatic, they all follow the same habit loop described above. For example, early in the morning, the act of waking up is a cue for you to go to the bathroom, and your mere presence in the bathroom triggers a craving to freshen up. You then react to this craving by brushing your teeth or taking a shower. Once you complete these responses, you feel refreshed and rewarded. In the background, this process completes a habit loop and repeats itself the next morning.

Many high performers understood that this very cycle can be leveraged to design rituals and routines to support their professional pursuits. Michael Phelps trained for 5–6 hours a day in the water, and the consistency made him one of the greatest swimmers in the world. The late Kobe Bryant woke up at 4:30 a.m. every day just so he could get a few extra hours of practice, and this dedication made him one of the most legendary basketball players in NBA history. The pursuit of excellence is never just achieved through will powers, it is also supported by well-designed habits and routines.

The Four Laws of Good Habit Formation

Certainly, putting habits in place is easier said than done, so what can we do to foster these habits? Clear explains, once you understand how habits work, you can follow a simple set of rules to create good habits and even break bad ones. He called these the Four Laws of Good Habit Formation:

• Make it obvious: If you want to reliably kickstart a habit loop, make the cues that remind you about the habit extremely obvious
• Make it attractive: If the triggered cue is associated with a strong craving, you are more likely to follow through with the habit
• Make it easy: For you to execute on the habit itself, the action or response needed to fulfill the craving must be very easy to perform
• Make it satisfying: Finally, to complete the loop, the reward you get from completing the action must be satisfying to satisfy the craving

By understanding these principles, you can now design tactics to make a good habit stick. If you often forget to do a learning project, find cues to remind you of this priority. If you often don’t have enough motivation to return to a project, find ways to make it more desirable. If your projects are too hard to execute on, simplify and make it easy to do. Finally, if your projects don’t give you enough satisfaction, find alternative ways to make it rewarding.

Engineer Habits & Routines For Learning Projects

Ever since I learned about the Four Laws of Good Habit Formation, I have become more intentional about how I execute my projects. While my previous learnings succeeded because I pushed myself hard, my more recent projects succeeded because of the routines that I designed. To make it concrete, I will use a Machine Learning project that I completed in 2018 as a case study.

First, I would encourage you to read my Project Plan Document to understand the motivation of this project. Next, you can read through my detailed time log to see my learning progression by the week. With those contexts in mind, I will go beyond what I have written down on Github and highlight the habits and routines that I practiced throughout the project.

Make It Obvious: Use Implementation Intention As My Cue

Like many other brave souls in Silicon Valley, I spend a few hours a day in commuting. Recognizing that my time is valuable, I was determined to not waste it. Each morning, as I stepped onto the waiting platform, I would tell myself that today is a new day, and I will re-engage with my learning projects for 50 minutes until I arrive at San Francisco. The act of pre-committing to a specific action at a specific time and a specific place is a goal-setting technique called Implementation Intention.

This technique has been used widely in many situations and has been proven to be extremely effective. The reason that it works is that it removes the decision making ahead of time and lowers the activation energy to kickstart a routine. When you pair this technique with an environment where you regularly interact, you effectively design a recurring cue that serves as a powerful reminder to prioritize what is important to you in that environment.

Note, interruptions and disruptions do occur in life: e.g. the train could be late, you could feel sick every once a while, or you might not be able to find a seat on the train because it’s too crowded. These things do happen, and that’s ok. As long as you treat your implementation intention seriously, you can adapt and still follow through your commitments.

Make It Attractive: Increase Craving Through Temptation Bundling

Suppose the train is on time, and I found a good seat, it can still feel like a struggle to dive right back into the learning materials. In these situations, a technique called Temptation Bundling can be very handy. At its core, the idea is to link behavior that you need to do with a behavior that you want to do. By bundling the hard thing with the pleasurable thing together, the hard thing becomes more attractive.

As a concrete example, grokking intellectually challenging concepts in Deep Learning isn’t always the easiest. On the other hand, I always really enjoy writing a neat learning summary whenever I finish a new concept, as I find that process soothing. By bundling these two activities, I convinced myself that if I could engage with the materials and learn a new concept (what I need to do), I will be able to produce a well-written summary by the end of the session (what I want to do). Furthermore, the fact that these two activities are sequentially related, means that bundling can be particularly effective.

Not every person struggling with Deep Learning is obsessed with well-written documentation. The general lesson here, however, is to identify what things you need to do, what things you want to do and think of ways of linking some of them together. By going through this design exercise, you can drastically improve your motivation to carry your routines forward.

Make It Easy: Remove Logistical Complexity Ahead of Time

Even when I am already engaging with my learnings during my commute, there could still be challenges. As an example, if I am working through a MOOC, I generally do not like to consume video lectures on my tiny phone screen. In addition, the internet connection can be unstable and the streaming experience can be sporadic, the whole experience can feel inconvenient. These logistical complexities can easily creep up on me and kill productivity. As a result, I decided to simplify and remove these complexities ahead of time.

For example, to overcome the inconvenience, I always pre-download the video lectures to my laptop before my commute. Once I am on the train, I could then play the lectures in 2X the speed, or even play back and forth without any internet interruptions. Furthermore, the logistics of deciding what materials to download on a given day is also solved, because I can go back to my learning schedule and my time log to identify which lessons to pursue next. The preparation process is lightweight and seamless.

Depending on the task at hand, there could be many ways to make it easier. Tactics such as starting small, reducing scope, and leveraging commitment devices are all worth trying when designing your routines.

Make It Satisfying: Using Github to Keep Track of My Progress

Finally, to celebrate the progress that I made, I used a combination of visual cues and public artifacts to reward myself. For example, I use Github’s commit heat map to visually track my progress. As much as I know the number of commits is a vanity metric, being able to visualize them at a glance is still very powerful. Jerry Seinfeld, the legendary comedian, uses this very same technique to motivate himself to write jokes every day:

“After a few days, you’ll have a chain. Just keep at it and the chain will grow longer every day. You’ll like seeing that chain, especially when you get a few weeks under your belt. Your only job is to not break the chain.”

While never breaking the chain is very difficult, Clear proposed a slightly more lenient version called Never Miss Twice, which I find both practical and empathetic. The idea is the following — it is acceptable to break the chain every once a while, but when you do, never miss your routine twice in a row. It is not just about adhering to a routine, but it is also about how fast and how consistent you can get back on track. Grit is the key to success.

Designing good learning habits takes time and thoughtful thinking. But once you have the right designs in place, they are not as hard to follow as you originally imagined. By following the strategies derived from the Four Laws of Good Habit Formation, you can greatly increase the odds of completing your work to drive high performance.

Transform Learnings Into Tools In Your Toolkit

Without continuous training and consistent practice, over time, even the greatest athletes and musicians would experience skill deterioration. This is even more true among knowledge workers, many of whom “practice” and “training” are less tangible and more abstract.

For example, even for some of my well-executed projects, without further practice, I tend to forget about 30% to 50% of what I initially learned in the following six months. Without more investments, knowledge and skills that are fresh to our memory can go away rather quickly. The good news is, depending on whether you are trying to learn procedures, concepts, or facts, we can adopt different strategies to extend the half-life of our learnings.

Procedures: Find Opportunity To Do “Practice Drills” On the Job

One strategy to effectively sharpen your skills is to directly practice and apply that skill on the job. Not only will you be able to correlate your current level of expertise with the quality of your output, but this strategy will also create urgency and accountability that vehemently forces you to do your work better.

In the early days of my self-directed learning journey, I largely saw these projects as standalone intellectual exercises — whether I applied the skills or not is not that consequential. Nowadays, I see self-directed learning as an investment in my career, and there is no better feedback than to see how well I can deploy these skills to facilitate my work. As an example, after my push to deliberately improve my Python skills, I forced myself to conduct analyses at work in Python to see how efficient I could be. Similarly, I took on projects where reading and writing Python code is essential to completing the work. I also proactively studied how Object-oriented Python is used to design some of Airbnb’s Data Engineering frameworks. Every activity deepened my muscle memory and made me more familiar with the core skills involved.

The road to mastery is a long one, and it is only through many practices will we get closer. Given that we spend a disproportionate amount of time at work, identifying work projects that will take your skill to the next level on the job is a high leverage exercise. At times, this might give you a feeling of defeat or a temporary loss in productivity, but by enduring the discomfort and working through the feedback, you will get better at these procedural learnings.

Concepts: Build Mental Models And Teach Others

Have you ever experienced a eureka moment where, all of a sudden, many ideas and concepts you previously struggled with now started to make sense? I certainly have, and I suppose these “aha moments” are not uncommon among avid learners. When these moments arrive, it is critically important to slow down and capture those thoughts. Here is a process that I recommend:

• Write those insights down: As it has been said, writing is thinking. By writing things down, it helps you to develop a logical narrative of your understanding, and it helps you to confront fuzzy ideas and half-baked thoughts. Writing is one of the most powerful ways to drive clarity.
• Validate your mental models with experts: With your mental models written down, share them with subject-matter experts who, in terms, have their mental model of the same subject. By learning and comparing different mental models, you can gut check if you are on the right track.
• Voraciously teach others what you learned: After confirming your intuition, find opportunities to teach and share your mental models. By articulating and sharing your perspectives, the power of teaching and explaining will help you to refine what you have learned.

The process above is a powerful way to translate amorphous understanding into concrete mental models. As an example, when things started to click for me for Data Engineering, I wrote down my thoughts and shared them with other experts in the company to get feedback. I summarized them in a series of blog posts and I found every possible opportunity to teach new hires about this topic. By applying my mental model in various situations, these exercises helped me to draw an increasingly clear mental picture in my mind.

Facts: Use Free Recall, Spaced Repetition, and Interleaving

Despite living in a world where facts are just a few keystrokes away, active memory recall can still be very useful in many situations. For example, being able to retrieve known facts from your brain can help you triangulate and validate new information on the spot. In a debate, being able to cite relevant facts can make your argument more compelling. Being able to remember facts well is quite underrated in this information age. When it comes to fact-based learning, I have found the following techniques quite useful:

• Free Recall: After consuming some new materials, immediately force yourself to summarize the main points from memory without looking at it
• Spaced Repetition: Spread out your exercises over time, repeat them often and do not force the materials onto yourself in a short cramming session
• Interleaving: Mix several topics together and use free recall and spaced repetition together. Instead of reviewing topic A & B in a sequence “AAAAAA” and “BBBBBB”, interleave them with “AAA-BBB-AAA-BBB”

For example, when trying to digest facts and ideas from a book, I often found free recall a more effective technique than sentence highlighting. By closing the book and forcing myself to reiterate the main points, I often realized how incoherent I was or how little I could explain these points the first time around. However, by repeating this retrieval process several times, and connecting them with other related materials (of which I also practice free recall), these spaced-out, interleaved exercises helped me to internalize facts much more effectively.

Closing

Warren Buffett, also known as the “Oracle of Omaha”, is one of the world’s most renowned investors and philanthropists. In addition to those two public roles, he always considered himself a teacher. Notably, he always advises young people to seek role models whose qualities and traits they admire and encourage them to make those qualities as their own.

To me, in addition to Buffett’s whimsical humor and strong integrity, what I admire him the most, is not his wealth, but his consistency. He still lives in the modest house he bought in 1958, he still drives by the same McDonald’s for breakfast every day, and he continues to conduct his life with the same principles and philosophies. His long-time investment partner and friend Charlie Munger calls him an absolute “lifelong learning machine”, and I suspect he did not become one without his habits for learning. He once said:

[Chains of habit are] too light to be felt until they are too heavy to be broken

I wrote this series partly because I believe many of us have a strong desire to learn and grow, but I also believe we do not discuss enough how to design and build systems and routines to support our learnings. I hope by sharing some of my own experiences and research from others, you too can be inspired to build your system and become a lifelong learning machine!

Happy learning!

A Little Bit of Slope Makes Up For A Lot of Y-Intercept

Motivation

Recently, I came across a blog post written by Simon Hørup Eskildsen on how he approaches reading. It was an inspiring read because I did not know anyone this deliberate about the pursuit of reading. This kind of meticulous, thoughtful, system-level design thinking reminds me of Ray Dalio’s Principle:

Think of yourself as a machine operating within a machine and know that you have the ability to alter your machines to produce better outcomes

While I do not see myself as a machine, I did examine various aspects of my life and found that I tend to apply this type of thinking more rigorously when it comes to learning. In graduate school, I experimented with various studying strategies to make my day more effective. At work, I have been asked by my coworkers how I managed to find time to pursue learning projects.

Like many others, I am easily distracted by Netflix and are often bombarded by social media like Twitter and Facebook. Nevertheless, over the years, I have designed a system to make my self-directed learning easier and more effective. In this post, I will share my approach to building technical learning projects, and I hope it will inspire you to design your own as well. If you already have a system that works well for you, I would love to hear it!

A Bit of Slope Makes Up For A Lot of Y-Intercept

Before I go into the details of my still-evolving system, it’s useful to first talk about why I even pursue learning projects in the first place. Among my many reasons, the most important one came from Stanford professor John Ousterhout. In his “Thoughts for the Weekend” series, Professor Outsterhout once shared with his students a concept called “A little bit of slope makes up for a lot of Y-intercept”.

If you have two lines, the red line and the blue line, and the red line has a lower Y-intercept but a greater slope […] then eventually the red line will cross the blue line. In a mathematical sense, it’s kind of obvious [a little bit of slope makes up for a lot of Y-intercept].

I think this is a pretty good guideline for life. What I mean is that how fast you learn is a lot more important than how much you know to begin with. So in general I say that people emphasize too much on how much they know and not how fast they’re learning.

When I took my first Data Science job in 2012, I barely knew git, SQL, and have never heard the term Data Engineering. I didn’t know how to design good experiments, and I know very little about Machine Learning beyond math and theory. Arguably, my Y-intercept was pretty low, but I focused on my slope anyway.

I love this philosophy, because it not only gives me more motivation to keep improving myself, but it also helps me to stay humble when working with less experienced people. We are all working along with our version of the slope, and in the long run, your learning rate is what determines your long-term success.

In this two-part series, I will talk about various strategies that I adopted along my journey climbing my slope. These include, but are not limited to:

• How to choose a useful learning project
• How to design and iterate on a learning plan
• How to execute on a learning project persistently
• How to internalize my learnings and make it useful for others

This post, part-I of the series, will explain several criteria I used when it comes to subject picking. Furthermore, I will talk about how to effectively design a learning plan, and explain why it is important to spend time doing so. In Part-II of this series, I will discuss the ingredients of good habits and how to leverage them to learn persistently. Finally, I will discuss why teaching and writing are the best ways to internalize what you have learned.

Let’s dive right in!

1. Choosing a Learning Project

In an era where learning resources are easily accessible via MOOCs, Youtube, and blog posts, the question is now less about accessibility but more about what subjects to learn. Some advocates to learn what you love until you love learning, others suggest learning skills that can withstand the test of time. My strategy, especially for developing professional skills, is heavily influenced by Scott Adams, the creator of the Dilbert comic strip.

If you want an average successful life, it doesn’t take much planning. Just stay out of trouble, go to school, and apply for jobs you might like. But if you want something extraordinary, you have two paths:

1. Become the best at one specific thing.
2. Become very good (top 25%) at two or more things.

The first strategy is difficult to the point of near impossibility. Few people will ever play in the NBA or make a platinum album. I don’t recommend anyone even try. The second strategy is fairly easy. Everyone has at least a few areas in which they could be in the top 25% with some effort […] Capitalism rewards things that are both rare and valuable. You make yourself rare by combining two or more “pretty goods” until no one else has your mix.

I adopted the second strategy because I enjoy having breath and diversity in my skill sets. As such, I spent a lot of time thinking about what my repertoire of skills should be, and they generally boil down to skills that are foundational, adjacent, and transferable. Let me explain each of them below.

Prioritize Foundational Skills

In the present world, the rate of technological change is often far too fast for any single person to digest. In a growing field where many people are contributing and competing, there is often an unending streak of new updates and announcements. As an example, in Deep Learning, we often hear debates about which learning framework will be the lingua franca of AI. Many of the technologies in debate today were not even popularized until a few years ago.

With this rate of change, I argue that the ability to use any particular framework or tool is not by itself a foundational skill. Instead, it is the workflow patterns that these tools enabled that are foundational. Frameworks and tooling are often useful because they abstract away tedious parts of our workflow, but without knowing why we are following these workflows in the first place, we are simply using the tools mechanically without souls.

I tried to follow this principle when pursuing my learning projects. For Deep Learning, instead of obsessing over whether to use Keras or PyTorch, I spent most of my time understanding the conceptual difference between shallow models v.s. deep models. I learned why Transfer Learning is a standard workflow in the era of deep modeling, and I investigated why certain embeddings are powerful for different learning tasks. Similarly, when I practiced Data Engineering, I looked beyond Airflow’s simple building blocks such as sensors, operators, and transfers. Instead, I tried to understand the core activities involved in doing good data engineering work — data modeling, backfilling, testing and monitoring, … etc.

Over time, I learned that by mastering the core concepts associated with a workflow, the mental model one develops can be easily re-applied when picking up new tools and frameworks. This is perhaps why great engineers can typically pick up new languages without too much struggle — they understand the core concepts underneath that are universally foundational.

Explore Adjacent Skills

Once I feel I have mastered a particular skill, I often expanded it by tapping into adjacent discipline, a term coined by Steven Sinofsky and advocated by Edmond Lau. Accordingly to Steven, adjacent disciplines as those that are immediately to the left or right of your own core expertise. For example, if you are a data scientist, your adjacent disciplines might be Data Engineering or Data Visualization. As a product manager, your adjacent disciplines might be A/B testing or product design. As a musician, your adjacent disciplines might be music composition or music production, so on and so forth.

The first obvious advantage of exploring adjacent skills is that it will make you more self-sufficient on the job. If you are a data scientist who needs a feature pipeline for machine learning, having the skill to build your own ETL pipeline means that you can iterate on feature engineering more efficiently. If you are an analyst who needs to include more dimensional cuts for a report, having the ability to modify table schema can help you to deliver results faster. This is why I believe data scientists who learned data engineering can usually take on bigger and more ambitious data projects.

Aside from this first benefit, learning adjacent disciplines often mean your knowledge compounds. Having learned Python and Object-Oriented Programming, these skills enabled me to understand better how different Data Engineering frameworks are implemented under the hood. By learning the challenges of building ETL jobs, I can appreciate much better why feature engineering in Machine Learning can be so time-consuming. Adjacent disciplines not only help you to transition into new areas, but it also helps you to connect related knowledge together.

The last and more subtle advantage of learning adjacent discipline is that it makes you a better cross-functional partner. By understanding the complexities and intricacies of your colleagues’ works, you can better empathize with the challenges that they face on the job.

Focus on Transferable Skills

A few years ago, I was interested in building a website using Flask and Python. While dabbling into the world of web development, I came across a template engine called Jinja. I soon learned that it is commonly used by developers because it simplifies writing HTML significantly via control flow and inheritance. I was very impressed by the ingenuity of this template engine at the time, but I never really used it again as a data scientist, not until when I arrived at Airbnb.

At Airbnb, I learned from experienced data engineers that many of the ideas in Jinja can be applied to Hive queries directly. By leveraging for-loop in Jinja, I can greatly shorten the SELECT statement. By using the if-else control flow, I can incorporate backfilling logic into the same SQL query.

When thinking about writing HTML and writing SQL, I realize that there’s a lot of similarity between the two activities. Both workflows leverage an expressive language to achieve certain tasks: HTML for rendering web pages, SQL for data computation, so it wasn’t surprising to see that the same technique can be transferred from one domain to another.

Transferable skills are like superpowers — learned once, and you get to apply it in several places. The initial cost of acquiring those skills might be high, but the variable cost for applying it elsewhere is relatively low. When thinking about what skills to master, focusing on transferable skill.

Takeaways

• Prioritize foundational skills: you will learn activities associated with a workflow and appreciate how tooling can abstract them away for you
• Explore adjacent disciplines: you will become more self-sufficient on the job, can take on larger and more ambitious projects, and develop more empathy for your colleagues
• Focus on transferable skills: you can apply these skills in different domains without much extra costs

Our society values what’s rare and valuable, and by being selective of what to learn, you are more likely to develop your own mix that is unique and not easily replaceable.

2. Designing a Learning Plan

Once you have chosen a specific skill to focus on, the next step is to develop a learning plan. In my opinion, the biggest mind shift here is that you need to treat yourself not only as a student but also as your own teacher. Playing these dual roles can seem a little bit daunting at first, especially if you are used to learning from an authority figure like a teacher or professor.

However, imagine what a teacher has to do before teaching a course. She has to define learning objectives for her students first. He most likely needs to sample different learning materials and develop a curriculum. Finally, she would need to set milestones to make sure students are held accountable.

Similarly, as a designer of your learning project, you will also need to go through these exercises to design an effective learning plan. Let us go over each of these activities in more detail.

Define Your Learning Objectives

One of the mistakes that I made in my earlier learning projects was that I rarely stated upfront what learning objectives I would like to achieve. I dived into one learning project after another haphazardly, with the hope that someday I will apply these skills to some problems — It never happened.

It does not hurt to be biased to action, but following unguided actions also means you could spend a lot of energy wandering around aimlessly. Reflecting on the lessons learned from earlier failures, I now try to identify one or two learning objectives upfront before diving into a learning project.

For example, when I pursued my Python learning project, I wrote down the learning objective that I want to eventually converge all my Data Science work from R to Python by the end of the calendar year. When pursuing my Deep Learning project, I set the goal to understand how Airbnb’s room classification model works under the hood. The key here is to set concrete, time-bounded, measurable goals. i.e. I can check exactly if I have achieved what I set out to do by a specific deadline.

For technical learning projects that are aimed to improve my professional skills, my learning objectives usually revolve around:

• adopting or making a specific change in how I work
• achieving a better understanding of a problem or solution, or
• generating learning materials that can be consumed by others

I will talk more about generating public work in Part-II of this series.

Identify Project Milestones

If your learning objective is relatively ambitious, then it is important to break down your learning objectives into tangible and achievable milestones. This “divide and conquer” approach has proven to be very effective for me.

As a concrete example, for my Flask learning project, I defined a learning objective to deploy a website that will render my Google calendar time logs in an interactive d3 visualization. I identified specific milestones for this goal: from reading the Google Calendar API, pulling and storing my data in a local database, to finally rendering the data in a d3 visualization. After completing many small milestones, I finally built a website that did exactly what I set out to you. I still remember the day when I rendered all the data on a heat map, it was magical!

Develop a Curriculum

In my experience, one of the most important but under-appreciated skills in learning is the ability to identify materials that are accessible and approachable to your learning patterns. For most of my learning projects, I usually spent about 2–3 weeks sampling materials until I develop a curriculum that I feel excited about. This is the period where I would browse the internet to identify any free or paid materials that are relevant to my topic — be it blog posts, online courses, books, or the like. I typically do this by creating a new github repository to keep track of all the materials that I find relevant, and slowly organize them into something more structured like below:

There are two reasons why sampling and developing your curriculum is important. First, without a very strong mental model, it is hard to know exactly what are the important concepts to learn. By standing on the shoulders of people who understand the materials very well, you will start to see the same core concepts emerging again and again. In my opinion, this is the first step toward understanding.

The second reason is that as a life-long student, you are no longer constrained by the textbook or teaching style of any single teacher. Depending on your background, you might want to mix and match between different learning materials that help you learn better. Over and over again, I have seen how the same concept can be explained horribly but also articulately, and it makes a world of a difference. It’s up to you to find the resources that resonant with your learning patterns.

As an example, during my Deep Learning project, I simultaneously sample materials from Stanford’s on-campus course, Coursera’s Deep Learning specialization, and a fast.ai course. I generally find Coursera’s material great for developing intuition at first, but almost always gravitated toward the Stanford course because I want to understand the math better. Furthermore, to practice with more hands-on exercise, I would usually go to the fast.ai course, where notebooks and code examples are abundantly available. It was quite common that I would study the same subject using three different materials in a given week.

One final note, if you are not yet confident in your ability to source your own learning materials, be sure to solicit feedback from friends or coworkers who are knowledgeable on the subject. This is a good way to make sure your curriculum has covered all the basic grounds.

Takeaways

When it comes to self-directed learning, it is very important to design a learning plan. This includes:

• Define your learning objectives: set clear goals on what behaviors or outcomes you would like to achieve
• Identify project milestones: divide and conquer so you can make steady progress without getting intimidated
• Develop a curriculum: find materials that you would enjoy according to your learning patterns. Consult experts to validate your curriculum

By the end of this design process, you would have a pretty solid learning plan, and that’s really half of the job done! The common mistake is that people generally do not spend enough time in this step.

Conclusion

Even if you have a strong desire to learn, it often still feels like a struggle to balance between self-directed learning and other important life priorities, just think about all the responsibilities, obligations, and distractions in life!

Without a system, chances are, we will not achieve our goals. In this post, I shared my approach to choosing and building a learning project, using my previous successes and failures. I hope by sharing some of my learnings, you will be inspire to design your own learning methods.

In the next post, I will talk about how to make steady progress on a learning project, and how to internalize your learnings by teaching others and producing public artifacts.

Happy learning!

How to think about the IC to Management transition

Motivation

I recently came across a great book called “The Manager’s Path”, written by Camille Fournier. In this book, Camille describes the possible career paths of a technical individual contributor (IC) as one continues to grow in her career. I find this book illuminating because it touches on a rather common topic in the tech industry — “Should I transition from IC track to management track? If so, when is the right time?”. If you have been working as an IC for the past few years, chances are, you have pondered on this question already.

Given that this topic is discussed so extensively in engineering (see here, here, and here) and design (see here, here, and here), you would think it is all figured out by now. However, for a field as new as data science, I haven’t seen a whole lot of discussions about this topic just yet. After working in the field for more than half a decade, I have developed my own perspectives on how to think about this topic, so here we go!

First, My Experience

I am not a data science manager, nor have I ever been one. During the past six years of my data science career, I have definitely considered becoming one, and have even been encouraged to. However, like any other important career decision, I am always convinced that this transition, if made, should be a deliberate one. Far too often, I have seen ICs turned managers for the wrong reasons, which made them and their teams suffer.

Recently, I worked with our data science leadership team to build the first data science tech lead pilot at Airbnb. Through various discussions with ICs and managers, we debated the pros and cons of staying on the IC track v.s. going into management. These conversations not only helped me to launch the pilot but also helped me to contextualize some of my own, often implicit, preferences.

On the one hand, I have always aspired to become a hands-on data scientist, and enjoy doing so from the trenches. On the other hand, I sometimes experience the challenge and inertia of leading without authority. Some people say that there’s diminishing returns for remaining as an IC for too long, and other managers have lamented how far they are drifted away from technical work. While all of these might ring true, I wanted to dig deeper to my inner compass.

The Framework From The Book “Drive”

A few years ago, I came across Daniel Pink’s bestselling book “Drive: The surprising truth of what motivates us“. In his book, Daniel talked about three important ingredients that make our works motivating:

• Purpose: the goal to do something meaningful
• Mastery: the urge to get better at stuff
• Autonomy: the desire to be self-directed

Since then, I have been using his framework for evaluating projects, jobs, or more generally career opportunities. In the next couple of sections, I will again use this framework to talk about the pros and cons of being an IC v.s. becoming a manager.

Purpose

In a well-run startup, everyone knows exactly what the company is trying to achieve — the cost of communication is low, and decisions are made swiftly. As the company continues to grow, communication and decision making become more localized, and misalignments could occur among people who do not communicate effectively. This is why we often hear the phrase “the first casualty of growth is communication”.

In a sufficiently large company, the purpose of managers is to clearly understand the goals of the company and consistently reinforce the company’s vision in order to align teams and individuals to work toward that vision in a fulfilling way. Very different from the role of an IC, a manager’s responsibility is to synthesize information, to provide context, creating the right environment, and do so consistently to create trust.

As described in Julie Zhou’s “Unintuitive Things I learned About Management”, one of the unintuitive things she learned as a manager is that it’s a lot more important to focus on the “why are we doing X?” rather than “how are we going to execute on X?”. The “why” is built on information sharing, trust, and alignment.

This is why management is a career change, not a promotion (see here). If you are ready to serve the mission by amplifying others, by coordinating with partners, and living by the company’s vision, then the managerial path might be a great fit for you. If not, then you should think carefully if you are serving the right purpose.

Mastery

This leap and transition from IC to management track is far greater than just a title change. It’s a fundamental shift of one’s core identity and how one functions. Your technical skill is no longer your most valuable currency, trust is. Code is no longer what you produce, it is relationships. Finally, be prepared to spend a lot of time talking, meeting, and writing Google docs with other people.

To adapt to this new reality, you need to develop soft skills with a higher bar — communication, active listening, empathy, … etc. Many ICs, especially the most effective ones, find this transition challenging, because getting better at these skills is often a lot more amorphous than say, writing more code or conducting more rigorous analyses. The path of mastery is fundamentally different.

In addition to learning new skills, you have to learn how to let go of skills that you have already mastered. You need to know how to delegate effectively and not spend too much time dwelling on execution. If you cannot change this mindset, you severely limit the potential of your team and their output. Paradoxically, the right behavior of a good manager is not to try to be the most technical person in the room.

When you learn new skills, you put other skills in the background. This is natural given the shift of focus. Having said that, this combination of needing to let go of the skills you have already mastered and expanding new skills beyond your comfort zone is a bitter pill to swallow. But, if you are ready to embrace new challenges, this can be an incredible opportunity building foundational and transferable skills.

Autonomy

Finally, as a manager, your schedule will also look fundamentally different. On this topic, I found Paul Graham’s essay “Maker’s Schedule, Manager’s Schedule” extremely relevant. Paul explains that “makers” need concentrated and uninterrupted time to create, usually in half-day chunks. Managers, on the other hand, typically jump in between meetings in hourly chunks.

As a manager, you largely need to keep your calendar open so you could have 1:1s, context sharing, communication, alignment, or even fire fighting. This schedule is a pretty big departure from what ICs are used to, and mixing the two schedules can lead to very poor productivity.

If you are someone who deeply subscribed to the philosophy of deep work, you probably strongly prefer the maker’s schedule. As a manager, you need to be more creative and creating the equivalent of deep working time for yourself. I have heard stories of how CEOs of large companies typically block off time so they can “think”. Deep work in the context of managerial work is possible, but you need to create that space for yourself.

The Concept of a Tour of Duty

Reid Hoffman, cofounder of LinkedIn, proposed the concept of a Tour of Duty in his 2014 book “The Alliance”. He characterizes a tour of duty as: a commitment by the employer and employee to a specific mission of finite duration. Transitioning to management, unarguably, will be a big shift, but I think when looking through the lens of the concept of a tour of duty, it is actually quite liberating.

For young working professionals (myself included), your first rodeo in management does not need to be a permanent one. A much healthier and liberating way to look at this transition is to think of it as a tour of duty. Think of it as a mission that is tailored for you in which you can learn orthogonal skills that are different from the technical skills you already have, and think of it as another way for you to provide value to the organization.

As you approach the end of the tour, you can work with your manager to analyze if such a tour is the right fit for you and for the company. It might not work out for you, but even then you have eliminate this career path, which is very useful for career planning. But if you do like it, new doors are opened for you.

Summary, Not Conclusion

It is probably clear by now that I am not being very prescriptive in this post (I myself am still trying to figure out). The lesson I am trying to preach, perhaps, is to recognize that your goals and aspirations will change over time, and that’s completely normal. The important thing is to make deliberate and informed choices that are aligned with your inner compass as transitions are made, and think from first principles on why you want what you want. It’s hard, but it’s worth trying.

Are you ready to become a data science manager?

So You Want Become a Data Science Manager? was originally published in Deliberate Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Motivation

In A Beginner’s Guide to Data Engineering series, I argued that academic institutions typically do not teach students the proper mental models when it comes to analytics workflows in real-life. Far too many classes only focus on the mechanics of data analysis without teaching concepts such as ETL or the importance of building robust data pipelines. Unfortunately, I see a similar pattern in machine learning education as well. Surely, studying the math behind ML and learning different algorithms are valuable. Yet, there exist crucial steps beyond model.fit(X, y) that are important in practice. In this post, I will share some of my learnings that I did not learn in school.

First, I will highlight the rise of Kaggle: why it has transformed our industry, what critical role it plays, but also where it fell short. In particular, I will contrast the workflow Kaggle reinforced with the typical development workflow of a real-life machine learning project. Throughout the post, I will give coloring examples around topics such as problem definition, feature engineering, model debugging, productionization, and feedback loops. By the end of this post, I hope readers will appreciate some of the complexity and challenges, but also joys, of real-life machine learning.

The Rise of Kaggle Competition

Ever since its inception in 2010, Kaggle has become the platform where data enthusiasts around the world compete to solve a wide variety of problems using machine learning. Over time, Kaggle has built an incredible repository of useful benchmark datasets and example notebooks (called kernels), turned modeling into a sport, and made some practitioners into Kaggle stars.

Common Task Framework

The model that Kaggle follows, is what Professor David Donoho referred to as the Common Task Framework (CTF) in his paper “50 years of Data Science”. Donoho argued that the secret sauce to machine learning’s success is partially driven by competitions:

It is no exaggeration to say that the combination of a predictive modeling culture together with CTF is the ‘secret sauce’ of machine learning. This combination leads directly to a total focus on optimization of empirical performance, which […] allows large numbers of researchers to compete at any given common task challenge, and allows for efficient […] judging of challenge winners.

Indeed, from DARPA’s machine translation research in the 1980s, the famous 2009 Netflix Prize, to the recent success of deep learning due to ImageNet challenge, the machine learning community continues to bring innovation to the mass under common task framework.

Where Kaggle Competition Fall Short

While Kaggle competitions have been tremendously educational, their workflows generally only reflect a small subset of what is involved in real-life machine learning projects. First of all, Kaggle hosts formulate the problems, not the participants. Not only are the loss functions and golden datasets used for evaluation pre-determined, but training labels / data are often handed to the participants on a silver platter. Furthermore, there is very little concern regarding how to integrate models into a decision process or a product. These are all conditions real projects are unlikely to meet in practice. As a result, a lot of the considerations in machine learning projects are lost in translation.

Machine Learning Workflow

When building a machine learning product, we are no longer developing models in isolation (what people sometimes called “Laptop Data Science”). Rather, we are building a system that interacts with real human beings. Not only do we need to think strategically about the problems we are solving for the end users, but we also need to ensure that the user experience is intuitive, predictions are accurate, and inference is efficient.

Think and Build a System End-to-End

These conditions mean that we almost never jump to modeling immediately, we have to think and build the system end-to-end. In Rachel Thomas’ fantastic post “What do machine learning practitioners actually do?”, she explains the typical workflow of a machine learning project:

Building a machine learning product is a multi-faceted and complex task […] machine learning practitioners may need to do during the process: understanding the context, preparing the data, building the model, productionization, and monitoring. […] Certainly, not every machine learning practitioner needs to do all of the above steps, but components of this process will be a part of many machine learning applications.

Kaggle is an amazing platform that focuses on model building, but less so on the rest of the steps described above. To help readers to better understand these other topics, I will highlight them using a combination of my personal experience and illuminating examples from other companies that I find useful. Below, I will discuss:

• Problem Definition: why thinking hard about your problem is crucial
• Data Collection: why setting up your {X, y} right is half of the job done
• Model Building: how to debug your model when it does not perform well
• Productionization: what “putting model into production” really means
• Feedback Loop: how unintended feedback loop can affect your system

1. Defining Problem Is Hard and Not Always Obvious

Let’s start with a case study, Airbnb Plus, a product whose mission is to bring high-quality homes to the Airbnb marketplace. While many employees are passionate about finding homes suitable for Plus, doing this at scale can be challenging. On our team, we use a combination of human evaluation and machine learning to identify high potential homes. This type of problem, which involves human evaluations + machine predictions, are becoming increasingly common.

Your First Iteration of the Model Is Often Not Your Last

As our human evaluators assess homes, training labels are generated as a by-product. Given that we already have a lot of features about a listing (price, bookings, reviews … etc), it was rather convenient to combine the two data sources (labels + features) to train our first home targeting model. At first, this approach worked well, and it brought enormous gains to our efficiency.

However, as the product continued to evolve, we started to experience the limitation of this simple approach. Specifically, as we evolved what qualifies a home to be “Plus” at the program level, the semantic meanings of our outcome labels had also changed. This business evolution posed non-trivial challenges to our learning task because our labels can become outdated rather quickly. We were essentially learning to predict a moving target!

Decomposing the Learning Task Requires Thinking

To de-risk our modeling effort, we had to re-think the problem formulation. Eventually, we decided to decompose our single, monolithic learning task into several independent modular tasks. This means that instead of directly classifying whether a listing is high potential or not, we focused on predicting more stable attributes that are indicative of high-quality. For example, instead of classifying the label is_home_high_potential directly, we framed the problem as is_home_high_potential = f(style, design, ...) , where f is our rule-based approach to codify how humans might use these modular predictions to make a final assessment.

More often than not, problem formulation requires deep domain knowledge, the ability to decompose problems, and a lot of patience. The most convenient training dataset should not drive how we formulate the problem, rather it should be the other way around. This is an important first skill to becoming an effective problem solver in machine learning.

Takeaway: Like software engineering, the principle of decomposition can be very important in machine learning as well. It allows us to break a complex problem or system into parts that are easier to conceive, understand, and learn.

2. Data Collection Is Often Non-trivial

At work, our data scientists and ML engineers often get together to talk about machine learning ideas passionately. While these discussions are always inspirational, they generally do not translate to project roadmaps immediately due to a common blocker — lack of training labels and feature pipelines.

Acquiring Quality Labels Is Challenging

On Airbnb Plus, we are lucky to have training labels generated as a by-product from our home assessments, but dedicated labelings are often rare because collecting them comes with a hefty time and monetary cost. In the absence of actual training labels, we could use other data as proxies for training labels, but they are not always high fidelity.

For example, when Airbnb developed its room classification model, we used image captions as our proxy label for the ground truth. While this approach gave us convenient labels as a head start, the label quality tends to be low for certain room types, especially for smaller spaces. For instance, scenes in a studio tend to be crammed together: a kitchen could be right next to a living room that is adjacent to a bedroom. This makes evaluation of ground truth hard to interpret, sometimes even in the eyes of human labelers.

In general, labeling in real-life is far trickier than simple tasks like telling apart hotdogs v.s. non-hotdogs. This nuance often makes inter-rater agreement hard to achieve, and is rather universal for serious modeling pursuits across different domains. Andrej Karpathy, in his talk building the software 2.0 stack, highlights some of Tesla’s labeling challenges for building self-driving cars. For example, he explains that labeling traffic lights and traffic lanes can sound simple but in reality difficult because the diversity of how different cities design the roads. More generally, he argues that in the software 2.0 world, we have not yet figure out the right IDEs or labeling tools to build software. These are all real data challenges that are not taught in school.

Building Feature Pipelines Is Time Consuming

Even when we have high quality labels, building feature pipelines can be a tedious and time-consuming process. For the model described in the previous section, we were lucky to re-use some of the listing-level features from another existing project. For problems that involved images as input, the feature engineering work is a lot more complex.

For example, before our room classification model, there was no image pipeline in place on our team that could be reused. A lot of data engineering work, from data ingestion, resizing images to size 224 x 224, using base64 encoding to storing thumbnails were required before we could build image models. Had this image pipeline not been in place, it would have slowed down a lot of our modeling work significantly. This is precisely why larger companies are building frameworks to make feature engineering easier (see Uber’s Michelangelo, Netflix’s Delorean, and Airbnb’s Zipline). When planning ML projects, it is always wise to budget time for feature engineering, because training data will not be handed to you on a silver platter.

Takeaway: You need to work hard to get your training data, it is often earned rather than given. Acquiring high-quality labels can be non-trivial, and building feature pipelines can be time-consuming. To the extent you can, reuse common features or even labels to solve similar problems in the same domain space.

3. Debugging and Improving ML Models is Hard

Suppose you have gone through the steps of defining a problem, acquiring labels and building a feature pipeline, you then moved on to build your first iteration of the model only to learn that the result is not so stellar. What would you do in this case to debug and improve your model?

Debugging Machine Learning Is Hard

The scenario described above is very common and is at the heart of any machine learning project. In his post “Why is Machine Learning Hard?”, Zayd Enam pointed out that machine learning is fundamentally a hard debugging problem because there are many possible paths of exploration and unfortunately the feedback loop is generally very slow.

Debugging for machine learning happens in two cases: 1) your algorithm doesn’t work, or 2) your algorithm doesn’t work well enough.

What is unique about machine learning is that it is ‘exponentially’ harder to figure out what is wrong […]. There is often a delay in debugging cycles between implementing a fix or upgrade and seeing the result. Very rarely does an algorithm work the first time and so this ends up being where the time is spent.

Debugging machine learning is a skill, and far too often we just try the most immediate, convenient, or “obvious” thing even though it might not be the right first thing to try. Of all the resources out there, I particularly appreciate Andrew Ng’s book Machine Learning Yearning. This approachable reference is very practical and he talks about things that I wish I had known way earlier!

Some Basic Debugging Skills

While I highly recommend everyone to read Andrew’s book, for the impatient, I will highlight a few tricks that I personally found to be useful in practice:

• Error Analysis: Learn from your model’s mistake. Specifically, hand picks 100 examples that your model got wrong from the development set and tally up the reasons why it got wrong. This can help you inspire new directions and prioritize improvement plans.

Error analysis is important because it gives you a very data-informed view of why your model is not performing well. This is something that I used to avoid doing because of its tedious nature, but over time have really come to embrace it as it gives me a lot of insight about the data and my models’ behavior.

• Understand Bias-Variance: There are two major sources of error in machine learning: bias and variance. High bias often means that your model is too simple to capture the complexity of the data, and high variance indicates that you have only learned the pattern at hand. To understand bias and variance in your models, the most effective debugging tool here is to plot the learning curve.

When both your training error and development set error are way higher than the desired performance, you are suffering from a high bias problem (under-fitting). In such a case, increasing your model capacity or switching to a more complex algorithm is likely to help you to learn the patterns better.

On the other hand, when your development set error is way higher than the training error, while the training error is relatively close to the desired performance, you are suffering the high variance problem (over-fitting). In such a case, you might want to try a simpler model, or use regularization. Alternatively, if the gap between training and development error is closing with more training examples, you might consider adding more training data to your learning task.

Takeaway: debugging machine learning is hard and the feedback loop is generally slow. Instead of tackling what you think is the next obvious thing, it’s important to be more principled about debugging. Error analysis, learning curve are all good starts, and I strongly encourage you to read Andrew’s Machine Learning Yearning to improve your debugging skill.

4. Your Paths to Model Productionization Might Vary

Assuming that you now have a satisfactory model to deploy, it is time to integrate your model into a decision process or product. This is what we refer to as model productionization. But what exactly does it mean? The answer depends on your use cases. Sometimes, the predictions will live outside of products completely and will be used only for strategic or business decisions. Other times, they will be an integrated part of a product experience.

Not Everything is Low Latency & Context Sensitive

The most useful framework I learned about this topic came from Sharath Rao, who currently leads machine learning efforts for consumer products at Instacart. In his DataEngConf talk, Sharath explains that implementation of machine learning models can usually be considered from two dimensions:

• Latency: How fast do the predictions need to be served to the end users?
• Context Sensitivity: Will we know the features ahead of inference time?

In the simplest case (bottom-left), for applications where predictions are mostly used for offline decisions, the model can be productionized simply as a batch scoring job. On the other hand, for models that are an integrated part of a product experience, e.g. search ranking, input features are generally not available until a user interacts with the product, and results often need to be returned really fast. In this case (top-right), online inference or real-time scoring is needed and SLA requirements are generally higher. Knowing the profile of your ML model can directly inform your implementation strategy.

Revenue Prediction Model, Illustrated in Multiple Use Cases

Let’s use the listing LTV model that I introduced earlier as an illustrative example. Suppose we are interested in using this model to prioritize markets to go after next year. Such an application is not consumer-facing, and we are only using the predictions for offline decision making, not in an online product. For this use case, we only need to productionize the model as an offline training, offline scoring, batch job so other data scientists can easily query the predictions from a table.

However, suppose we are now interested in showcasing the predicted host payouts in a consumer-facing product in order to inform users their financial potentials. One challenge we need to consider is how to surface the predictions within a product.

In the case where contextual data is not needed, one common strategy is to store the model results as key-value pairs in a key-value store, e.g. in the form of {key: dim_market, value: revenue prediction}. For this use case, the revenue prediction can be easily looked up based on the market in which the listing is located. A more involved product might allow users to specify their location, room size, and capacity so earning potentials can be personalized. In such a use case, we will not know the features until a user enters the information, so predictions need to be computed in real-time. Depending on the use cases, your path to productionization might vary.

Takeaway: Taking models to production can mean different things depending on the context, use cases, and infrastructure at the company. Having basic familiarity with concepts such as latency and context sensitivity will greatly inform your implementation strategy.

5. Feedback Loop Can Help You or Hurt You

Models that are an integrated part of a product experience, or what we referred to as data products, often involve feedback loops. When done right, feedback loops can help us to create better experiences. However, feedback loops can also create unintended negative consequences, such as bias or inaccurate model performance measurements.

User Feedback Can Make Your Model Better

One of the most unexpected skills that I learned about real-life machine learning is the ability to spot opportunities for users to provide model feedback via product interactions. These decisions might seem only relevant to UI/UX at first, but they can actually have a profound impact on the quality of the features that the data product offers.

For example, Netflix decided last year to move away from the star-rating system to a thumbs up/down system, reportedly because its simplicity prompts more users to provide feedback, which in terms help Netflix to make their recommendations better. Similarly, Facebook, Twitter, Quora, and other social networks have long designed features such as likes, retweets, and comments which not only make the product more interactive, but also allow these companies to monetize better via personalization.

Creating feedback opportunities in product, instrumenting and capturing these feedback, and integrating it back into model development is important for both improving user experience as well as optimizing the companies’ business objectives and bottom lines.

Feedback Loops Can Also Bias Model Performance

While feedback loops can be powerful, they can also have unintended, negative consequences. One important topic is that models that are biased will amplify the bias the feedback loop introduces (see here). Other times, feedback loop can affect our ability to measure model performance accurately.

This latter phenomenon is best illustrated by Michael Manapat, who explains this bias based on his experience building fraud models at Stripe. In his example, he pointed out that when a live fraud model enforces certain policy (e.g. block a transaction if its fraud score is above certain threshold), the system never gets to observe the ground truth for those blocked transactions, regardless of whether they are fraudulent or not. This blind spot can affect our ability to measure the effectiveness of a model running live in production.

Why? When obvious fraudulent transactions are blocked, the ones that remained with ground truth that we can observe are typically false negative transactions that are harder to get right. When we re-train our models on these “harder” examples, our model performance will necessarily be worse than what it really is performing in production.

Michael’s solution to this bias is to inject randomness in production traffic to understand the counterfactuals. Specifically, for transactions that are deemed fraudulent, we will let a small percentage of transactions pass, regardless of their scores, so we can observe the ground truth. Using these additional labels, we can then re-adjust the calculation for model performance. This approach is simple but not entirely obvious. In fact, it took me a long while before spotting the same feedback loop in my model, and it is not until I encountered Michael’s talk that I found a solution.

Takeaway: Feedback loops in machine learning models are subtle. Knowing how to leverage feedback loops can help you to build a better user experience, and being aware of feedback loops can inform you to calculate the performance of your live system more accurately.

Conclusion

Throughout this post, I gave concrete examples around topics such as problem definition, feature engineering, model debugging, productionization, and dealing with feedback loops. The main underlying theme here is that building a machine learning system involves a lot more nuances than just fitting a model on a laptop. While the materials that I have covered here are only a subset of the topics that one would encounter in practice, I hope that they have been informative in helping you to move beyond “Laptop Data Science”.

Happy Machine Learning!

At Last, The Finale

From Part I of this series, we learned that a robust data foundation is an important prerequisite for pursuing analytics, be it business intelligence, experimentation, or machine learning. From Part II, we delved deeply into the specifics of Airflow and discussed techniques such as data modeling, star schema, and normalization. Furthermore, we walked through examples of data pipelines and covered ETL best practices. These are all important skills to learn to become an effective data scientist.

Typically, as companies climb up the hierarchy of data analytics and scale beyond a small team, complexity and development costs often increase. At Airbnb, we have more than 100+ contributors who authored Airflow pipelines. This makes enforcing ETL best practices, upholding data quality, and standardizing workflows increasingly challenging. Luckily, one of the antidotes to complexity is the power of abstraction. This principle, of course, is no exception when it comes to data engineering.

In data engineering, abstraction often means identifying and automating ETL patterns that are common in peoples’ workflows. In this final post, we will define the concept of a data engineering framework. We will dissect typical design patterns for building such frameworks, and finally, highlight a few specific examples we frequently use at Airbnb. By the end of this final post, I hope readers will be able to leverage the power of abstractions to build their own frameworks.

A Common Scenario

Suppose your company is now big enough to have several teams working on different parts of its product. Each team has its own OKR, product roadmap, and key performance indicators. As the embedded data scientist on the team, you are responsible for creating an analytics dashboard to track how the business is doing. You roll up your sleeves and get to work.

You start with data modeling and schema design. You identify relevant fact and dimension tables in order to calculate meaningful metrics organized by various useful dimensional cuts. You spend quite some time joining the tables to create a final denormalized table, and finally you backfill all the historical data. While this work is impactful, after working on a few dashboards, you have come to feel that the ETL workflow has become rather repetitive. In fact, many data scientists you talk to at the company are also creating dashboards using a similar workflow for their respective teams.

Not surprisingly, with product launches comes experiment deep dives — you carefully join an experiment assignment table with a fact table in order to aggregate user-level metrics. You compute test statistics from different treatment arms in order to calculate p-values and confidence intervals for your experiment. After talking to a few other data scientists, you once again realized that this is yet another workflow that is common among data scientists. In fact, it feels like a lot of what data scientists do on a day-to-day basis can be bucketed into a few distinct but common workflows.

This makes you wonder — is it possible to automate (at least partially) these workflows? The answer, is, of course, a resounding yes!

From Pipelines To Frameworks

As we have already learned from Part II, Airflow DAGs can be arbitrarily complex. Sometimes, the data computation even follows a control-flow like logic. For example, if you are interested in branching out the data flow after a specific conditional check, you can apply the BranchPythonOperator. If you want a workflow to continue only if a condition is met, you can use the ShortCircuitOpeartor. These operators, combined with the principle of configuration as code are what makes Airflow ETLs versatile and flexible. However, Airflow can do even more.

Our discussion so far has been limited to the design of a single, standalone pipeline, but we can apply the same principle to pipeline generation — a way to programmatically and dynamically generate DAGs on the fly. This is essentially what a data engineering framework does: it generates different instantiations of Airflow DAGs that automate data workflows. Here is how Maxime, the original author of Airflow, describes it:

To build workflows dynamically from code ... A very simple example would be an Airflow script that reads a YAML config file with a list of table names, and creates a little workflow for each table, that may do things like loading the table into a target database, perhaps apply rules from the config file around sampling, data retention, anonymization … Now you have this abstraction … you can create new chunks of workflows without doing much work. It turns out there are tons of use cases for this type of approach.

These tools are important because they enable data scientists to move up the data value chain much more quickly than they otherwise could.

• Imagine that when an experiment reporting framework auto-generates user-level metrics and computes statistics for an experiment, data scientists could now devote more time to analyzing changes in key metrics, interpreting user behavior, and highlighting the impact of product changes.
• Similarly, when a metrics framework automatically generates OLAP tables on the fly, data scientists can spend more time understanding trends, identifying gaps, and relaying product changes to business changes.
• Another example is a framework that abstracts away the engineering work required for productionizing an offline batch ML model. Data scientists no longer need to worry about package dependencies, setting up a virtual environment, or deployment. They can spend time on modeling instead.

The implication of these frameworks is profound because they drastically improve how data scientists work. These are precisely technologies that enable data scientists to provide value at scale.

Design Patterns For Data Engineering Frameworks

As useful as they are, DE frameworks are rarely born out of thin air. I once asked Airbnb’s first data engineer how he was able to create so many useful frameworks for everyone. His response was: “There really is no magic, when you have done certain task enough times, you started to see patterns that can be automated.” When you see your work as workflows, new possibilities arise.

When thinking about which workflow to automate, the framework designer needs to start by thinking about the end user’s experience. There are generally three layers of a well-designed data engineering framework: the input layer, the data processing layer, and the output layer.

• Input: This is where an end user specifies how their DAGs should be configured. User experience really matters here. Typically, the input could be a static configuration file (e.g. YAML or HOCON), or it could be something as elaborate as a web UI. The goal here is to capture user needs.
• Data Processing: This is the core of any data engineering framework, where ETL pipelines are instantiated dynamically. The code that achieves this is generally referred to as a DAG factory, which whimsically captures the notion that DAGs are being created one at a time, like in a factory.
• Output: The DAGs generated from the previous step create derived data, and the output is often saved in a downstream Hive table, presented in a well-designed UI / visualization layer, or simply consumed by downstream pipelines or frameworks.

This might all sound rather abstract. Therefore, in the next few sections, I will highlight specific examples that we leverage at Airbnb to make this more concrete. When reading the sections below, keep in mind which workflow each framework is trying to automate and pay attention to the input and output layers of the framework.

1. Incremental Computation Framework

It is quite common for data scientists to calculate computationally intensive metrics like a cumulative sum or the time since the first or last event. For example, we might wish to report the total number of users who ever engaged with a new product or to compute a histogram of days since users have last returned. The naive approach would be to query a fact table and take the sum, max, or min over all date partitions in order to calculate these desired metrics. However, this querying pattern is rather inefficient.

Why? This solution violates the ETL principle of load data incrementally since the required computation scans the entire fact table. Ideally, we would build a summary table to pre-compute these metrics so an end-user only needs to reference the metric in a single or latest date partition of the summary table. This pattern is so common that our data engineer built a framework called Incremental Computation Framework.

• Input: A HOCON configuration file where a user specifies which metrics or events to pre-compute, which subject key to group by, and which fact table to query from in order to build the summary table.
• Data Processing: An Airflow script that builds the summary table incrementally: namely, union the summary table from the previous date partition with today’s fact table to update the expensive metrics: cumsum_metric_today = f(cumsum_metric_yesterday, metric_today), where f can be a sum, min/max, or any other aggregation functions.
• Output: Optimized summary table where cumulative sum, days since first / last event or other expensive metrics can be queried from one and only one single date partition from the summary table.

What workflow does this framework automate? It helps users to avoid inefficient querying patterns and automates away the tedious aggregation that we otherwise would need to do, one date partition at a time.

2. Backfill Framework

As we have already discussed in Part II, backfilling is an important but time-consuming step in any data engineering work. Once an ETL pipeline is built, we need to retrospectively visit earlier data in order to reconstruct history. Some of the backfilling strategies we have discussed include dynamic partitions and baking backfill logic into SQL using Jinja templates. However, even with these techniques, backfilling can still be tedious. An engineer once told me: “I do not ever want to hear the word backfill again”, which gives you some idea how tedious backfilling can be :)

For example, if we need to backfill a few years worth of data, it would be much more efficient to break and parallelize such a task into mini-backfills. However, managing these long-running parallel processes can be rather cumbersome. In addition, we often need to perform sanity checks before inserting backfilled data into a production table. Given that backfilling is such a common but far too often unpleasant experience, we built a Backfill Framework to automate this workflow:

• Input: A simple UI where users can specify the job name, the start_date and end_date of the backfill job, how many processes we want to parallelize the backfill for, and how many days each process should backfill for.
• Data Processing: Once a user specifies how the backfill job would be run, the framework creates a Airflow pipeline that automatically parallelizes the backfill tasks, performs sanity-checks, and swaps staging tables with production tables.
• Output: A fully backfilled table ready for consumption.

What workflow does this framework automate? It automates away many of the ad-hoc backfilling scripts people have to run on their own machines. It automates quality assurance by setting up automatic comparison. Finally, it swaps the staging table with the production table after QA tests.

3. Global Metrics Framework

Up until recently, data scientists at Airbnb spent quite a lot of time when it came to building ETLs for analytics and dashboards. As we discussed earlier, a lot of work here is to identify the correct data sources, to define metrics and dimensions, and to create the final denormalized tables. While different teams might have different key performance metrics and, as a result, different fact tables, important dimensions for the business are usually quite consistent and slowly changing.

For example, data scientists who work on the host-side of the marketplace typically care about dimensional cuts such as the listing’s market, type, or capacity. Similarly, data scientists on the guest-side care about dimensions such as guest stage, origin market, or destination market. With this insight, it became clear that a lot of ETL pipelines actually involved joins of many fact tables with a much smaller set of dimension tables. This is what motivated the creation of Global Metrics Framework.

• Input: A much more involved HOCON configuration file that specifies one or more metrics in an atomic fact table, dimension sets that one wishes to include in the final table, primary keys and foreign keys to be used for joins, and a slew of other useful information to track table creations.
• Data Processing: The framework identifies the metrics and the dimensional cuts that it needs to aggregate and cut by, joins the dimension tables with the atomic fact tables to create the denormalized tables automatically.
• Output: One or more Hive tables with the same set of metrics but possibly different sets of dimensions are created. This means that one or more denormalized tables can be created on the fly, and all these data sources are further made available in Druid for visualization in Superset.

What workflow does this framework automate? It automates away the common data engineering work that is required for the creation of denormalized tables that can be later used for dashboards, analytics, and more. The product manager of this project called it the “Denormalization Machine”.

4. Experimentation Reporting Framework

Many of the data-driven technology companies have built their own internal experimentation platforms, and Airbnb is no exception. Given that I had worked on a similar team at Twitter, I can appreciate how complex ETL in experimentation pipelines can be. They often involve several modular DAGs, each consisting of thousands of tasks.

Despite its complexity, our experimentation reporting framework actually follows the same design pattern described above. The only difference here is that each layer is far more complex than the layers mentioned in all the examples earlier. However, such an investment is often worthwhile and even necessary, because it enables product teams in the company to run hundreds or thousands of experiments concurrently without needing to hire hundreds or thousands of data scientists.

• Input: Instead of a simple configuration file or a simple UI, a fully fledged UI is built here so users can specify the type of experiment to run, which target or secondary metrics to track, what are the experiment buckets and their relative sizes, etc. Anything that is relevant for launching and computing the experiment data is captured in this step.
• Data Processing: The metrics pipeline that computes, for each experiment, the subject-level metrics and their corresponding dimensions. The sheer combinations of these metrics and dimensions are what makes the computation super complex. In fact, it is often the case that experimentation pipelines are the most complex ETL job at a company.
• Output: Instead of a simple output table, there is a lot of downstream processing involved in this step. For example, statistics such as p-value, confidence interval, significance, and minimum detectable effect are calculated here. Depending on the maturity of the reporting framework, users might be able to do metrics capping or variance reduction. Each step requires a separate calculation before being served in the final UI.

What workflow does this framework automate? The hundreds and thousands of experiment deep dives that data scientists otherwise need to carry out.

Conclusion

By now, I hope you have come to appreciate the power of abstraction in DE frameworks. These frameworks are incredible multipliers to data scientists’ work and workflow. It was really mind-blowing when I learned about how much of my day-to-day work can be abstracted away. As a strong believer of the philosophy that analytics are built upon layers, I see these frameworks as the foundational pieces that need to be in place first. With more and more of these frameworks shared and discussed, I am curious what our work will look like in the next few years to come.

That’s it for the series: if you have gotten this far, I want to congratulate you for learning the basics of data engineering. More importantly, thank you for reading along. As I mentioned many times, none of these ideas are my own, but I was really fortunate to learn these concepts from some of the best data engineering talents today. Given that data engineering is a very important but often under-appreciated area, I hope I have done a small part in advocating for it! If you are intrigued by this series and want to learn more about data engineering and specifically Airflow, I would recommend starting with this list of resources.

Keep learning, and happy data engineering!

I want to thank once again my friend Jason Goodman for providing feedback for this series. All third party trademarks in this post are the property of their respective owners. Special thanks to Max, Arthur, Aaron, Michael, and Lauren for teaching me (directly or indirectly) all things Data Engineering related.