Interpreted Dance

Build, Buy, Or Wait

2026-04-28T00:00:00-06:00

It’s a weird exciting time in software engineering. LLMs advance. We get better at using them. Background agents hit critical mass in January 2026 with Stripe’s Minions, and Ramp’s background agents. Cursor launched computer use in their cloud agents and automations in February and March 2026. Anthropic has been pushing hooking Claude Code up to GitHub Actions for a while.

At work we’ve been talking a lot about background agents. They are useful. We have some capabilities deployed, but most of our LLM use is manually triggered by humans. We could invest some time (much less than a few years ago, thanks to LLMs!) building background agents. It would be fun! It’s also not what customers pay us for. We could pay for something like Cursor Cloud Agents or one of the many background-agent-like vendors. We tie ourselves to a vendor, get a bunch of cool functionality that we don’t have to build, and deal with or paper over any mismatches with our needs. Or we can just wait, and in six months more vendors will appear and models will improve and the big LLM vendors will ship a bunch of stuff that may even kill this whole category. AI-assisted Engineering is moving fast enough that this has happened several times already. I know of companies that built their own half-baked internal agentic coding tools, only to look longingly at OpenCode and Claude Code.

Examples Of Build, Buy, Or Wait In Computing

Front-End Data Fetching

This same build, buy, or wait dynamic shows up all over software engineering history. In the mid-2010s, big companies were struggling with ad-hoc data-fetching logic sprinkled everywhere in the app. Netflix made Falcor, then-Facebook was pushing/promising Relay and GraphQL, and most devs just . . . kept fetching data with REST. Several years later, Apollo GraphQL became a thing, but most React devs ended up just using something like React Query or other wrappers to make data fetching easier and a bit more declarative without going all in on a vendor or GraphQL solution. I think “wait” won this battle outside of giant megacorps since most front-end teams are not 500+ people and don’t feel the same pain as Netflix/Facebook.

Cluster Orchestration

In the late 2000s and early 2010s, cloud, containers, and orchestration was a big problem. If you have one million CPU cores and a giant pile of heterogeneous work that needs to run across them, how do you do that? “Allocate enough capacity to a team to cover the peak” was wasteful and risky. Best case is the team’s capacity estimates are right on, and you allocate enough to cover the peak, and waste a bunch of compute otherwise. If the capacity allocations are off, you waste even more compute or crumble under the peak.

If you squint, Hadoop both wins the award for “most fun open source name to say” and is kindof a solution to this problem, but was more focused on batch processing and not general computing. Mesos was an early solution from the University of California, Berkeley and was open-sourced in 2011 and joined Apache in 2013. Kubernetes (K8s for street cred) is descended from internal Google tools like Borg and Omega. It launched in 2014 at DockerCon, which is no coincidence, since it uses containers as the core abstraction of “stuff to be run on compute”. If you went all-in on Mesos, like Twitter did, you probably missed out on containers since you invested engineering effort in a different branch of the tree. There are a bunch of other solutions like Hashicorp Nomad or Docker Swarm, but Kubernetes is the default now. If you solved this problem in 2015, you might have picked Mesos or built your own thing. If you solved this problem in 2020, you picked K8s, like everybody else, and many problems were already solved for you. Twitter even ended up moving to Kubernetes, I’m sure at great expense.

The problem with going all in on K8s is now you have K8s. It’s powerful and deep and complicated and a pain to operate well. If you waited even longer, you could have ended up using lambdas or ECS or one of its equivalent across cloud providers, or even something like Cloudflare workers or whatever the new “do a bunch of work without worrying a ton about infrastructure” thing is. K8s and big infra decisions like this often have high switching costs. If you know switching costs are high before you adopt something, waiting looks more appealing.

How To Solve This Generally

I dunno, but that won’t stop me from writing about it! I’d estimate along a few dimensions:

How fast is the space advancing or evolving?
How expensive is it to build your own thing vs buying?
What’s the opportunity cost of waiting?
How related is it to what makes you money?
How expensive is it to switch if you build/buy and change your mind?

Faster advancement makes waiting look better. If the opportunity cost of waiting is large enough, building or buying might still be worth it. You get the idea. Think of this as a five-dimensional space and imagine a couple of hyperplanes splitting the space up into build, buy and wait. “But Jamison, isn’t this just a pretentious way of saying ‘it depends’?” Yes! But at least I got to google “what is the five-dimension equivalent of a plane in 3d.”

Back To Background Agents

For background agents, we chose “buy”. Switching costs are low, opportunity cost feels reasonably high, we don’t make money from them but expect a lot of value, and building is fairly expensive. We’re mostly using Cursor Automations. It feels like a good decision so far. They have some gaps but they’re improving quickly. They just added support for posting videos of computer use sessions to PRs, which makes it easier to trust their output. I think we’re better off than we would have been if we built our own thing or waited.

Build, Buy, Or Wait was originally published by Jamison Dance at Interpreted Dance on April 28, 2026.

Software Engineering With LLMs

2026-02-02T00:00:00-07:00

LLMs and coding agents like Claude Code crank out code but have poor taste. They are irrepressibly-enthusiastic junior savants. Irrepressibly enthusiastic junior savants are super useful! However, when used carelessly you’re at the mercy of the model. That’s fine for prototypes or small codebases, but in a larger codebase doing Real Software Engineering with a team of engineers, heavy agent usage without good taste amplifies chaos and degrades the team’s shared theory of the program. Agents used well can greatly increase the capacity of careful, thorough, experienced software engineers.

Frontier models like Opus 4.5 and GPT 5.2 Codex are now good enough that operator skill is the biggest bottleneck to getting great outcomes for coding. Here are some scattered things I’ve learned that have helped me get better output from LLMs on a team in an existing codebase of ~1 million LoC.

Carefully Craft Context

Provide good context to the agent. See context engineering for a broad overview of this. Commonly useful pieces of context like a description of your product and broad architectural or documentation patterns should go in your AGENTS.md or its equivalent so you don’t have to re-create it every time.

LLM performance degrades as context fills up. Crafting context is about carefully providing just the right information. Too much info can be worse than not enough. “Here is our whole codebase, good luck” (at one point a cutting-edge technique when context windows exploded) will do much worse than “here are the five files describe the interface to PDF uploading and generation, the problem we’re currently working on.” Agents have gotten better at iterating and compacting, but the wrong context can still set them off down a long and wrong path.

Use sub-agents to avoid filling up context. This keeps your main conversation’s context slim, which keeps performance better longer. Sub-agents are more effective the more open-ended and research-heavy your task is. I don’t always reach for it, but if I’m diving in to the unknown or hit a roadblock, I’ll ask Claude to spin up one or more sub-agents to go dig in to specifics.

Good context also includes a clear description of the problem you are trying to solve. Clearly articulating a problem is a classic senior software engineer skill. It’s even higher-leverage now with LLMs. If I don’t know the problem well enough to articulate it clearly, I’ll often separate out exploration and discovery in to a separate session. Then, once I understand it more clearly, I can hand off the clear description to a new session and start over with a fresh context window. Claude Code plan mode even supports this natively with the “Clear context and implement plan” option when exiting plan mode.

Write Your Prompt In A File

Since the initial context is so important, I’ll sometimes spend a few minutes writing and editing the prompt in a separate file. This is annoying in Claude Code directly, so I put it in a file and then copy-paste.

A written prompt in a file especially helps if the agent goes down a rabbit-hole to answer some question or problem, consuming lots of the context. Once it figures out the answer, I add the findings in the prompt and start over with fresh context.

Start Over

Don’t be afraid to throw away your session and start over! Since you wrote your prompt in a file (right???), this is easy. Tweak the prompt with what you learn. Ask the agent to record what it learned or did in a markdown file that you can feed to a new session. Long sessions can pin you in local maximums as the LLM goes down a rabbit-hole. LLMs will gleefully keep solving problems created by their earlier solutions forever. This can be a trap! It takes human judgement to recognize that the endless proliferation of problems is a signal that you’ve gone terribly astray. That signal is weaker when it’s the LLM writing the code, so pay more attention to it. Stepping back and starting over is useful in general software engineering, and it’s cheaper and easier to do this with LLMs than people.

Give LLMs Good Feedback Loops

Just like humans, LLMs can iterate faster with good feedback loops. Type checking, test suites and linting can tell an LLM where it’s going wrong. Encoding correctness in systems that give fast feedback is higher-leverage than ever. Perhaps this will encourage more type systems, formal methods, etc? Interesting times.

Great LLM-driven manual QA feels like an unsolved problem here. LLMs create good test plans for features with some nudging. I HATE HATE HATE manual testing and am generally lazy and feckless about exhaustively checking things. I want to get the LLM to execute the test plan for me. If you’ve got this nailed, tell me how! Hopefully I’ll have something to report here in the next few weeks.

Tell LLMs The Theory Of Your Program

Programming As Theory Building is my favorite piece of writing on building software in teams. Go read it, but my TL;DR is that good programs don’t just accomplish a task. They have a coherent theory, and the team knows the theory. The program is deliberately structured. The team knows why the program is structured that way. There are answers for how you accomplish common things in the codebase or what abstractions to use, and why those abstractions exist.

For LLMs to do well on existing codebases, you need to explicitly write down the theory of the program in prompts, documentation, and tooling. I’ve seen huge performance improvements in agents after documenting implicit theories or architecture. The agent doesn’t have to explore the codebase to learn them anew and doesn’t slam in head-scratchingly weird workarounds for carefully-enforced abstractions or reinvent solutions to already-solved problems.

Documenting the theory of your program helps keep context clean and your codebase more consistent. These docs are also useful for humans. Humans can ask questions and retain context, but good docs, ESPECIALLY that focus on theory-building, help guide questions and direct towards an understanding of the theory.

Creating A Theory Of Your Program

This advice isn’t much help if you don’t have a theory of your program. Creating or reifying the theory of an existing program is a whole separate topic, but good news - LLMs can help you with that as well! They’re much better at uncovering theory than coming up with one out of a mess of spaghetti. Human judgement is still the best tool for broad architectural changes. Good taste, which LLMs lack, is essential for designing a theory of your program. LLM-enhanced capacity also makes the large refactors that are necessary for imposing a coherent theory on an ad-hoc codebase way more palatable. Exciting times!

Use LLMs To Improve Future LLM Performance

Barring improvements in underlying models, the only way agents get better is if their inputs and environmental feedback get better. The newest junior developer onboarding to your codebase gains and retains context over time. Agents don’t learn ANY context about your codebase or business or patterns or preferences. YOU learn, or your codebase changes to become easier for AI to manipulate correctly.

You can use agents to help with this! A simple, dumb solution is to ask the agent to create durable artifacts recording things it learned from this session. “Add this pattern to our AGENTS.md file”, or “write out a markdown file explaining this architecture” is a common way I end a session. These AI-generated artifacts usually have the tell-tale stink of AI slop and thus benefit from human curation, but they can also be better than nothing.

Tell the agents to write tests and create lint rules. Complex eslint plugins that touch on semantics, not just style, are beyond my feeble human capacity. I will make my irrepressible junior savant robot friend write them for me. Agents should be self-improving because they should improve the environment they run in.

Agent Experience Is Developer Experience

Investing in developer experience (DX so I don’t have to type this out every time) makes agents more effective, and making agents more effective can have huge returns. Agent experience is developer experience. You should probably invest more in DX than you were before using agents.

Larger engineering orgs (100+ engineers) typically dedicate a single-digit percentage of headcount to an engineering productivity team. They build frameworks and component libraries, create developer tools, operate CI/CD, make releases faster, or do other stuff that makes all the other engineers work more effectively. Businesses don’t like that many engineers working on non-customer-facing things, but at large enough scale, shared problems with developer efficiency are visible enough to justify the investment. Smaller orgs generally rely on individual heroics or skunkworks projects which can lead to poor DX.

LLMs make investments in DX much higher leverage. If a small team of five engineers is managing hundreds of parallel agents, a small gain in efficiency by steering them better through great abstractions, excellent docs, easy sandbox environments, faster deploys, comprehensive testing, a consistent design system, great error messages, etc etc, is spread across those hundreds of agents, not just the individual engineers.

LLMs also make investment in DX cheaper. I’ve done a TON of ops and observability work that would have been just annoying enough before LLMs that I would have never gotten to it.

Take advantage of this virtuous cycle. LLMs make it easier to build systems around your software to make it easier to build and run. They also do better when these systems exist.

You Don’t Have To Be On The Bleeding Edge

There is a cacophony of LLM hype on Twitter, Reddit, YouTube, and elsewhere. My friend Chad called this LLM fatigue, and it feels pretty accurate if you lived through the JavaScript Fatigue days. There are infinity frameworks and tools and courses and abstractions. Prompts become commands become hooks become skills become plugins become some new thing. “You’re falling behind if you’re not running ~~clawdbot~~ ~~moltbot~~ OpenClaw.” “Oh, you’re not using ~~GasTown~~ ~~Conductor~~ a rack of raspberry Pis running Kimi K2.5? How gauche.”

It gets exhausting.

This is the most excited I’ve ever been about software engineering. There’s so much to learn. Under the hype, there is real benefit. I can do more and go faster than before.

The noise can make it feel like you’re hopelessly behind. You’re not behind. Most developers are barely using LLMs. Most of the noise is people trying to sell you something. Great ideas will stick around. I’ve found peace in being a few months behind and learning at my own pace. It’s still enough, and I’m still getting better.

Acknowledgements

Thanks to Kiran for teaching me how rad LLMs are for software development and filtering out some of the noise and Chad for bouncing ideas around.

Software Engineering With LLMs was originally published by Jamison Dance at Interpreted Dance on February 02, 2026.

LLMs Are Microtransactions For Software Engineering

2025-08-10T00:00:00-06:00

Sometimes I finish a session in Claude Code and run the /cost command. “It cost $2.37 to fix that bug,” etc etc. My own time is involved, of course, but seeing the cost in tokens and vendor spend is helping me think about building and tending software in a way I haven’t before. Costs in software engineering are usually fuzzy. How much does a feature or product cost? Do you count salaried employee time? When do you start counting? Do you count hammock time or shower time or the opportunity cost of not building other stuff? Cloud spend can sometimes be part of that cost, but is often a very small part for small startups, and if it’s a feature on an existing product, it’s hard to attribute fractions of cloud spend to that specific feature. This fuzziness often leads me to throw my hands up at the idea of measuring cost at all and go for vibes - this feature “feels” expensive, this refactoring doesn’t. That’s not even getting in to value, which is at least as hard to measure as cost. “Fractional not-churned custom” is not a unit we can easily measure in without a magical machine to fork the timelines in to control and experiment.

All of that is why I think it’s nice to glance at an LLM bill and say “that bug cost $2.37 in LLM spend to fix”. Sure, it doesn’t account for the hard-to-quantify costs of gathering all the context to spend the $2.37 on an LLM in a way that would actually fix the bug. But I still like it. It kinda feels like buying gems in one of those horrible match-three mobile games. Spending money to buy engineering outcomes like quality, speed, security, etc, has been the promise of basically every engineering-focused vendor. I can now buy level-skips or whatever to deal with some tedious work to improve dev environments, clean up some janky code, fix a bug, etc. I am happily doing it.

LLMs Are Microtransactions For Software Engineering was originally published by Jamison Dance at Interpreted Dance on August 10, 2025.

Practicing Hard Conversations Makes Them Eaiser

2025-06-09T00:00:00-06:00

Practicing hard conversations makes them easier. This is true even if the live conversation doesn’t look much like your practice.

Hard Conversations As Graphs

Hard conversations feel like a graph traversal/search problem to me. Nodes are topics, facts, decisions, emotional beats. Edges are transitions, questions, clarifications. You can have cycles, edge weights (some topics are easier to move to than others), etc. Which path will we take? What node will we end up on?

Hard conversations go in unpredictable directions, and some of the paths are bad. “What if they yell at me?” “What if they destroy my arguments with facts and logic?” “What if they lose respect for me?” Bad news: the graph is infinite at human scale. It’s impossible to prep for all paths.

The benefit of practice isn’t from exhaustively searching the graph and visiting every possibility. It comes from the increased calm and sense of preparation you feel during the difficult conversation.

Another reason hard conversations are hard is they can be emotionally charged. Emotions are useful signals about what is going on in our brains but crank up the conflict resolution difficulty level. Now you’re doing multiplayer graph traversal during an earthquake.

Practice By Sketching And Talking

Try sketching a graph! Where will you start? Where might you go next? Practice some edges. What phrases are your transitions? Where do you want to end up? Spending a few minutes drawing and saying phrases out loud helps me. Maybe don’t tell them you did this - saying “I prepped for this discussion by reading up on Dijkstra’s algorithm” does not lend itself to emotional connection. You know what does? Practice.

Practicing Hard Conversations Makes Them Eaiser was originally published by Jamison Dance at Interpreted Dance on June 09, 2025.

A Short List of Great Software Writing That Circulates In My Brain

2025-03-02T00:00:00-07:00

A handful of pieces of great software writing float around in my brain. They rotate over time, loosely influenced by what I’m reading and what I’m working on. A current list is:

They are generally more strategic than tactical, which helps them age well.

I wonder what the list will be in a year?

A Short List of Great Software Writing That Circulates In My Brain was originally published by Jamison Dance at Interpreted Dance on March 02, 2025.

Capatilities And Possibilities On Engineering Teams

2023-05-17T00:00:00-06:00

Lately I’ve been thinking about possibilities and capabilities on engineering teams. I was chatting with an engineer about tracing, spans, and getting detailed timing information from a complex browser application. It was fun to think about! We had a bunch of different pieces that helped with different subsets of the problem. We could squint our eyes and see a way to combine them in to something that would kindof sortof work.

I was struck by how hard it felt to do something seemingly routine like “figure out how long stuff takes in the browser.” It shouldn’t be this hard! Why did it require novel work and brainpower? Hadn’t we worked on measuring performance several times? (We had.) Measuring front end performance was possible, but I wanted it to be easy. I wanted it to be a capability.

What Is A Capability?

When I say capability I mean a skill or outcome that is cheap and easy for an engineering team. It might be useful to contrast capabilities and possibilities. Something is possible if someone on the engineering team could pull it off, sometimes with extraordinary effort or at high cost. Capabilities are cheap, boring, routine, repeatable, and predictable.

Capabilities are context-dependent. Different teams will have different capabilities. If your crack team of Rails developers decides to pivot to ~~mobile~~ ~~blockchain~~ ~~video~~ generative AI, it’s probably possible for them to stumble their way through deploying and operating large language models, and thus stumble in to enormous pots of VC funding. However, it won’t be easy. “Figuring out how to do new things” is an important part of the job of software engineering. It’s also more expensive to figure out how to do new things than to do things you already know how to do well. If instead the team has been working in AI since it was called machine learning and has heard of things like “MLOps” and knows what GPT stands for, deploying and operating large language models might be a capability. It also might not be! Rails application development is probably not a capability of this team.

After the team has developed a capability, doing the thing doesn’t require specific skill or expertise. Developing capabilities requires skill and expertise and deliberate effort.

Capabilities And Leverage

It’s possible for a team to have expertise at something but not turn that expertise in to a capability. This can happen on teams with long tenure. Engineers just know the domain so well that the right thing seems obvious and they don’t build tooling or processes around it. How many times have you seen deployment processes that involve bloodletting and careful consideration of the phases of the moon that the team is just used to?

Identifying and developing capabilities can be a high-leverage activity. Senior engineers and managers should think about capabilities, identify gaps, and help teams develop capabilities. Picking capabilities to invest in can often be a judgement call, but if you get it right, you can make the right things easy.

Capatilities And Possibilities On Engineering Teams was originally published by Jamison Dance at Interpreted Dance on May 17, 2023.

How To Annoy Everyone You Admire With Go

2020-08-10T00:00:00-06:00

My Twitter feed is a nightmarish cacophony - cute art, police brutality, security vulnerabilities with catchy nicknames, dire economic news, fun jokes, squabbles over programming languages, etc. You know the dril.

Time for a change! I follow people I like and admire and am friends with, but the aggregate feed of all these people I like and admire and am friends with somehow makes me sad. Perhaps I am just fragile?

Anyways, the best solution to this social/personal problem is clearly to write a computer program. My feed is too overwhelming, but I don’t want to lose touch with the people I follow (“friends” in the Twitter API parlance). I will move my friends to a list and unfollow basically everybody.

The Twitter UI for lists is a pain and involves lots of clicking.

Let’s use the Twitter API to load all my friends and add them to this list!

The Code

package main

import (
	"fmt"
	"strings"

	"github.com/dghubble/go-twitter/twitter"
	"github.com/dghubble/oauth1"
)

const (
	// lol no
	apiKey      = "⊙▽⊙"
	apiSecret   = "¯\_(ツ)_/¯"
	accessToken = "(>._.)>"
	secret      = "^_^"
	listID      = 1234
)

func main() {
	config := oauth1.NewConfig(apiKey, apiSecret)
	token := oauth1.NewToken(accessToken, secret)
	httpClient := config.Client(oauth1.NoContext, token)

	client := twitter.NewClient(httpClient)

	// Cleverly avoid the if err != nil boilerplate by not checking for errors
	friendIds, _, _ := client.Friends.IDs(nil)

	// You can only add 100 users to a list at a time, so group in to slices of 100 ids.
	// This took an embarassing amount of time to get right.
	ids := make([][]int64, 0, 0)
	for i := 0; i <= len(friendIds.IDs)/100; i++ {
		end := (i + 1) * 100
		if len(friendIds.IDs) < end {
			end = len(friendIds.IDs)
		}
		ids = append(ids, friendIds.IDs[i*99:end])
	}

	for _, hundredIds := range ids {
		str := strings.Trim(strings.Join(strings.Fields(fmt.Sprint(hundredIds)), ","), "[]")
		client.Lists.MembersCreateAll(&twitter.ListsMembersCreateAllParams{ListID: listID, UserID: str})
	}

	// No need to check for errors if your heart is pure.
	res, _, _ := client.Lists.Show(&twitter.ListsShowParams{ListID: listID})
	fmt.Printf("%s: %d", res.Name, res.MemberCount)
}

About The Code

This code uses several Go patterns, including “just don’t check errors” and “hope everything works”. I didn’t even need generics. Go really does help you write fast and correct code! I also use other best practices like “search GitHub for libraries and install the first one that shows up”. My in-depth analysis led me to the github.com/dghubble/go-twitter, which worked fine.

If the list you’re trying to add people to is public, this code will also send them a notification each time you add them to the list! This is a great way to remind people you are friends with and admire that you exist, just like high school. This is especially effective when I run this code dozens of times using all dozen of my brain cells to debug the for loop.

In case you’re wondering the code font is Times New Roman and the editor is Kubernetes.

Why I Chose Golang

¯\_(ツ)_/¯

A Powerful Conclusion Goes Here

My original goal was to bring some silence to my Twitter feed by replacing following with lists. I still didn’t un-follow anyone because it made me feel bad, so now I just double-follow everyone. In the end, writing a computer program in Go did not solve all my personal/social problems. Perhaps Rust?

How To Annoy Everyone You Admire With Go was originally published by Jamison Dance at Interpreted Dance on August 10, 2020.

Motorcycles and Engineering Teams and Speed

2020-02-24T00:00:00-07:00

My motorcycle goes about 140 mph, and gets there pretty quickly. I do not need to go 140. It would be wildly dangerous, it’s not great for the bike, and my ego is not so fragile that it is harmed by riding safely. However, I do need to occasionally crank the throttle to speed by a distracted commuter on a cell phone or someone weaving in a lane or a pickup truck’s blind spot. A fast bike doesn’t mean I go fast all the time. I sprint around an obstacle and drop back down to a normal speed.

Software teams shouldn’t work at top speed as their normal operating mode. Just like you wouldn’t normally ride a motorcycle with the throttle wide open, engineering teams should operate at a comfortable pace. They occasionally need to go fast, like if a real (ACTUALLY REAL, often software deadlines are timelines, not deadlines) deadline approaches or to fix a critical bug. If your over-allocate the team’s time, they can’t react well to changes, engineer autonomy is reduced, and people burn out. Just like my bike goes fast but I don’t always ride it fast, teams need to be able to go fast but should normally have some unallocated time.

If teams have a lot of defined and prioritized work, it can be hard to take on new opportunities. That would “push out” the existing work, and since it’s already assigned and prioritized and defined, it must be important, right? I’ve observed this phenomenon in myself and on teams: if the team is swamped, new ideas and opportunities look harmful because they’ll increase the workload even more. Without some slack time the team doesn’t have time to stop, breath, evaluate, and decide if this new thing should bump out the exiting work. The team transforms in to an assembly line, heads down, pulling stuff off a queue, frantic to keep up with the conveyor belt. That’s not a good feeling.

Software developers generally want to do good work and they know their own pain pretty well. If you give a motivated and talented developer slack time, the result is generally great. They’ll fix a nagging broken thing that has been causing more pain than you realized, or they’ll explore an opportunity you hadn’t thought of, or they’ll jump in to help someone who’s struggling, or they will rest and recover for when you actually need to sprint again.

Motorcycles and Engineering Teams and Speed was originally published by Jamison Dance at Interpreted Dance on February 24, 2020.

Pairing As A Manager

2020-01-04T00:00:00-07:00

I manage a team developers. Every two weeks, I pair-program with each person on the team for an hour. If you’re a software developer managing developers, the ability to pair-program is a unique skill you have that you should take advantage of.

Pair programming helps me gather information, enforces communication and collaboration norms, and (sometimes) helps us solve tricky technical problems.

Gathering Information

I typically pair on whatever the developer is working on. Managing means writing code is squeezed out (as it should be) in favor of other work. Instead of producing directly, I support the team in producing. One side-effect of being out of the code is losing touch with what it’s like to work in the codebase, infrastructure, and environment. I don’t feel the pain of poor infrastructure or design, or the payoff of great tooling or tech choices.

Part of my job is to provide an environment for developers to do their best work. I pair to see if I’m providing that environment technically. Is it hard to make changes? Are our builds flaky? Do we take too long to spin up new services? The team can tell me, but I learn different things when experiencing it directly while pairing. I use this information to prioritize technical investment. We might push off some feature work to speed up our deploys or refactor a gnarly module.

Communication and Collaboration Norms

The team is fully distributed and rarely meets in person. Working remotely requires individual communication skills and a shared culture of collaboration. We don’t get ad-hoc collaboration for free like an in-person team. We also miss out on a lot of signal around mood, body language, facial expressions, etc. That’s a weakness of remote teams. To fortify our team against this, we have to deliberately craft our team culture and values to encourage collaboration, asking questions, reaching out, responding, and working together well. We don’t pair full time (I would quickly dissolve in to slime and ooze away to another job in such an environment). However, we pair regularly and with little ceremony. This helps us share information, solve problems, get to know each other better, and make up for missing those out-of-band communication channels like lunch, passing in the office, etc.

By pairing as a manager, I signal to the team that collaboration is important enough for me to choose to do it over other things. This rolls down to other team members. We pair more after I instituted it, and individuals pair more with each other without any direct encouragement from me. Pairing as a manager helps our team communicate more and makes us a stronger remote team.

Solving Technical Problems

The least important reason to pair as a manager is to crank stuff out. Sometimes I can help unblock or share context and really move a task along. Sometimes I have insight or help to offer as another set of eyeballs connected to a brain that understands code. Sometimes I just follow along and learn. I think I’m slightly less effective as a pair than I was as an individual contributor because I have less context in the code. However, I still think it’s a net positive for the developers I’m pairing with. I should probably ask them, though would they tell me if I was wasting their time? 🤔

Potential Downsides

Pairing as a manager has downsides. The power dynamics can get strange. Developers are pairing with their boss. They could feel pressure to give a good impression, or feel like expressing doubt or missteps will reflect poorly on them. I try my best to head off these concerns, but I can’t completely eliminate power dynamics.

Pair As A Manager

Despite downsides, the benefits of gathering direct information, modeling norms of communication and collaboration, and working together to solve hard problems outweigh the costs.

Have you paired with your team as a manager? Do you have different experiences? Are you trying this out for the first time? Let me know what you think on Twitter at @jamison_dance.

Pairing As A Manager was originally published by Jamison Dance at Interpreted Dance on January 04, 2020.

Keep Interfaces Small

2019-12-13T00:00:00-07:00

Keep interfaces small. Other things equal, a larger interface is always worse than a smaller interface. What do I mean when I say interface? An interface is the exposed functionality of a module.

An interface is the exposed functionality of a module.

For an HTTP service, the routes are the interface. For a library, exported or exposed functions, methods, and data structures are the interface. Each thing (variable, type, function, route, concept) exposed to consumers is part of that module’s interface. An interface is an implicit contract to consumers - both “this is stuff you can use that won’t change”, and “this is stuff you might need to use.”

Larger interfaces are worse for the developer of the module because each exposed thing is a fixed point that is hard or impossible to change. Again, interfaces are contracts to users. Breaking contracts comes with large costs. Migration, education, breaking consumers that refuse to migrate, long upgrade cycles, etc. In some environments, it is basically impossible to change an interface once it has any consumers! Large interfaces have large costs on the creator and consumer side.

Like Sandi Metz says, good design is easy to change. Creating an interface costs the module developer flexibility, but usually it is worth it since it provides stability to users. This is good (no one would use a module that had no guarantees of stability), but also comes at a cost. You’re locking yourself down and limiting your flexibility to respond to future change. Smaller interfaces lock you down less.

If you’re making libraries or modules consumed by programs you write, a larger public API increases the number of connections between different parts of your program. That is more parts for you to keep track of and update when you want to change things. The best interface is the smallest possible one that can do the job. Whenever you can, inline abstractions, hide implementation details, and don’t add things to the module’s interface. Your default approach should be to keep interfaces as small as possible.

Larger interfaces are worse for users because there are more things to learn and keep track of. It increases the cognitive load of using the module. Larger interfaces also encourage more dependencies between a module and its consumers.

So do whatever you can to keep interfaces small. Audit a module’s exported concepts and structures. Do they pay for themselves? Does the added functionality pay for the reduced flexibility? Is it worth the added burden of knowledge on the user?

Keep Interfaces Small was originally published by Jamison Dance at Interpreted Dance on December 13, 2019.

Reclaiming Email

2019-10-11T00:00:00-06:00

I recently took six weeks of paternity leave. Paternity leave is rad! The kiddo and mom are doing great.

Coming back to my work inbox was not great. While on leave I received ~8,000 emails that weren’t filtered by my existing rules. 43 emails were actionable or worth reading. That’s 0.5%. Those 43 emails were pretty important, so I can’t just ignore all email. For reasons I can’t just unsubscribe from all those emails. That means I’m trapped in a nightmare of clicking rectangles to make numbers go down. Email is effectively useless if 99.5% of messages are junk, but I still have to sift through them all because some of them are very important.

Email should be the system of record for decisions and announcements. If instead email is mostly noise, it’s tougher to share important things. You have to turn to the frantic buzz of chat or maybe just yell really loudly in the office. What can I do about this? How can I increase the signal in email and make it more helpful?

After talking to my boss and poking around the Internet for advice, I came up with a solution. Previously I was trying a block list. Each time I got an email that seemed less than useful, I’d create a rule to filter out emails like that in the future. Instead I moved to an allow list to make sure I focused on emails I knew were important and just skimmed everything else. I set up three folders: one for my reports, one my boss and their bosses, and one for everything I’m CC’d on.

I check the folders for my bosses and reports a few times a day. I want to be responsive to those folks. I check the CC folder once per day. This is straight from Scott Hansleman. I read the rest of my inbox every once in a while. This has drastically cut down on the amount of time I spend clicking the delete button, which has also taken away one of the great sources of joy in my life, which is reading things and then knowing I don’t have to do anything because of them. I think I’ll survive.

There is a chance I’ll miss something very important. That chance is already there now, and also I spend much less time flailing in a loop between email and chat, just making numbers get smaller forever without actually getting anything done. This helps me sift the important from the less important and stem the endless flow of distraction. What have you found useful for reclaiming email?

Reclaiming Email was originally published by Jamison Dance at Interpreted Dance on October 11, 2019.

How To Screw Something Up

2019-09-22T00:00:00-06:00

Screwing up hurts. A while ago I made a hire I was excited about. It turned out to be a really terrible fit for the team and company culture. It blew up in the team’s face leading to pain, failed projects, and an acrimonious departure. What should you do when something fails, especially something you’re responsible for?

Maybe you led a meeting that devolved in to insults, or a project collapsed in a smoldering pile of open JIRA tickets and frustrated users, or your posh and expensive rebranding ended up looking suspiciously like an armpit someone copied from clip art, or you got caught borrowing money from your company to buy assets that you then lease back to your own company. After discarding the immediate bad ideas that pop in to your head (mash a few keys in your email client and blame whatever name autocompletes, pivot to the blockchain, file for IPO), Bob Sutton on the Dear HBR podcast says you should say three things:

I screwed up.
Here is what we learned.
Here is how we’ll fix it.

The Steps

Explicitly say “I screwed up.” In the short term, staying silent about failure can feel more comfortable than drawing attention to it by admitting it. When something fails, people generally already know. Acknowledging it lets people know you’re not okay with the way it ended up. Being open also avoids the shame trap of worrying that everyone will someday find out you failed, which can really amp up existing imposter syndrome. If you’re a leader or manager, taking the blame for failures that happen on your watch is part of the job.
Say “Here is what I learned.” You should have learned something when things go wrong. If you learned “someone else did a bad job”, you have not actually learned anything. What could you have done to avoid this knowing what you know now? How could you have coached someone, or prepared better, or eliminated risk, or not gotten caught by those pesky regulators (👺)?
Say “Here is how we will fix it.” Your future plan should be more than “next time we’re in the exact same situation, we won’t screw it up.” What can you change now? How have you grown from reflecting on screwing up? Having a clear plan of how to fix a failure transforms it from shame to learning and improvement. Postmortems in technical incidents are an example of this. Instead of just wallowing in the muck and blame of an outage, good postmortems result in some concrete improvement to make to infrastructure, training, code, or processes to make future failures less likely or less costly.

An Example

In my earlier example of a wrong hire and subsequent bad performance, I would talk to the affected people (probably my boss and team in this situation) and say something like this:

“I made a bad hire. It hurt the team’s morale, safety, and productivity. I’ve learned I need to give earlier feedback more clearly when there are performance and behavior problems, and to look out for some red flags in the interview process. For the first three months after a new hire joins I’ll do a performance review. I’ll add the following interview questions to my interviews: etc etc etc”

The Conclusion!

Screwing up hurts, but failure is a great teacher. Learning and improving from failure and demonstrating that to others takes some of the sting out of it. I suspect it lowers the long-term cost of failure by increasing the benefits which can let you try more failure-prone things. That seems great!

How To Screw Something Up was originally published by Jamison Dance at Interpreted Dance on September 22, 2019.

Fact-checking Technical Claims

2019-09-13T00:00:00-06:00

A while ago I read a post that included the following quote:

Each hour spent on code review saves 33 hours of maintenance.

This struck me as suspicious. I mistrusted the exactness, certainty, and magnitude. Software engineering is a fuzzy field and absolutes and exact numbers are not often found in the research. I decided to dig in to understand where this number came from. Thankfully, the author cited a source: Russell, Glen W. “Experience with Inspection in Ultralarge-Scale Developments,” IEEE Software, Vol. 8, №1 (January 1991), pp. 25–31. Let’s check it out!

The Source

The article is behind a paywall, but oh my gosh, a stranger just bumped in to me and in the confusion this link was knocked out of their stylish briefcase and fell on the ground in front of me: https://cyber.sci-hub.se/MTAuMTEwOS81Mi42MjkyOQ==/russell1991.pdf. Weird, the stranger disappeared as suddenly as they arrived! Oh well, let’s dig in to the article and see what it says.

The first thing that jumped out is that the source is not a peer reviewed scientific study. It’s the equivalent of a blog post from 1991. Blog posts are great (like this one, hopefully?), but someone writing an article in a journal isn’t a stamp of absolute truth.

Code Inspection

I also noticed the source is about code inspection which is pretty different from the modern form of code review. Page 25 of the article describes code inspection. It’s an in-person group exercise where a team of at least three people (moderator, one or more inspectors, and the code author) go through a section of code line by line, paraphrasing what every line should do. The article recommends inspectors attend overview meetings, read design docs, and generally put in some time to prepare for the code inspection. The article suggests a maximum pace of 150 lines of code inspected per hour. This is a very time-consuming process for several people. Code inspection is much more exhaustive than the modern idea of code reviews on pull requests, where the expectations are usually unclear but generally call for a much lower amount of time put in by the reviewer. The difference between modern code review and code inspection as described in the article make me a more suspicious of the claim that an hour of code review saves 33 hours of maintenance. Let’s keep reading.

Some Details on 33 Hours

The article is about code inspections at Bell-Northern Research, a company that made telecom switches. I imagine these are pretty specialized and expensive pieces of hardware.

The author’s company used code inspection on a relatively large project and tracked the number of defects found.

[I]nspections […] yielded between 0.8 and 1 defect per [person]-hour.

The author then compares this cost to the cost of finding and fixing defects in the field.

Statistics collected from large BNR software projects show that each defect in software released to customers and subsequently reported as a problem requires an average of 4.5 man-days to repair. Each hour spent on inspection thus avoids an average of 33 hours of subsequent maintenance effort, assuming a 7.5-hour workday.

Here we come to the source of the 33 hours number. The author is comparing the average number of person-days to fix defects in these deployed switches, doing some QUICK MATHS to convert it to hours, and presenting that number.

There are a few issues with taking that number and turning in to a statement like “one hour of code review saves 33 hours of maintenance.”

That comparison assumes each defect found would take roughly the average time to fix. Maybe code inspection only finds defects that would take less time to fix, or maybe it only finds the really gnarly bugs that would take more time to fix. That 4.5 person-days number might be too low, or too high.
It assumes code inspection would have found the defects reported by customers. That’s begging the question. Maybe customer-reported defects are a totally different class than code inspection defects. The author even states in the article that “Inspection and testing are implement complementary because each technique uncovers different types of errors.” Developers are pretty good at understanding normal behavior under expected conditions, but developer expectations pretty often don’t match what actually happens in the real world. I routinely encounter bugs where the system breaks in situations I’d never imagine it to be in.
Most of us aren’t working on telecom switches in the 1980s or doing formal code inspection as described in the article. The maintenance time might include the time to drive out to the hardware and apply a patch for all we know! I think it is much easier to deploy a fix to a SaaS product or web-based software than a switch from the 1980s, and I suspect most code review processes are less efficient at finding bugs than code inspection. The one hours - 33 hours applies in very specific situations and context, and applying it outside that context is extrapolating.

The Source Is Good!

I’m glad I read the underlying source. It’s got some interesting ideas! Code inspection is a neat idea, and reading the article raised some questions for me. Given how effective inspection seems, should we be spending more time on code reviews than we do? How much time? How do we set clear expectations about how much time and effort we should be taking in code reviews? Should we be tracking defects found in code reviews? Later the author compares how efficient code inspection is at finding defects compared to “system testing”. The author includes this rad clip-art diagram describing system testing.

From the diagram, “systems testing” looks like executing test cases (maybe manual, maybe automated?) on a real system running in a production-like environment. The author reports code inspection finds twice as many defects as system testing. Maybe we should be focusing more time on code reviews compared to testing? These are all interesting things to think about.

Conclusion

I think it’s important to question suspicious quotes. Find the sources for claims and read them, or doubt those claims if there are no sources provided. Software development is fuzzy and mostly based in folklore. By checking sources, we can push for a bit more data so we’re at least telling historical fiction instead of developer fables.

If you like empirical research in to software engineering, you might enjoy @hillelogram (https://hillelwayne.com/talks/what-we-know-we-dont-know/ is great, his twitter feed is great, his blog is great) and @danluu (https://danluu.com/deconstruct-files/ and https://danluu.com/empirical-pl/ are great, but also his twitter feed).

Fact-checking Technical Claims was originally published by Jamison Dance at Interpreted Dance on September 13, 2019.

Engineering Management Books

2019-09-10T00:00:00-06:00

I read a fair amount about engineering management or things tangentially related to it. I really, really want to get better, and reading is one way I try to do that. Here are a few things I’ve read along with some thoughts about each one. I linked to the books I liked.

Books I Loved

Radical Candor

I’m a people-pleaser. I want to be nice and I want everyone to like me. It turns out “being too nice” is a common failure mode for managers. Wanting to be nice can lead to avoiding conflict, not delivering difficult feedback, low performance, and lack of growth. Radical Candor talks about becoming direct enough to give hard feedback that helps people know where they stand and grow and improve while still caring about people.

The core metaphor is a graph with “how much you care” and “how directly you challenge” as the two axes. The idea is to both care about people and challenge them when they need to be challenged. You can fail by not caring about people and/or not challenging them.

I return to this book quite often. It’s rad.

High Output Management

This book was super helpful for me as a glimpse in to the mind of manager singularly focused on output. My natural tendency is to focus on people and feelings. Andy Grove is the author and long-time CEO of Intel. As the title indicates, he focuses on output. A phrase he repeats in this book is “the output of a manager is the output of their team and the teams around them.” This focus on actual measurable output and the things you do to increase it is a helpful perspective for me. Focusing on output leads to dealing with performance problems, setting clear expectations, giving good feedback, and a bunch of direct conversations that do not come naturally to me. I don’t think I’m wired like Andy Grove (few are), but I really enjoyed seeing such a different leadership style focused on getting things done first, and seeing him arrive at focusing on people in service of that goal.

The Manager’s Path

Camille Fournier’s book is the best overview of a technical management career path I’ve read. It walks through a typical career ladder from tech lead to manager to manager of managers to director to executive explaining the job at each level. Ladders and responsibilities can vary pretty widely between companies, but this book gives a solid cross-industry baseline which I hadn’t seen before reading it. When I have questions about career path or growth I return to this book.

The Making of a Manager

A guided tour through Julie Zhou’s career at Facebook from designer to director. She is a great writer. I’ve enjoyed her newsletter where she answers management, design and soft-skill questions, and this book is as great as I hoped it would be. She outlines her career path and talks about challenges and things she learned at each level grounded in very practical real-world experience. I took several pages of notes from this book. It feels immediately applicable. Highly recommended.

Performance Appraisal and Human Development

This book is basically about how to evaluate performance, deliver clear feedback on performance, and coach to improve performance. My dad gave me this book when I was struggling with giving clear feedback to my team. It was fun to read through and see his marginalia from when he was my age.

My dad's highlighting style involves lots of scribbles.

I suspect giving clear feedback on performance will be an eternal struggle, but this book helped me get better. It’s from the 1977 but I was surprised by how relevant it still felt. It was refreshingly free from hype or spin or sales pitches growth-hacking marketing speaker that inhabits modern business writing.

Performance feedback is about behavior, not psychology. It’s very hard to change essential characteristics of a person. If your feedback is based on who they are (“you seem to lack confidence”), instead behavior is that is contributing to poor performance (“when your ideas are challenged in meetings, you don’t defend them which makes it seem like you don’t support them strongly”), it’s much harder for the recipient to act on. Focus on behavior, since it gives clear direction on what to improve - you change the behavior.

The Coward’s Guide To Conflict

There is a theme here. Another book on dealing with conflict, but this one felt much more applicable and practical than Crucial Conversations. I’m glad I didn’t judge it by the cover, because it looks pretty cheesy.

From this book I learned that conflict is unavoidable, so a strategy of avoiding conflict doesn’t work out long term. You can strongly dislike it, be intimidated by it, get nervous around it, and still do a good job productively working through conflict. Avoiding conflict leads to not addressing important issues on a team or project, which makes teams less healthy and makes projects fail.

Highly Recommended But Not My Jam

The E-Myth Manager

A boss asked the whole team to read this book. I hated it. I think the thesis of the book was that we need to be more empowered somehow? I can’t remember any of it, because all I can remember is how stupid the business fable at the center of this book was. It was basically some wise elder executive taking an eager young pup under his wing, answering questions like “Golly gee, how did you ever manage to unite the team and produce more shareholder value?” Then the wise elder executive would take the young pup out golfing, sit majestically on the golf cart, and tell a story about synergy or something. Incredibly cheesy and contrived and patronizing. It turns out I really don’t like story-driven business books, because most business writers are not good storytellers. It just feels like reading bad Jack Welch fan-fiction.

Five Dysfunctions of a Team

I do not remember what the five dysfunctions of a team are. I read this right when I started my current job, and for some reason none of it stuck in my head. I think there was something about trust? Again, it felt very fable-based instead of principle-based or pragmatic.

The Dev-Ops Handbook

This book felt like mostly styrofoam, and the nuggets of good stuff were interspersed in with lots of fluff. Maybe I’m not the audience for the book. I could see it being more useful for leadership at large companies trying to get up to speed, but it felt overly vague and obvious to me.

Measure What Matters

This book would be a great blog post, and is a great one-hour talk. It is a very fluffy book. The basic idea is great. You should use objectives and key results (OKR’s) to structure your work. You should be setting broad goals (“objectives”) with measurable intermediate steps (“key results”) to achieve those goals, and you should be periodically evaluating both your broad goals and your intermediate steps. There are some other ideas about cascading these throughout the org chart, so theoretically your objectives help fulfill your boss’s key results. This idea is from Andy Grove, and he talks about it in High Output Management, a better book all-around that you should read instead. The rest of this book is back-patting stories about the magic of setting goals, including a particularly cringe-worthy triumphant success story about how YouTube obsessed over getting to one billion hours of watch time as a measurable metric of success.

Our job was to keep people engaged and hanging out with us. By definition, viewers are happier watching seven minutes of a ten-minute video (or even two minutes of a ten-minute video) than all of a one-minute video.

WAT

They prioritized longer videos in recommendations, tweaked their recommendation algorithms to optimize for watch time, cursed the summer because people went outside more and spent less time on YouTube, and maybe accidentally led to the rise in modern extremism. But hey, at least they achieved their key result of increasing watch time! Ok, gotta move on before I turn this in to a rant about engagement. Anyways, read the blog posts about OKR’s, but maybe skip the book.

Crucial Conversations

I want to get better at hard conversations - conversations with strong underlying disagreement, or where people don’t get along, or where the outcome could be painful to one or both sides. I’ve read this book a few times because its promise sounds so enticing - what if these conversations weren’t so stressful and awful? What if I felt good about the outcome? Unfortunately, I haven’t been able to apply the advice in this book successfully yet. It felt very fuzzy and impractical. A phrase that recurs in the book is “the shared pool of meaning”, which refers to the shared understanding the two people having the conversations have. You’re supposed to add to the shared pool of meaning which will make it easier to resolve conflict and discuss hard things. In hard conversations I haven’t found the problem to be easily solved by coming back to what we had in common, since usually these conversations stem from some deep conflict in personalities or values.

For each of these books I know smart people who love them, but they were not for me.

Others I’ve Enjoyed

I’m always looking for new recommendations. What books have you enjoyed?

Engineering Management Books was originally published by Jamison Dance at Interpreted Dance on September 10, 2019.

Protocols and Etiquette

2019-08-24T00:00:00-06:00

Recently I’ve been teaching networking concepts to front-end developers. Protocols come up frequently in networking, but I’ve found it hard to explain what a protocol is in general, separate from a talking about a specific protocol.

The word “protocol” gets tossed quite a bit in computing. It shows up in a bunch of acronyms. HTTP is Hyper Text Transfer Protocol, TCP is Transmission Control Protocol, and I’ve renamed dinner to Food Ingestion Protocol in order to spark joy in my family. It has not worked.

What is a protocol actually though? A protocol is a definition for how things communicate. Think of it like an etiquette.

Apparently there an etiquette for communicating with your knife and fork at 🎩FANCY👸 meals. This image filled me with a spicy blast of anxiety as I thought of all the times I accidentally dropped my utensils on the plate in the wrong configuration telling my mom I didn’t like the macaroni and cheese she made for me.

An etiquette is a set of rules for communicating. Etiquette defines what format you use to send messages. In this case utensils arranged in shapes send messages. Etiquette tells the sender and receiver what those messages mean and how to respond to them. Probably a butler will be summoned or something. It defines the message format and message meaning - the syntax and semantics. A different etiquette would have a different way of sending messages and different meanings. For example, there is an etiquette for Business Emails: eschew emojis, keep them short and easy to scan, say “per my last email” instead of “are you kidding me”, etc.

You could think of HTTP like an etiquette for how two applications communicate. Just like fork etiquette, HTTP defines a message format. Clients communicate by sending HTTP requests to a hostname at a specific path with a specified verb, headers, and body. Servers reply by sending HTTP responses with a status code, headers and a response body. HTTP defines what those messages mean, and gives rules for sending and receiving them.

You can read about the etiquette of HTTP in the RFCs: mainly https://tools.ietf.org/html/rfc7230 and https://tools.ietf.org/html/rfc7231.

Network protocols don’t actually exist as physical artifacts. There isn’t a solid block of aluminum that represents HTTP sitting under a glass case somewhere. They are written down sometimes (you can read the definition of the HTTP protocol in the RFCs), but they only get reified by code running on a routers and clients and servers and switches. They are abstract concepts that affect our life as developers and users of software consumers. TCP slow start is a tool for congestion control built in to TCP. Basically, packets sent at the beginning of a TCP connection are smaller and gradually ramp up in size. That means if you make many TCP connections for sending small files, each of those connections will go through slow start. This translates to common front-end perf advice - bundle your CSS or JavaScript instead of serving each file individually. It also is a core motivation for HTTP/2 - by multiplexing HTTP requests in to the same TCP connection, we avoid slow start without having to manually bundle files.

Protocols and Etiquette was originally published by Jamison Dance at Interpreted Dance on August 24, 2019.

Curating Community And Culture

2019-08-14T00:00:00-06:00

I co-host a podcast, Soft Skills Engineering, with my friend Dave Smith. We recently decided to create a Slack for listeners to chat about soft skills. As this group grows, it’s important to me that the culture and values of the podcast also persist in to this chat room. How do you curate community and culture? I’ve started a few meets, run a few conferences, and started a couple of podcasts now, so I’ve wrestled with this problem a bit. Here are some thoughts.

Deliberate Culture

Whenever there is a group, there is a culture. Culture can be organic and evolved or deliberately created and curated. Like gas expanding in a vacuum takes the shape of its container, community will assume the shape of its culture. Deliberate thought gives you a chance to shape the container. Internet culture that evolves organically tends towards caustic chaos (think about what an average Facebook wall or Twitter feed or 4chan thread looks like).

Group Size

🚨ALERT🚨: wild speculation on group dynamics, anthropology, sociology, etc, none of which I am an expert on.

Dunbar’s number hypothesizes that “humans can comfortably maintain 150 stable relationships.” If true, this puts a rough upper bound on the size of communities that rely purely on relationships and convention. Beyond ~150 we might not be able to to keep models in our heads of other people or know much about them that persists beyond direct interactions. This might be why large loosely-structured communities so often devolve. As the community grows, the proportion of the community that you don’t have a relationship with but still communicate grows, making it easier to misunderstand, attack, stereotype, etc. The Eternal September is an early internet example of this, but I suspect that as long as humans have formed groups there have been original members decrying the loss of the original culture as the group scales beyond relationship-size.

Deliberate Thought

Social dynamics make culture hard to scale, but I think you can counteract them with deliberate thought and structure. If you want to create and maintain a defined culture, you need to spend some brain cycles on it. What does your ideal culture look like? What kinds of messages, ideas, and interactions do you want to encourage and discourage? You should have some answers for this. It’s ok if your ideas evolve over time, but thought up front will go a long way towards setting the tone for your community.

I want the Soft Skills chat to feel friendly, comfortable, inclusive, safe, and a bit whimsical. I want people to share experiences and advice with each other, joke around, and support each other.

Writing

Now that you have some idea of what you want your community to feel like, write that down! Writing things down makes your ideas scale. It gives your group something to refer to. It helps clarify your thinking.

I made https://softskills.audio/slack with some communication guidelines, what folks should do when they join the chat room, and a code of conduct. I also created a slack bot that welcomes new members with a private message and reminds them to read the communication guidelines.

Action

After thinking and writing things down, act to make it stick. Curated culture requires deliberate action to encourage or discourage certain behaviors and establish structure for interactions. Sometimes curation means clarifying rejecting behavior that doesn’t fit the culture.

Within the first few weeks of setting up the chat room there was a relatively minor disagreement. It got a bit heated. I was torn between talking it out privately or addressing it in the main channel where it happened. Publicly call-outs are less comfortable and potentially embarrass the recipient. Eventually I decided re-enforcing expectations publicly was important enough to kindly address what happened. I replied with the following message in the thread:

[Y]our comments here don’t feel kind, constructive, or helpful, and feel demeaning. This is against our code of conduct (https://softskills.audio/code-of-conduct, specifically the section on Respect). Please work harder to communicate with respect in this slack.

Problems that happen publicly but are handled privately give the impression that they aren’t solved, and I wanted to make it clear we care about this stuff. I don’t really know the long-term effect of commenting publicly in threads like this, and I don’t think the person on the receiving end enjoyed it, but I hope it clarifies expectations without making people feel ostracized for messing up.

As part of making an inclusive culture stick, I created automated replies to gently nudge towards inclusive language. So far the reaction to this has been positive - people have edited their posts to be more inclusive, no one has dogpiled on them about it, and I’ve seen several messages of appreciation from the community.

Some Results

I’m very happy with the results. The community feels delightful! New members join the #intros channel and introduce themselves. A significant fraction of the members jump in to say hi in a thread for each new person that joins. People bring up questions or concerns they’re facing at work, and other members weigh in with thoughtful advice or relevant experience. Small numbers make community easier (see the earlier discussion of Dunbar’s number), so to some extent we’re operating in easy mode, but I hope deliberate action at the beginning helps the culture scale as the community grows.

Copy Our Stuff

Our welcome message, code of conduct and incident handling policy are on GitHub: https://github.com/soft-skills-engineering/website/blob/gh-pages/slack.md, https://github.com/soft-skills-engineering/website/blob/gh-pages/code-of-conduct.md, and https://github.com/soft-skills-engineering/website/blob/gh-pages/incident-process.md. We copied then from somewhere else (with attribution), so you can copy them to! We use https://github.com/orliesaurus/greetingslack to welcome new members.

Here is our list of Slackbot replies when someone says "you guys":

    Do you mean *team*? Let's use inclusive language; many people are not guys.
    Do you mean *folks*? Let's use inclusive language; many people are not guys.
    Do you mean *all*? Let's use inclusive language; many people are not guys.
    Do you mean *everybody*? Let's use inclusive language; many people are not guys.
    Do you mean *y'all*? Let's use inclusive language; many people are not guys.
    Do you mean *you all*? Let's use inclusive language; many people are not guys.
    Instead of *guys*, perhaps you mean *pals*? Let's use inclusive language; many people are not guys.
    Instead of *guys*, perhaps you mean *gang*? Let's use inclusive language; many people are not guys.
    Instead of *guys*, perhaps you mean *crew*? It's more inclusive than *guys*. :)
    Instead of *guys*, perhaps you mean *people*? It's more inclusive than *guys*. :)
    Instead of *guys*, perhaps you mean *y'all*? It's more inclusive than *guys*. :)
    Instead of *guys*, perhaps you mean *everyone*? It's more inclusive than *guys*. :)
    Instead of *guys*, perhaps you mean *folks*? It's more inclusive than *guys*. :)
    Instead of *guys*, perhaps you mean *fellow humans*? It's more inclusive than *guys*. :)
    Instead of *guys*, have you considered a more gender-neutral pronoun like *folks*? It's more inclusive than *guys*. :)
    Have you considered using a different pronoun instead of *guys*? Because many here (including me) aren't guys. :)

Community Is Everywhere

Curating culture and community applies to more than just setting up chat rooms. Community and culture applies any time a group of humans organize to do a thing. If you’re starting a meetup, joining a team, forming a company, running a project, entering a bootcamp, putting on a conference, or something else, you have the opportunity to deliberately influence the community and culture.

I have plenty to learn still. What have you seen help curate community and culture? I’d love to talk more about this! You can bug me on Twitter at @jamison_dance or email me at hi@jamison.dance.

Curating Community And Culture was originally published by Jamison Dance at Interpreted Dance on August 14, 2019.

Deconstruct Conf 2019 Recap

2019-07-16T00:00:00-06:00

I just got back from Deconstruct Conf. It was great! Deconstruct is a generalist software conference that focuses on fantastic speakers giving wonderful talks. I’ve also heard it described as TED talks for developers, or “that Gary Bernhardt conference”. It features wonderful speakers with both new and established voices, unlimited scholarships for under-represented folks or students, a gorgeous venue, no sponsorships, no swag, and a minimalist and bare-bones aesthetic. I could write a whole post on what I love about it as a conference organizer, but this one will focus on my experience as an attendee. I’ll try to summarize a few moments and themes I really enjoyed.

Games!

Ayla Myers talked about her adventures in multiplayer game development and the repeated struggles she had while trying to make it work. The content of the talk was great - the woes of trusting the client, dealing with lag and the struggles with client-side prediction. I’ve played games for quite a while but never really understood the reasons for strange behavior in laggy network conditions. She hand-made pixel art for her slides, which were gorgeous and full of character. I especially love the penguin jumping animation:

The structure of the talk was great. It played out like a fable - every three years, a return to the struggle of multiplayer game programming. Her Sisyphus reference felt very apropos. Ayla is hilarious too. She absolutely killed it on stage. She is a very talented technologist, game developer, and storyteller.

Ayla’s talk led me to playing around with Pico8. She made building games seem approachable and fun, as long as you stay away from multiplayer game engines.

Files, File Systems and Storage

Dan Luu talked at great speed about the file API, filesystems and disks, and how broken every single layer is. A phrase I wrote down was “impossible to use correctly, even for experts.” Dan quoted heavily from research, both from academia and his own work, to try to answer questions empirically that are often just debated on Reddit.

The talk was motivated by one of those offhanded smug dismissals programmers love to toss out about how easy a difficult thing is in software development, and that the jokers getting it wrong just simply need to do [obvious thing x]. It turns out safely writing files is complicated!

I really love Dan’s blog, so seeing more of this in person was wonderful. He is constantly peeling back layers and poking at conventional computing wisdom, and this talk felt like a fantastic entry in to that genre. Also, I love the shorthand of ✓∑ for checksum.

You can see a transcript of the talk here.

Intentional Friction

This phrase from Erica Gomez’s talk has been stuck in my head since her talk. She looked at the history of aviation, regulation, agile, software, and it’s focus on speed. The contrast between the way airplanes have been built and verified - ponderous but thorough - with the breakneck push for SPEED in software was fascinating. The talk highlighted the adoption of software-style pace in building the Boeing 737 Max, which famously has some software problems leading to crashes. The thesis of the talk is that the constant push for speed is expensive in hard-to-quantify ways. Burnout, ballooning tech debt, ethical problems, and a bunch more. The incentives are all set up to push us to go faster. The solution - intentional friction! Add things that slow you down and give you time to think, fix, secure, and consider. Pair programming, code reviews, curtailing working hours, refactoring and revisiting old designs are all things that slow us down but add value. We should be pushing for this!

There was also a thread about how people are an essential part of technical systems, with some great quotes from primary sources. She found the first reference to “software”, and it turns it was used to describe people, not programs!

“In addition, the interactions between these various elements, hardware and software (people), must be recognized and included as the glue that holds the system together.” - Richard Carhart, Proceedings of the Second Nation Symposium on Quality Control and Reliability in Electronics, 1956

People have always been an essential part of technical systems. http://www.historyofinformation.com/detail.php?entryid=936 looks like a fascinating link to dig more in to this.

I loved this talk. I took tons of notes, and have been thinking about its themes since I saw it. What kinds of intentional friction are most useful? How can I add them in ways that are helpful to the team but also to our users? What is my responsibility to push back against the push for ever more, ever faster?

Voice-Computer Interfaces

Emily spoke about her own personal struggles with RSI that led to her writing code and controlling her computer exclusively with her voice. It was probably the best live demo I’ve seen. She controlled her presentation with her voice, showed how it would work if you used the built-in operating system tools for coding with voice (basically impossible), and then moved to demoing and explaining her setup. She demoed writing Perl and Python (jaw-droppingly fast). She then dove deep in to the tools she uses, the ways she’s customized her environment, her custom snippets, phonetic alphabet, using ordinal numbers to repeat commands, etc. It felt like someone showing off their very well-optimized editor setup, except it was all controlled by voice, and it worked really well. It felt like a glimpse of the future, but it works today.

She uses Dragon for speech-to-text and Talon to program and customize her environment. Her Talon configs are here. The person who makes Talon has a Patreon, which should be making more money than it does!

This was also a great talk about accessibility, despite not being explicitly about accessibility. Apps that aren’t accessible don’t lend themselves well to being controlled by voice.

https://www.youtube.com/watch?v=Mz3JeYfBTcY is a version of the talk given at a different conference. This talk was rad.

Research and History

One of my favorite things about Deconstruct is how much value the talks place on history and the past. Computing has a long and rich history of great ideas and valuable experience. That often gets lost in the rush towards the new hotness of the day. Several of the talks focused on or featured research - Hillel Wayne talked about his journey through primary sources to try to answer the question “Why do we ask people to reverse linked lists in interviews?”. Erica Gomez’s talk dove deeply in to aviation, regulation, and software history. Dan’s talk featured both academic research on filesystems and disks, and his own personal research verifying results. Kyle Kingsbury shared new findings from Jepsen, his project on verifying correctness in distributed systems. Dan Abramov’s talk was partially about the history of best practices, and examining the consequences of emphasizing Don’t Repeat Yourself without looking at the damage it might cause.

Deconstruct is one of my favorite conferences and this year lived up to my expectations. It featured kind people, a relaxed atmosphere and thought-provoking talks. I’m excited to go back.

Deconstruct Conf 2019 Recap was originally published by Jamison Dance at Interpreted Dance on July 16, 2019.

How To Write A Bug Report

2019-05-01T00:00:00-06:00

A bug report has three main parts: what you expected to see, what you saw and steps to reproduce. This structure makes it easier for the team resolving to fix bugs you report.

What You Expected To See

Including what you expected to see provides context. Maybe you know how the feature should work, but the bug fixer could be looking at this feature for the first time. “What you expected to see” doesn’t have to be a detailed design document, but it should give the bug fixer a rough idea of what was wrong with what you saw. For example:

“I expected some indication that a pug test was running.”

Sometimes what you expected to see is really obvious. If the UI is rendering as a blank screen, you could leave out saying “it should render the app.” For most things though, adding what you expected to see is important detail for the bug fixer.

What You Saw

What tipped you off that something was wrong? Saying “test pug release is broken” doesn’t give very much information. How is it broken? Is it testing the spider release instead? Is it crashing your browser?

Instead, say what you saw that seemed wrong:

“Nothing happened on the screen either time I clicked the button, but when I looked out my window my front yard was filled with delightful pugs.”

Include screenshots, stack traces, responses, error messages, or other output from the system if you have them.

Steps To Reproduce

How did you reveal the bug? The developer fixing the bug will need to reproduce it. Clear steps to reproduce make this easy, which makes fixing the bug easier. Here’s an example:

Log in to www.release-the-pugs.com with a username of “user” and a password of “password”.
Click the “Test Pug Release” button in the top left corner.
Wait 30 seconds.
Click the “Test Pug Release” button again.

The steps to reproduce should include specific data you used, like request bodies, form input values or configuration options. You’re trying to give enough context that the bug fixer can see the problem in their test environment and fix it.

For really obvious things, you could combine the parts of a bug report in to one sentence: “The login screen in production is rendering as a blank screen.” For most things though, more detail and structure is better.

Sadly, software has bugs. Writing good bug reports helps us fix them faster.

This is a repost of a private I wrote for my team at Walmart Labs. Thanks to Chase Adams for editing help.

How To Write A Bug Report was originally published by Jamison Dance at Interpreted Dance on May 01, 2019.

Starting A New Job And Noticing Things

2019-03-16T00:00:00-06:00

Here is a typical interaction I see with new hires at every place I’ve worked.

New Hire: “I’m really excited to start here, but I just have a few questions.”

Employee: “We are so glad to have you!”

New Hire: “Why is there a machine on your desk that punches you in the face every 10 minutes? That seems like it would make it hard to get any work done.”

Employee: “Oh, it is so much better now. It used to have big spiky brass knuckles on it and now it is just this pillowy soft boxing glove. Things are really improving here.”

I’m going to mis-quote Julia Evans from memory because I can’t find the original source.

Starting a new job gives you the super-power of noticing things.

As a new team member, you have the power to notice weird or broken things that longer-term folks are just used to already. Maybe it’s a complex deploy process, performance problems that seem to be ignored, broken features, long-standing bugs, a punishing on-call schedule, strangely political code reviews, or the team member everyone avoids.

This super-power declines in me over time as I acclimate to my environment. Weird things become routine, pain repeatedly felt generates scar tissue, and I stop noticing broken things. That is one benefit to bringing in new team members - it gives you a chance to notice things through their eyes.

The power to notice things is different from the power to fix things. Fixing long-standing annoyances that have become routine is rarely controversial, but cultural problems are much harder to change. You notice things by being new, but you fix things, especially hard cultural things, by being a trusted team member. Still, noticing is the first step to fixing.

If you’re starting a new job, use your superpower. Point out weird things you see. Expect them to get better. If a new team-member joins your team, use their superpower. Explicitly tell them you want to know what they notice. Make it safe to report weird things they see. Fix them. Your whole team will benefit.

Starting A New Job And Noticing Things was originally published by Jamison Dance at Interpreted Dance on March 16, 2019.

Binding Multiple Sockets To The Same Port

2018-05-27T00:00:00-06:00

I’ve been a professional developer for about a decade, but there are tons of things I still don’t know. That is part of what is rad about software: there are so many things to learn!

I’m reading a book about networking to try to learn more of these things. While reading the section about UDP I saw this sentence:

[T]he transport layer assigns a port number in the range 1024 to 65535 that is currently not being used by any other UDP port in the host.

“that is currently not being used by any other UDP port” caught my eye. I always thought only one socket of any kind, TCP or UDP, could be bound to a port at once.

I can write a program to check this. Let’s do that!

Binding two UDP sockets to the same port

This should not work. Here is a small go program that demonstrates this:

package main

import (
	"log"
	"net"
)

func main() {
	addr := &net.UDPAddr{
		IP:   net.IPv4(127, 0, 0, 1),
		Port: 5556,
	}
	if _, err := net.ListenUDP("udp", addr); err != nil {
		log.Fatal("error making first udp socket", err)
	}

	if _, err := net.ListenUDP("udp", addr); err != nil {
		log.Fatal("error making second udp socket", err)
	}
}

As expected, running this causes an error:

➜  socket-stuffs go run main.go
2018/05/27 17:19:33 error making second udp socketlisten udp 127.0.0.1:5556: bind: address already in use
exit status 1

Binding a TCP and UDP socket to the same port

Time to see if the book is FULL OF LIES or not:

package main

import (
	"log"
	"net"
)

func main() {
	addr := &net.UDPAddr{
		IP:   net.IPv4(127, 0, 0, 1),
		Port: 5556,
	}
	if _, err := net.ListenUDP("udp", addr); err != nil {
		log.Fatal("error making udp socket", err)
	}

	if _, err := net.Listen("tcp", "127.0.0.1:5556"); err != nil {
		log.Fatal("error mkaing tcp socket", err)
	}
}

This works:

➜  socket-stuffs go run main.go
➜  socket-stuffs echo $?
0

An aside: I’m not sure why the signature for net.ListenUDP is different from net.Listen. Why do you make a struct in net.ListenUDP, and mash all the fields in to a string in net.Listen? ¯\_(ツ)_/¯

Why does this happen?

I’m not sure why this happens. I thought ports were managed by the operating system, and binding to a port “used up” a slot for that port in some internal kernel data structure. It looks like there is a slot per (port, protocol) pair? I’d like to understand more about those data structures and how the operating system manages them.

Binding Multiple Sockets To The Same Port was originally published by Jamison Dance at Interpreted Dance on May 27, 2018.