Micro-Iterations, Maps & “Interruptibility”

Image

When most software developers are asked about productivity, one thing that often gets mentioned is interruptions. We perceive ourselves to be more productive when we can maintain a deep state of concentration (“flow”) for extended periods, and interruptions break that flow. They may cite research that shows that it can take 15-20 minutes to get back into that state of concentration – to “reload the program”, so to speak. 2-3 interruptions in an hour will cost you that hour, they’ll tell you.

So, “interruptions are bad” is widely-received wisdom among software developers. And, like so much received wisdom in our profession, it’s not true.

Not all interruptions are bad. Not all interruptions waste our time. If someone’s editing the same source file as me, maybe an interruption could save us both time. If someone’s writing code based on a misunderstanding about the requirement, maybe sticking your head around their proverbial door might not be such a bad idea.

More generally, when we’re in that state of flow, one thing we’re not doing is communicating. If we think of development teams as concurrent systems, the dependencies between the work we’re doing will often make it necessary for us to synchronise. There’s actually a limited amount we can do genuinely in parallel. And, I suppose, that’s why we’re a team.

The longer we go without synchronising – without communicating – the more out of step we get with each other; more misunderstandings, more conflicts, more duplication of effort, and ultimately more work later getting back in sync.

So, we need to focus, because – y’know – programming, but we also need to stay in sync. How do we square this circle?

First of all, why does it take so long to get back into that state of flow? It’s a matter of cognitive load. We’re carrying a model of the code, the requirements, the architecture, in our heads – it’s all loaded into “main memory”, if you like. When we’re interrupted, it’s like terminating the program – it all has to be loaded back into memory again.

What if we could reduce that cognitive load? This is where working in smaller steps can help enormously. Micro-iterative processes like Test-Driven Development and refactoring let us focus on one thing – solve one problem – at a time. The amount of detail we need to load into memory is much smaller.

Working in tiny cycles that end with all the tests passing, and the changes committed, we find ourselves in a “safe zone” many times an hour where we could be interrupted without breaking our concentration.

Image

We can combine micro-iterations with “maps” – like test lists, high-level design sketches (e.g., a sequence diagram), UI storyboards, Mikado Method plans – that help us keep our place in a longer task, externalising most of the cognitive load. So we’re progressing one step at a time, and we know roughly where we’re going.

Developers who work this way do indeed report that interruptions are less of a problem. When the tests are green and the changes are committed, we’re free to talk, 5, 10, 20 times an hour.

Of course, flow still matters – let’s call them “micro-flows” – and if I’m in the middle of writing the code to pass a test, I don’t want to be interrupted. It’s very helpful to be able to observe team members working, so we can see when they might be interruptible. Some teams even have protocols and/or visual signals, kind of like those red lights outside movie stage doors.

Anyhoo, yes, we can have our cake and eat it; by reducing and externalising cognitive load, and working in small iterations that bring us back to tested, backed-up code many times an hour, we can achieve the focus we need – typically, much more focus on every change we make than in extended periods of “flow” – and communicate often enough to stay in sync with the team.

And the added benefit is that, if being interruptible means all our tests are passing and our changes have been committed, it also means the code’s shippable – 5, 10, 20 times an hour.

What If FOMO Itself Is Why We Miss Out?

Image

In my early 20s, I went to Inverness with a group of friends for a long weekend.

On the looong journey up, we became obsessed with finding the Loch Ness monster, and ultimately had no fun when we were there at all.

We spent the whole 4 days cycling up and down the loch with cheap cameras we’d bought from the local branch of Dixons.

Up and down we went, splitting into pairs to cover more of the loch. We’d start straight after breakfast, and ride the south road until it got dark – which was midnight at that time of year.

We’d take breaks, of course. We’d meet up, often at the cafe & shop in Foyers, so we could sit on the bench overlooking the falls and discuss our strategies for improving our odds of spotting Nessie. What if we split up individually, cover more ground? What if half of us go to the north shore? What if Nessie mostly comes out at night? Should we buy powerful torches, work in shifts? Should we hire a boat? One with sonar, perhaps? etc etc.

And, of course, we had all the books with us, which are very easy to find in Inverness. Nessie is big business in that part of the world, and the local owners of hotels, shops, bars and tour boats are always happy to encourage tourists hoping to catch a glimpse of the monster.

Dozens of eyewitness accounts and blurry (and in many cases now-known-to-be fake) photos, dozens of maps guiding us to where people had seen the wee beastie. We became convinced that the reason we hadn’t seen her was because WE WERE DOING IT WRONG.

Needless to say, after we got back, we all felt pretty stupid. They told us at the visitor centre why there really couldn’t be an animal of that size actually living in the loch, let alone a prehistoric one.

They methodically laid out all the evidence – lack of fish stocks to support such a large predator, the relatively young age of the loch (10,000 years), the fact that a plesiosaur has to surface often for air – on and on it went, nail after nail in Nessie’s coffin.

We listened to all of that, took it all in, and then we walked out of the visitor centre and thought “But what if a giant dinosaur got into the loch?” and then wasted 4 days of valuable holiday (i.e., drinking) time looking for something that almost certainly isn’t there.

All that beautiful scenery, all that culture and history. All those pubs, all that whiskey! We missed out on it completely.

…..

Now, none of that actually happened. But if it had happened, I suspect we might have goaded each other on with the threat of “missing out” on seeing the monster. Despite the complete lack of any credible, verifiable evidence that it exists. And despite all the very valid scientific reasons why it almost certainly doesn’t.

But FOMO is a powerful motivator.

D’ya see what I’m getting at here?

Enterprise Refactoring Requires Enterprise Tests

Image

A book that had quite an impact on me was Ubiquity: The Science of History by Mark Buchanan. It proposes that many catastrophic events are ultimately caused by the interconnectedness of things (banking system collapses, forest fires etc). The more interconnected the system, the more catastrophically effects can “ripple” out through those connections.

In software, we see these network effects as failures propagate through dependencies between components and systems. We also see how change can propagate for the same reasons. I’ve watched many dev organisations grapple with what could have been small changes to their software that ended up impacting every team because their applications, components and services were so tightly coupled.

At the level of source files (e.g., .java files or .py files), we can decouple modules by moving responsibilities to where the majority of their dependencies are. Things that get used together belong together, and things that get used together change together (and fail together).

Moving a feature from one Java class to another is easy as peas, even doing it with manual edits. Moving a feature from a Java web service to a Python web service maintained by a different team, or from a COBOL CICS system to a C# application, requires an order of magnitude more coordination. Which is why, when “Feature Envy” appears at that level – when a system or component is coupled to multiple features of a different system or component, indicating that behaviour may be in the wrong place – most organisations do nothing about it.

What we might call “enterprise refactoring” is a discipline that could benefit many organisations, though. And what distinguishes refactoring at any level of code organisation from just changing stuff willy-nilly is testing.

If I move a method from one Java class to another, I retest that code at a higher level for behaviours that involve both classes.

When we move a feature from one system or component to another, we again need to retest at a higher level, checking scenarios that involve both components or both systems – and these are typically business scenarios.

Many dev orgs lack tests at that level. They may see tests failing when a change breaks a component, but no tests fail when that change breaks the business. (This is why so many businesses can be blissfully unaware that, say, order fulfilment isn’t working. The POS system’s working. The warehouse system’s working. The shipping system’s working. But somewhere between them, the ball gets dropped.)

Indeed, too many organisations don’t actually know how their software’s being used in those wider contexts – business use cases, if you like. They understand what their system or component does, but have no real visibility of how it’s being used.

Businesses are systems, too. They have users and business use cases. These use cases are realised by internal processes, and they often involve multiple interacting software systems.

Refactoring those internal processes to localise the “ripple effect” of change is on a whole other level.