In my last article (Forget “Code Quality” and “Correctness” — Verifiability and Agility are the True First Principles of Software Engineering), I challenged conventional wisdom regarding code quality and correctness. Instead, I argued that verifiability and agility are the true first principles from which everything else stems. Now, let’s move from theory to practice.

This article focuses on agility — how can we build codebases that adapt to ever-changing requirements quickly, with minimal effort and never accumulate code debt. By mastering the techniques of code-size reduction and modular design, we can keep our projects flexible, scalable, and manageable over time.

Problem Statement

When I use the word agility, I’m not referring to “agile methodologies.” I’m talking about the dictionary definition: the ability to move quickly and easily. That’s what we want. We want our codebases to quickly and easily adapt to future, unforeseen requirements. But what does that mean exactly? We’re going to work toward a very specific goal:

The Agility Goal: Minimize the cost — in terms of developer hours — needed to deliver the next new requirement, averaged over the lifetime of the code.

There are two main challenges with this goal. First, how do you minimize the effort to meet any specific new requirement? Second, how do you set up the codebase to respond well to all future, unpredictable, and sometimes unimaginable requirements?

To answer these questions, we’ll work backward. Let’s assume we’ve just finished modifying the codebase to meet the latest requirements. By looking at the git history, we can see exactly what changes were made. The challenge wasn’t just making those edits — it was determining what needed to be changed.

To quantify the challenge of designing those edits, the first question is what knowledge you needed to do that design work. Let’s assume that, knowing the requirements, you already had a good guess at the algorithms and strategy you needed to solve the problem. The next and hardest task was figuring out how to implement that solution in actual code. To do this, you needed to understand all parts of the codebase that could possibly be relevant. Specifically, you had to grasp everything your changes might depend on, and everything that might depend on your changes. In other words, for both the edits you finalized and the exploratory steps you took, you needed to understand all the upstream and downstream dependencies involved.

I call this the “surface area” of an edit. Imagine zooming out until you can see your entire codebase as a 2D image. Now, highlight all the upstream and downstream implications of your edit. That highlighted region represents the surface area you had to understand to meet the new requirements — and ensure you didn’t break anything else along the way.

The Surface Area of a Change (“surface area” for short): the set union of all code-points a) that you changed or considered changing, b) all code-points those changes could depend on and c) all code-points that could be affected by your changes, directly or indirectly.

The real complexity emerges from the fact that any pair of code points in that surface area could potentially interact with each other. If they could further interact with something outside that surface area, you have to expand your surface area to include that too. You have to consider all potential upstream and downstream dependencies, not just the actual ones. For one, it’s impossible to fully understand all the actual dependencies in just one screenful of code, let alone 10, 100, or more screens of code for a more typical “surface area” of a change. For two, it’s not just “what is” that matters, but also “what you could potentially change it to be” that you have to contend with.

The complexity of a change is proportional to the number of potential interaction across the surface area of your change. This forms a fully connected graph of potential dependencies, and the cost of understanding all these interactions scales quadratically. If the surface area doubles in size, the complexity quadruples. If it’s 10x larger, the complexity multiplies by 100x. That is why this simple measurement, the size of your change’s surface area, will dominate all other software engineering considerations.

This is the fundamental nature of code debt. As code bases grow, if nothing is done to combat the problem, the cost of each subsequent change will grow quadratically. Consequently, productivity will plummet over time.

This is the true “cost” of meeting new requirements. The cost scales quadratically: O(n²) where “n” is the size of the surface area. Therefore, to minimize the cost of meeting future requirements, the dominant factor is minimizing the size of the surface area of the required changes.

Code Debt: Ever increasing surface-area for each subsequent change resulting in the collapse of productivity over time.

Let’s take a moment to consider what other software engineering considerations might matter. How about using a more functional vs a more stateful language or style? Being more functional reduces side effects. In other words, it reduces what other parts of the code your changes could impact. Which means it is simply one strategy for reducing the surface area of future changes.

How about well formatted, consistent and linted code? While some of that does help with human comprehension, it is just a constant-factor in the O(k*n²) equation. It doesn’t reduce the complexity of code-code interactions. It’s worth doing, but not if it increases n! Any technology, tool or coding standard that increases the surface area of the next change will slow you down.

So… how do we minimize the surface area of changes due to future, unknowable requirements? We need a strategy.

Surface Area Minimization Strategy

To overcome the challenges of complexity and minimize the surface area of future changes, we need two complementary strategies: code-size reduction and modular design. Each plays a key role in keeping codebases agile.

1. Code-Size Reduction

The first obvious way to reduce the surface area of future changes is to reduce our code size in general. Obviously, this will help, and since each eliminated token reduces the base of the quadratic function, it has more impact than you might think. A 10% decrease in code size results in a 20% decrease in complexity. A 2x decrease in code size results in a 4x reduction in complexity.

It gets even better. As you eliminate each extra token and line, you’ll uncover new opportunities for further code-size reductions that were previously hidden in the noise. Sometimes this can lead to cascade unraveling of code complexity. Let’s say you’ve got two different data structures for users. This happens pretty often actually: you’ll have one for the user record in your database that your ORM provides and you’ll have one for your API. Now, having two different but similar data structures means your code needs to handle both. You’ll need to start writing functions that can take both. You’ll need to add converter functions between the two, etc. All due to the fact that you have extra complexity due to having two data-types when one would suffice. Now you’ve added mountains of additional complexity to manage it. If instead you standardized on just one user data-type (typically the API one) and immediately convert to and from it when reading from the database, all other code in your code-base becomes much simpler.

The general process is, as you work, you’ll begin to spot repeated patterns in your code (like two nearly identical but slightly different user-record types). You’ll remember the cardinal rule: “Don’t Repeat Yourself” (DRY). When you DRY up a chunk of code, not only do you greatly decrease your code size, but you end up with a reusable abstraction. These “baby abstractions” are the first step toward an even more powerful strategy for minimizing code complexity: modular design.

Every single token you eliminate from your code matters. Not only does it decrease the base of quadratic-complexity scaling, but it often unlocks new, previous invisible opportunities to simplify and reduce your code. When in doubt between two versions of code, always select the smaller one.

2. Modular Design

While reducing code size is an essential first step, it’s not enough by itself. When code-size reduction hits its limits, Modular design can give you nearly boundless scaling potential. Modular design doesn’t just reduce the size of your code — it fundamentally changes how your code interacts with itself. By encapsulating implementation behind interfaces, you can reduce the surface area of a change by 10x or more. This happens because the outside world only needs to interact with a slim interface rather than the whole implementation, instantly cutting down complexity.

The magic doesn’t stop there. Modular design is recursive. Once you wrap one module behind an interface, you can repeat the process, creating modules within modules. This recursive approach means that the 10x reduction becomes exponential. Take 10 modules, each with an interface 10% the size of their implementation, and wrap them in a module. Now you have an interface which is a mere 1% the size of your implementation — a 100x100 = 10,000x decrease in complexity! And, for exactly the same reasons why operations on tree data structures take O(ln n) time, fully modularizing a codebase can convert complexity scaling from quadratic to sweet, sweet logarithmic scaling.

And before you object to this as an absurd example and that in the real world it can’t be that good… consider that your current project has about 100 million lines of code. You read that right. I’m not exaggerating. We are all creating projects who’s total size, if you include the entire engineering stack that makes it possible, the OS, the CPU, the language, the database, the runtime, the dozens of 3rd party modules not to mention the cloud infrastructure, is easily that much code. How can we use that much code if complexity scales quadratically with code size? We can use it because all those technologies are exceptionally well honed modules that hide most their complexity behind much, much simpler interfaces. Modular. Design. Works.

3. Anticipating Unknown, and Unknowable Future Requirements

Above, we established two goals we are attempting to solve:

1. minimize the time it takes to make a specific change — which means minimizing the surface area of that change
2. set up our codebase to easily adapt to endless, unanticipatable changes — which means setting up the codebase such that future changes’ surface areas will be minimal

We’ve established that in general, minimizing our overall codebase size and maximizing modular isolation can work to minimize the surface area of a change, and that is by far the most effective way to minimize the time it takes to implement it. What we haven’t looked at quite so closely is how this helps us prepare for the future. We know there will be more requirements, and while we might be able to guess at some of them, we know there will be just as many unforeseen ones.

It turns out that even though we may not know what future requirements may be exactly, for most projects, most future requirements come in similar shapes. If you are building a database backed website, it’s highly likely that future requirements will either require that you add new tables or columns to the database and/or new web pages or web-page elements. Knowing that, you can measure and minimize how much code (measured in language tokens) it takes to add a new web page or new record type to your codebase. Minimize how many files you need to touch, how many lines you need to change in those files and how many new lines you need to add. The difference between an agile code base and an everyday code base under these simple metrics can be 10x or more.

More generally, our two strategies from above for minimizing surface-areas will work well for any future requirement. Keeping the overall code-size minimal decreases the amount of code that could possibly be in the next change’s surface-area. Modular design, though, is even more powerful. I’ll put it in terms of the 5 Essential Elements of Modular Design:

• Purpose: If we maximize the clarity of purpose for every module in our system, new requirements are less likely to feel disconnected from our existing work. Instead, they will naturally align with the meta-structure we’ve established, often requiring changes to just one or two modules.
• Interface: A well defined interface together with strong encapsulation means that usually, if you are altering just one module, you can completely ignore all downstream dependencies — as long as you are only making minor or patch changes.
• Encapsulation: If your modules’ encapsulation is strong, i.e. your abstractions aren’t very leaky, you can truly ignore their internals. If, on the other hand, you have weak encapsulation, then “the surface area of the next change” will often include internals of your modules negating most of the advantages of having modules in the first place.
• Connection: A well modularized codebase is like Lego bricks. Under more extreme new requirements, which seemingly require larger refactors, a modular codebase can be reorganized like Legos. Many of the internals of modules won’t need to be changed nor contemplated — i.e. they will be outside the surface-area of the change. Further, a well modularized codebase is like a well-honed toolbox. When a new, unanticipated requirement comes in, you’ll have a wealth of tools to tackle the problem.
• Implementation: A well-architected module, one that is strong along the other 4 elements, gives you the opportunity to invest in a stronger implementation: performance, reliably, extensible. This transforms a module from being a nice place to hide some complexity to being an invaluable tool in your arsenal against ever-changing needs.

Finally, a bit of hard honesty. Though this method will greatly reduce the need for major refactors, nothing can completely eliminate them. Fundamental choices made at the beginning of a project, which may have been right at the time, may now be the wrong answer, especially after a major pivot. Still, if you’ve modularized your system, you have the best chance of reusing at least some of your code untouched.

The Path of Mastery

When I tell people all software engineering comes down to just one thing: write less code, they either don’t believe me. How could it be as simple as just making your code smaller? I hope now you have some sense that not only is it the dominant factor in any project’s chances of success, but also it isn’t such a simple idea after all. It’s a path of mastery. It’s something you get better and better at the more you practice it, without limit. Using “write less code” as a honing tool can be invaluable at guiding you down this path. It is a simple, objective way to measure your progress and to evaluate new ideas. Each time, you can ask “does this new technique actually reduce code-size, or more specifically, the surface-area of my next change?”, compare the two chunks of code, measure their size, choose the smaller one, and continue down the path of endless, objective improvement.

Anecdotally, I’ve found that decreasing code-size, one token at a time, is often like pulling on the thread of a sweater. At first progress is slow, but suddenly large parts of code collapse into far smaller and simpler solutions. Complexity begets complexity. Once you have overly complex code you must write more code to manage that complexity, compounding the problem. Conversely, just a little extra simplicity can often reveal even greater opportunities for simplification.

This insight has been critical to me in honing my own craft. I now evaluate all software technologies and methodologies against how well they minimize the surface-area of adapting to requirements.

I’ve written extensively on modular design and code-size reduction. If you are interested in further reading, I recommend The 5 Essential Elements of Modular Design, which outlines the 5 measures of a module and your overall modular system. I also recommend my more recent Organic Modular Design as a practical guide how to do modular design in the real world. If you’d like a concrete example of DRYing up a system, making it more modular and dramatically reducing it’s overall complexity (and size), check out Modular Redux — An Example of Applied Modular-Design for Mastering Scalable, Shared State in React.

Agility in the Age of LLMs

I feel I can’t really talk about code agility without addressing the elephant in the room: LLMs like ChatGPT and Claude are changing the game… and also not changing the game. They are tremendous at accelerating code generation, but so far, you still need to understand the code they generate. They screw up, they get lost and they still don’t really understand what they’re doing. As long as we, as software engineers, need to understand our code, everything I’ve said above applies.

I suspect that even when we don’t need to look at our code line by line, the ability to think systematically about problems and break them up into well-encapsulated sub-problems will still be essential to human-ai collaboration. While writing small code, token-by-token may matter less, mastering modular design will likely be eternally useful.

2026 Update: Even today, with Claude Code, Cursor and Codex, LLMs are just as constrained by the n-squared complexity scaling problem as humans are. Regularly coaching your coding agents to keep the code-base well modularized, organically, is still a critical, winning move. Share this article with them and ask them to suggest opportunities for improvement.

Conclusion

If you care about actually creating real world impact as a software engineer, then you care about how quickly you can deliver that impact. Everything we call “code quality” is really just code agility in disguise. Many so-called code-quality tools and practices actually hurt agility and, in my opinion, should be discarded.

Agility is essential to continually delivering success. By obsessively reducing code size and embracing continuous, organic modular design, you can create systems that remain flexible and manageable, even as they grow. Through strong encapsulation and recursive modular structures, you transform the quadratic scaling of complexity into something much more tractable. By making these practices a daily habit, you’ll ensure your codebase remains agile, scalable, and ready for the ever-evolving demands of real-world projects.

We’ve all been on projects bogged down by extreme technical debt — where productivity grinds to a halt, and no one wants to touch the code for fear of breaking something in production. It’s no mystery why projects slow down over time: in codebases where any piece of code can interact with any other, complexity scales quadratically with code size. I’ve talked a lot about this topic (links below), and the solution to this n-squared complexity growth is clear: modular design. But I realized I’ve never talked about how to do modular design in practice.

Modular design isn’t just for architects and principal engineers — it’s something every developer should be doing daily. Rather than being a rigid upfront exercise, modular design should evolve naturally alongside your code. I call my approach Organic Modular Design (OMD). It’s a practical, incremental way to discover modular opportunities as you build, ensuring that the modules you create are both relevant and useful to the project at every stage — and it helps keep your project maximially modular throughout, squashing the dreaded n-squared scaling and replacing it with smooth logarithmic sailing.

Also available on my Code & Optimism podcast and youtube.com/@code-and-optimism channel.

The Modular Lifecycle

At its heart, Organic Modular Design is about the modular lifecycle. At any given moment, your codebase is a garden of modules, each at various stages of maturity. I’ve talked a great deal about the 5 Essential Elements of Modular Design: Purpose, Interface, Encapsulation. Connection and Interface (PIE/CI— more resources linked at the end). These metrics provide an objective way to measure the quality of a module. With OMD, our goal as engineers is to continuously improve the modules we encounter in our daily work, nudging them forward in their lifecycle by enhancing one or more of the PIE/CI metrics. In this way, we can keep our codebase maximally modular throughout the development process.

Lifecycle Stage 0: Modular Genesis

Every module begins with the recognition of patterns. As you add new features or functionality, you’ll start to notice repetition or similarities in how tasks are structured. These emerging patterns signal the opportunity to introduce modularity.

The key is to act as soon as these patterns appear. Don’t wait until the repetition becomes overwhelming. When you spot recurring logic, it’s time to DRY (Don’t Repeat Yourself) up the code by extracting it into a function. This is the first step in Organic Modular Design — the function is the smallest possible module and represents the genesis of modularity in your codebase. You just created the seed of a new module. This is the first step toward reducing complexity as part of the modular lifecycle.

Pro Tip: DRY up patterns as soon as you see them; don’t wait.

Lifecycle Stage 1: Function Modules

After DRYing up your first repetition, you’ve created the smallest form of a module — a function. Functions are inherently strong modules because they come with language-enforced encapsulation. The implementation is hidden behind the function’s interface (its inputs and outputs), ensuring a separation between what the function does and how it does it.

At this stage, the function’s purpose — its name and stated intent — might still be unclear, and that’s perfectly fine. You don’t need to get the purpose perfect right away. As the function’s role in the codebase becomes clearer over time, you can refine its name and scope. As related functions start to emerge, you’ll naturally gather them into a file — leading to the next step in the modular lifecycle.

Lifecycle Stage 2: File Modules

As you accumulate DRYed-up functions, you naturally begin organizing them into files. And just like functions, files are modules. In fact, the file itself becomes the next level of modularity in your codebase.

At this stage, you’ll start applying the PIE/CI principles more explicitly. Every time you touch a file, think about its purpose — is the file serving a clear, cohesive role? The name of the file should reflect that purpose. It’s fine to start with something generic like lib, but as your codebase grows, more specific names like stringLib are better, and names like dateFormatters are even more precise. The more specific the name, the better it defines the module's boundaries—what it does and what it doesn't do. This specificity is crucial for clear modular design.

It’s perfectly OK to start by exporting all the functions you’ve gathered into the file. However, the real benefit of modular design comes from increasing the ratio of interface to implementation. A smaller, well-defined interface compared to a larger implementation leads to a massive reduction in complexity for clients of that module. For instance, if the interface is just 10% of the implementation’s size, it results in a 100x reduction in complexity. This leverage is the key to achieving logarithmic scaling and avoiding the quadratic complexity that can cripple projects.

Note, though exports are explicit in JavaScript, other languages like Python export everything by default. Given the power of “defaults,” Python typically takes a bit more intentional effort to minimize the exported interface.

As you accumulate a few of these file-modules, you may notice a group of related files. When that happens, you’ll start thinking about the next step in OMD: organizing them into a folder.

Lifecycle Stage 3: Folder Modules

As you accumulate a few file-modules, you might notice that several of them are related. This is when you’ll naturally start thinking about grouping them into a folder.

Up until now, I’ve covered concepts that most engineers are familiar with — treating functions and files as modules. However, in most JavaScript projects, folders are treated as loose collections of files, with no defined interface or encapsulation. This is where projects lose modularity and fall into the trap of growing complexity. In typical JavaScript codebases, any file can import from any other file, leading to a fully connected graph of dependencies. The result? Complexity scales quadratically with the number of files.

To avoid this, it’s critical to treat every folder as a module: define its purpose, explicitly specify its exports, and treat its files and subfolders as private — nothing outside your folder should import them directly.

Thankfully, JavaScript (and other languages) provides tools to support folders as modules. In JavaScript, you can use an index.js file to define the public interface of the folder. While JavaScript doesn’t enforce encapsulation for folder-modules, you and your team can adopt the rule to never import from private files inside a folder. For example, importing from "../../utils" is fine, but "../../utils/stringLib" breaks the module's encapsulation. Avoid adding a / after a word character in import strings.

It’s fine to start with an index.js that exports everything using export *. Sometimes, keeping export *s forever makes sense—especially for files whose job is to define the interface. As the module evolves, the goal is to minimize the number of exports over time, by either removing entire files from exports or exporting only what's necessary.

Another pattern to consider is the aggregate export, where the index.js file imports specific items—like route handlers—and exports a single variable, such as an array containing all the items. This approach works well when the client only cares about the set as a whole, allowing you to expand the module without requiring client changes.

The most common way to limit exports is to export each item individually. However, this is often more tedious than necessary, especially in nested folder structures. One of the above methods is often more efficient.

The key is that you don’t have to be perfect. You just have to make the folder a little more modular each time. As the module’s purpose becomes clearer, you’ll naturally reduce the exports and strengthen its boundaries.

Pro Tip: Treat every folder as a module. Never import from internal files — always go through the folder’s public interface.

Lifecycle Stage 4: Packaged Modules

At some point, you might want to consider packaging up your module. There are two main reasons for doing this. The first is straightforward: your module may mature enough that you want to reuse it in other projects or share it with the larger community by open-sourcing it. As the module gets used in new contexts, it can reveal important shortcomings or even bugs that were previously missed.

The second reason may be even more important. Packaging a module is the final step in the modular lifecycle and presents a major opportunity for improvement along each of the PIE/CI metrics:

• Purpose: Step back and carefully consider the module’s name. If you’re going to publish it, you’re committing to it long-term. The name should clearly reflect the module’s purpose, be short, and memorable. This is also the time to refine the README, with a concise pitch, a quick-start tutorial, and example uses.
• Interface: It’s time to document the interface, not just in code but in words. Provide a detailed explanation of how to use your module in the README, or even consider creating dedicated documentation.
• Encapsulation: Packaging enforces the strongest modular boundaries. As you package the module, you may discover unexpected dependencies on other parts of your project or third-party libraries. You might also find areas where your project is violating the module’s encapsulation by relying on private files or sub-folders.
• Connection: Packaging makes your upstream dependencies explicit in the package.json, giving you a chance to streamline or eliminate unnecessary dependencies. It’s also the perfect time to implement semantic versioning, ensuring clients can safely upgrade without encountering breaking changes.
• Implementation: With the module isolated, improving test coverage, code quality, and iterating on the implementation becomes easier, all without impacting the main project.

Packaging a module takes modular design to its highest level. It fully enforces boundaries, clarifies the module’s purpose, and ensures that it’s ready for use in contexts beyond its original project.

Embracing The Modular Lifecycle

As we’ve discussed, modular design starts with recognizing patterns in your code and DRYing them up — this is the foundation. The moment you spot repetition, you should immediately refactor it into a function. As your functions accumulate, the next step is to organize them into files. It’s perfectly fine to start with a broad interface where you export everything, but over time, work to refine that interface — reducing the number of exports and improving the ratio of interface to implementation. From there, related files naturally group into folders. Every folder should be treated as a module, with explicit exports and internal files and subfolders kept private. Like with files, it’s okay to start by exporting everything, but you should incrementally improve it. Eventually, some folders will mature into full-fledged packages. At this level, you create a fully mature module, complete with documentation, semantic versioning, and maximum encapsulation, ensuring a clean interface for external use.

The modular lifecycle is exactly that — a continuous cycle within your project. Once a module matures into a file, folder, or package, new “baby” modules start to emerge inside them. As your project evolves, you’ll discover new patterns within existing modules, which give rise to smaller, nested modules that will grow and eventually follow the same lifecycle. Mature modules give birth to new ones, and this cycle continues as long as the project is alive and growing. In this way, modular design is not a one-time task — it’s an ongoing, organic evolution of your codebase.

While it’s best to adopt Organic Modular Design (OMD) from the beginning of a project, any project — no matter how messy or old — can adopt OMD and begin the journey of controlling complexity through modular design, one day at a time.

By embracing Organic Modular Design, you can continuously improve your codebase in small, manageable steps. Each stage of the modular lifecycle brings increasing clarity, flexibility, and scalability to your project. The key is to treat modularity as an ongoing process that evolves alongside your project. Without modular design, complexity in a project will inevitably scale quadratically with its code size, making it harder to maintain. But by maximizing modularity, complexity can scale logarithmically instead, making your codebase more manageable and adaptable over time. As a bonus, you’ll also produce highly reusable packages that accelerate development in both this and future projects.

Additional Resources on Modular Design

• From my YouTube Channel, “Code and Optimism”: 5 Essential Elements of Modular Design, More about Purpose, More about Interface, More about Encapsulation, More about Connection, and More about Implementation. About Modular Testing.
• More from me on Medium: 5 Essential Elements of Modular Design and A Concrete, Step by Step Example of Modularizing Existing Code

Forget “Code Quality” and “Correctness” — Verifiability and Agility are the True First Principles of Software Engineering

You wouldn’t be challenged for saying great software engineering revolves around code quality and correctness, yet you’d be profoundly wrong. “Code quality” and “correctness” are inherently meaningless. What do these terms even mean? Every software team comes up with its own answer to these questions, usually based on a mix of dogma, personal opinions, and prejudice. The result often leads to decisions and practices that actively reduce a project’s chances of success.

If your principles are not based in reality, you are in serious danger of doing more harm than good.

As senior engineers, we’ve all experienced the frustration of trying to apply so-called ‘best practices’ only to find they don’t seem to work for our particular project. We end up questioning ourselves, wondering if we’re implementing them incorrectly. The reality is, the ‘best practice’ might not actually be ‘best’ for every situation — or at least, not for the specific challenges we face.

So, if best practices are unreliable, and “code quality” and “correctness” are meaningless, what truly matters? The answer can be expressed under two first principles: code verifiability and code agility. In this article, we’ll explore why thinking in terms of “code quality” and “correctness” is flawed and how it leads many engineering teams astray, ultimately hurting their ability to deliver valuable software in a timely manner. From there we’ll examine why verifiability and agility are better models by which to evaluate our software engineering decisions.

Principle #1: Forget Correctness — Strive for Verifiability

The problem with “code correctness” is that it doesn’t define what correctness is or why it matters. Without that guidance, engineers fill in the blanks themselves, creating arbitrary ideas of what correctness should be. They turn to tools that promise to enforce “correctness,” but without a clear measure, these tools often end up making it harder to build valuable software.

The whole point of building software is to solve real-world problems, and solving real-world problems means establishing real-world requirements. It begins by selecting a good problem to solve, then devising an effective strategy for solving it. Only then can we identify the set of essential, real-world requirements needed to solve the problem.

These requirements must be both essential and realistic. A set of requirements is essential if it is minimal yet sufficient to solve the problem. Realistic requirements can’t demand 100% accuracy or 100% reliability. Instead, they must specify the minimal number of “nines” — such as 99%, 99.9%, or 99.99% — of accuracy and reliability required. Since each additional “nine” costs an order of magnitude more to achieve, it is critical to select the minimal level of accuracy and reliability that is just enough to solve the problem.

The only thing that matters when building software is that it meets the strategy’s specific set of essential, realistic requirements. No one cares what the inside of the software looks like (other than the engineers). Much of what passes for the dogma of “software correctness” is irrelevant to any specific project and its unique requirements.

What matters isn’t that the software is “correct” in some arbitrary or abstract sense, but rather: a) does it meet the specific set of requirements, and b) how do we know? Thus, when it comes to software engineering — the practice of making all the millions of design decisions, from the top-level architecture down to the name of every variable — what ultimately matters is verifiability. How easy is it to verify if the software we built meets its requirements?

Verifiable (adjective): able to be checked to be true or accurate

Arbitrary notions of correctness often make it more difficult to validate that the code does what it needs to do. Verifiability puts correctness into context, asking: “Does this code fulfill the requirements?” That’s the only form of correctness that matters. Every single decision, big or small, should be measured by how easily we can verify if it helps us meet requirements.

Principle #2: Forget Quality — You Need Agility

As we’ve established, the goal of building software is to solve real-world problems. In reality, though, both the problem and the strategy for solving it evolve over time. As a result, the requirements for the software also change. Because of real-world competition and a constantly changing problem landscape, we must deliver our solution in a timely manner to achieve the impact we desire.

Therefore, as engineers, we must not aim to solve a fixed set of requirements but design our code to quickly adapt to ever-changing demands. We must engineer our code to be agile.

Agile (adjective): able to move quickly and easily.

Code agility is how quickly and easily you can modify your software to meet new needs. However, the common fixation on “code quality” often undermines this goal. The problem with “code quality” is that it doesn’t define what quality means or how it supports the software’s evolving purpose. Without clear guidance, engineers fill this void with arbitrary standards, often prioritizing aesthetics or trendy best practices that can actually hinder agility.

Engineers, in their pursuit of “quality,” may shape their code to fit an image of neatness and elegance, adding layers of abstraction or complexity. While these choices might look “clean” or “well-structured,” they often come at the cost of making the software harder to change when new requirements arise. In reality, what the code looks like doesn’t matter unless it impacts how quickly and efficiently the software can adapt to changes.

Agility provides a far clearer target. It isn’t about crafting beautiful or fashionable code; it’s about building software that can evolve with minimal friction. The faster you can adapt your software to meet changing requirements, the more valuable it becomes. That’s the only kind of “quality” worth pursuing: the kind that keeps your software light, flexible, and ready to meet whatever new challenges come its way.

Code is a Liability

Code is not an asset; it’s a liability. Every line of code brings a cost — the cost of engineering time and money needed to maintain it. The real asset is the value that the code delivers in the real world. No one outside of the engineering team cares what’s inside the code: not what cloud resources are used, what language was selected, which libraries are included, or even how the CI/CD pipeline is set up. All of that is just noise.

What truly matters is verifiably creating the impact we want in the real world, and doing it in a timely manner.

Because code itself is a liability, we as engineers must set aside our pride and personal preferences. The choices we make are not about satisfying aesthetic tastes or showcasing technical prowess. Instead, we need to continuously and objectively evaluate every software engineering decision. This means everything: the architecture, the language, the services, the libraries we use, right down to each variable name, file name, and how we structure functions, classes, and modules. If it doesn’t have a clear, net positive impact on verifiability and agility, it’s the wrong decision.

Software is a Powerful Tool for Solving Real-World Problems

While code itself may be a liability, software is a powerful tool for delivering real-world impact. Every decision must align with solving the problem at hand. It’s not just about solving it; it’s about solving it verifiably and with agility. We should focus on building software that delivers value, adapts easily to evolving requirements, and can be verified against the goals we’ve set. Code is merely a means to an end — a tool that must be wielded carefully to minimize its liability while maximizing the value it delivers.

As senior engineers, we have a unique responsibility and opportunity to shape our teams’ approach to software development. By proposing verifiability and agility as the benchmarks for decision-making, we can help our teams make effective, objective choices that are aligned with maximizing our project’s chances of success. In the end, it’s not about chasing an abstract idea of “code quality” or “correctness.” It’s about building software that meets real-world requirements, adapts to change, and consistently delivers value.

Further Reading…

I’m working on follow-up articles on this topic. The first one is out:

• Embracing Software Agility

Much of my previous work is also related. You can start with this one if your are curious how to put Agility into action:

• Organic Modular Design

Technology moves fast. So do customer expectations. Keeping an application up to date in this dynamic environment requires continuous innovation. However, the increasing pace of change in both technology and user expectations means that obsolescence is always nipping at your heels.

At some point, even the most well-designed and well-maintained products become unacceptably expensive and time-consuming to maintain. Beyond these hard costs, an outdated product can result in shrinking market share, ballooning costs, security risks, performance issues, or all of the above. At some point, the cost of a major rewrite becomes justified.

A well-done rewrite offers many opportunities. Discarding out-dated assumptions embedded into your product and tuning it for the modern market can reinvigorate your competitive edge. Taking advantage of modern hardware and software platforms can decrease operational costs while increasing scalability, reliability and security. A ground-up modernizing of your software can radically improve your agility, performance, and profitability.

Determining the right moment to pursue this level of change can be challenging. However, in our experience, the ROI tipping point often comes sooner than you might think-especially when taking into account the opportunity costs of keeping a less-competitive product alive and missing out on accelerated innovation.

Below are five common business scenarios that can show a compelling ROI for modernizing your software with a major rewrite. In each section, I’ll provide an overview of the problem, some typical symptoms, and some diagnostic questions to ask. Then, I’ll offer a few tips on how to get started thinking about your rewrite and what ROI goals you might want to target.

1. Reliability and Performance

Even a system that was once top-of-the-line can become a lumbering dinosaur a few years down the road. Whether customer expectations have grown or the technology just wasn’t built to handle today’s workloads, it’s showing its age. It might be slow, behave in unpredictable ways, or incur unacceptable levels of downtime.

Symptoms:

• Excessive downtime
• Excessive customer complaints about reliability and performance
• High average latencies on requests
• Excessive alarms waking your team up at night

Diagnostic questions:Solution:

Assess your application from the inside out. If you were to rewrite the core using today’s technologies, how much better would it be? Depending on how much of the back end needs rewriting, you might also consider re-inventing the front-end product experience as well. Somewhere around the 50% mark rewriting the entire application becomes easier than rewriting just half of it.

ROI goals:

• Turn detractors into enthusiastic supporters
• Redirect your engineering team from fighting fires to adding value to the product
• Sleep through the night

2. Scalability Limits

Your technology is a victim of success. You’re growing like crazy, but your service architecture can’t keep up. Sometimes your service scaled in a completely different way than you planned for. Sometimes there just wasn’t time to build a robust scalable platform for your MVP. Perhaps you performed a “lift-and-shift” of your on-premises infrastructure without taking full advantage of your cloud provider’s scaling technologies. To avoid flattening your growth curve and driving away customers, you might need to invest in a major modernization of your infrastructure.

Alternatively, perhaps you are courting a new enterprise customer that is an order of magnitude larger than any previous customer you’ve worked with. Perhaps your multi-tenant architecture was designed to scale in the number of tenants but was never designed to scale individual tenants to very large sizes. To service these larger customers, you’ll need whole new scaling capabilities.

Symptoms:

• Can’t keep up with growing demand
• Web servers’ latency spiking as they reach 100% utilization during peak hours
• Database servers falling over, requiring manual intervention to get back up and running
• Spending an excessive amount of time manually scaling your infrastructure to manage load
• Courting a big, new client, but the system architecturally can’t support a client that large

Diagnostic questions:

• If you experienced overnight 10x growth in your customer base, could your app keep up?
• Do you find yourself worrying that you’re going to land a customer that’s TOO big?
• Is your current architecture impacting customers and throttling your growth?
• Is your app particularly intensive in terms of storage, computation, concurrency, or other specific factors?
• Are you ready to scale well under your wildest best-case scenarios?

Solution:

Estimate your realistic, best-case, and amazing-case scaling needs for the next 5 years. Draft the best modern architecture to meet your needs. Now that you know where you are going, you need to figure out how to get there. Estimate how long it will take to build the new architecture, taking into account you’ll still be maintaining the existing infrastructure in the interim, and most importantly make a well thought through migration plan. The best migration plans include running both systems side-by-side for a substantial period.

ROI goals:

• Accelerate business growth by uncorking your bottleneck
• Easy, effortless, stress-free scaling
• Support larger customers

3. New Opportunities for Fueling Growth

You see a major opportunity to serve the emerging needs of existing customers or to pivot to a new market altogether. Your current app gets you part of the way there, but there’s a big gap between where it is today and what you envision. Whether you need to optimize it for a different customer size, usage pattern, or feature set, rewriting the app may be more cost-effective than bolting on new functionality.

Symptoms:Diagnostic questions:Solution:

Rewrite the app with the new market in mind, prioritizing the ability to make future change easier. By decomposing the app into discrete services using containers, APIs, microservices, and other modern architecture, you’ll be positioned for greater agility going forward. You’ll also be able to reuse those components in other solutions. Make sure you engage with the target user base early and often to validate the business case and optimize the value proposition in real time.

ROI goals:

• Open new markets or revenue streams
• Support larger customers
• Increase profitability with a more valuable product or by entering a more lucrative market
• Reinvigorate your product with an agile, modern platform that can quickly respond to ever changing business requirements

4. Operational Costs

The problem is simple-you’re overspending on operations. The solution might be simple, too-like turning off unused servers or changing which cloud instances you use. More often, chronically high operational costs are a symptom of sub-optimal architecture as it relates to the underlying platform. The bill can add up quickly, dwarfing what it would have cost to refactor the app in the first place. By the time you are spending six figures or more a month, a strategic rewrite can often pay for itself in a few months.

Symptoms:Diagnostic questions:

• Which aspects of your infrastructure are driving the cost, and which are driving the value
• Have you already performed significant cost optimization and are finding that you are still paying too much?
• Estimating pre- and post-rewrite operational costs, how long before the rewrite pays for itself?
• Are you using high-cost SaaS services when it might be cheaper to build an in-house solution?

Solution:

Audit your app to understand where the majority of your spending resides. Determine if it’s best to optimize your application code, refactor your cloud infrastructure or do both. The key is often tailoring your app to the pricing structure of your cloud provider. Be sure to understand their pricing model thoroughly before proceeding with your plan. As mentioned above in the “Scaling” scenario, you’ll need a good migration strategy so you can seamlessly swap over to the new back-end without affecting customers.

ROI goals:

• Slash your cloud-bill in half, three quarters or more
• Increase profitability
• Structure your cloud usage to maintain profitability even as you scale

5. Regulatory Change

Regulatory complexity continues to grow, especially in compliance-heavy industries like healthcare and finance. In some cases, new regulations can stymie the growth trajectory of an existing application. Or, you may want to expand into a new market with stricter requirements and find that your app was not designed to meet them. Retrofitting an existing app to meet compliance requirements may not be possible, especially when those requirements are related to the technology stack itself.

Symptoms:Diagnostic questions

• What compliance issues will come into play if you expand into new markets or territories?
• What is the opportunity cost of delaying entry into those opportunities?
• What aspects of compliance are pro forma and which require demonstrating specific capabilities?
• How easily can you adapt to future regulatory changes?

Solution:

Start by understanding the regulatory requirements thoroughly. Consulate with compliance experts to verify your assumptions. Next, audit your existing solution to determine the increment of change required. This can range from updating the encryption settings on your cloud infrastructure to fully rewriting the application. If deeper change is required, take the opportunity to evaluate your product’s design as a whole and re-tune it for modern market conditions.

ROI goals:

• Protect your current markets by staying current with regulation
• Move into new markets with new regulatory capabilities
• Win more business by demonstrating your commitment to compliance rigor

The High Cost of the Status Quo

There’s almost always a way to keep something running a little longer or hack in a solution for a given business need. But it’s important to evaluate the opportunity cost of burning cash running in place versus investing in the future with a larger, more thoughtful rewrite. Technical debt eats away at the agility of an aging system and eventually comes due in the form of downtime, high costs, and missed revenue. Development gets more and more expensive and innovation slows to a crawl.

The decision to rewrite an application is rarely an easy one. It can be expensive, complex, and time-consuming. But it can also reduce costs over the long term, increase profitability, and open up new markets.

The good news is major rewrites don’t have to be the high-risk projects they are infamous for. By definition you’ve already done the hardest part of any project-fitting a product to a market need. Armed with your deep understanding of that market and a few critical risk-management strategies, a well-managed rewrite project has an excellent chance of success.

At GenUI, we specialize in modernizing complex applications in ways that deliver ongoing value and measurable business results. We can evaluate your existing situation and recommend the most cost-efficient way to achieve your business goals. If a rewrite is justified, we’ll help you get it done faster, with less risk, and with an eye toward your future. Let’s talk about getting your app up to speed.

Originally published at https://www.genui.com.

S3P’s unique, parallel S3 bucket listing algorithm can list 35,000 items/second and copy up to 9 gigabytes / second.

AWS S3 offers industry-leading scalability, data availability, security, and performance. It’s also relatively easy to work with, at least when working with one file at a time. However, once you load a bucket with terabytes of data and millions of files, doing anything over the whole bucket becomes unwieldy. Even getting statistics about file counts and file sizes becomes challenging.

We created S3P to solve two problems: large-bucket content discovery and cross-bucket copying. In our tests S3P was 15x to 100x faster than AWS CLI. We sustained item listing speeds of 35,000 items per second and copy speeds of nearly 9 gigabytes per second.

Key Takeaways

• S3P uses a massively parallel algorithm for scanning S3 buckets
• S3P scans S3 buckets up to 15x faster than aws-cli
• S3P copies S3 buckets up to 100x faster than aws-cli

How S3P Came to Be

We needed to refactor a client’s AWS infrastructure into a HIPAA compliant environment. As part of work, we had to move about 500 terabytes of data into new S3 buckets. Since we only had a month to finish the whole project. We knew right away this was going to be trouble.

After some preliminary tests with aws s3 sync we found we could get a max of about 150 megabytes/second throughput. A little back-of-the-envelope math shows it would take over 40 days to complete. Clearly this wasn't going to work.

We also had another problem. The client had dozens of buckets and no good information on the contents of those buckets. Not only did we need to transfer a large amount of data in a small amount of time, we had no idea exactly what data needed transferred and no idea what the actual file counts were. By default, S3 doesn’t tell you anything about bucket size or file count. S3 + CloudWatch tells you a little, but it isn’t very detailed.

The S3 API is structured around listing items in the bucket sequentially. Just listing 5 million files with aws s3 ls at about 1400 files per second would take an hour.

We had two problems to solve:

• List S3 items at least 10x faster to aid discovery
• Copy S3 items between buckets in the same region at least 10x if not 100x faster.

We solved the listing problem with a cleaver use of divide-and-conquer and AWS S3 SDK’s startAfter option introduced with listObjectsV2. This was the hardest part of the project. Doing efficient bisection over an unknown key distribution can be tricky, but we made it work. It’s blazing fast, performing 100 or more S3 list-object requests in parallel.

Copying S3 items was more straightforward. We just needed to fire off simultaneous calls to copy-object. We overcame copy-object’s lack of support for objects larger than 5gigabytes by shelling out to aws s3 cp for these larger files. What seemed like a cludge at first ended up being a performance win since it allowed us to leverage multiple CPU cores.

All of this aggressive parallel copying needed limits however. Without some control over the concurrency, the poor NodeJS instance ran out of memory and crashed. There is a sweet spot for parallel listing as well. Too many parallel list-object requests and performance declined. We built tools and options for controlling the concurrency of listing, copying ‘small files’ with the SDK and copying ‘large files’ with the aws-cli.

Though the current implementation of S3P tops out at about 35,000 items/second for listing and 9 gigabytes/second for copying, we believe there is room for even more performance improvements by forking NodeJS or even distributing the work across multiple machines. There is no documentation about the maximum performance S3 can provide, but whatever it is, S3P’s techniques should allow us to reach it.

If you have S3 data in the petabytes range, and you want an even faster S3P to solve your problems, let us know, we’d love to help you.

Instead of a 40 day aws-sync, with S3P we were able to copy our client’s 500 terabytes of S3 data in less than 24 hours.

Alternatives

Before building our own tool, we investigated several alternative solutions to accelerating same-region cross-bucket copying. This Stack Overflow thread nicely sums up the options:

• Fastest way to sync two Amazon S3 buckets — Stack Overflow

The two main pure-Amazon options were:

• Command Runner — Amazon EMR
• Amazon S3 Transfer Acceleration — Amazon Simple Storage Service

I thought EMR would do the trick. Throwing a bunch of parallel hardware at a problem seemed like the right idea. I followed the steps and did a test with 10 large EC2 instances. Shockingly, it still only produced about 150 megabytes/second throughput. It should have been at least 10x faster. I’m not an expert on Amazon-flavored map reduce (EMR), and the Amazon-provided “Command Runner” tooling had almost no doc. I suspect the problem was in part due to the fact that listing S3 buckets is slow. Regardless, we had a short deadline on this project and didn’t have time to dive in and debug this option. It turns out that was a very good choice, because S3 speed, in this case, doesn’t require huge hardware.

I also experimented with S3 transfer acceleration endpoints, but I was never able to significantly improve on the 150 Mb/s transfer rate. The documentation of S3 transfer acceleration focuses almost exclusively on uploading data /into/ s3, not transferring between s3 buckets. Reading between the lines, I don’t think this actually accelerates transfers between buckets within the same region. Another dead end.

The remaining suggestions basically amounted to parallelizing the copy operations within a single machine. These techniques do accelerate copying, but without solving the serial listing speed problem, they, too, have limitations.

Try Out S3P

S3P is written in JavaScript and is available as an NPM package. If you have NodeJS installed, you can run it directly from the command line:

# run s3p and get the top-level help page 
npx s3p  

# run s3p and get a summary of the contents of your bucket: 
npx s3p summarize --bucket my-bucket-name

NOTE: S3P uses the same credentials system as the AWS Cli. More on how to configure your AWS-CLI credentials: Configuring the AWS CLI — AWS Command Line Interface

S3P supports many useful operations right out of the box:

• compare: Compare two buckets and produce a summary of their differences.
• copy: Copy all files from one bucket to another bucket.
• list: List all matching files.
• list-buckets: List all your S3 buckets.
• summarize: Scan all items in one bucket and produce a summary of all the items.
• sync: Only copy files which do not exist in the target bucket.

S3P also supports powerful options such as specifying custom key re-writing functions in JavaScript, but on the command-line:

# copy an entire bucket and add .backup to the end of every file 
npx s3p copy --bucket my-from-bucket --to-bucket my-to-bucket --to-key "js:(key) => key + '.backup'"

You can also iterate over specific sub-ranges of the bucket either by specifying a prefix or a combination of start-after and stop-at keys:

# summarize all items with keys starting with "my-folder/" 
# Example match: "my-folder/2015-05-23.log" 
npx s3p summarize --bucket my-from-bucket --prefix my-folder/  

# summarize all items with keys > "2018" and <= "2019".  
# Example match: "2018-10-12-report.txt" 
npx s3p summarize --bucket my-from-bucket --start-after 2018 --stop-at 2019

S3P has an API you can call directly from JavaScript. The API gives you programatic access to all S3P’s massively parallel S3 listing capabilities.

You can read more about alternative installation options and using S3P as a library in the S3P readme.

S3 Costs

With all these massively parallel S3 operations you may be wondering how much this might drive up your S3 bill. The key thing to note is that S3 pricing is not based on speed. The cost is purely based on how much you do, not how fast you do it.

S3P uses two S3 operations: list-objects and copy-object. Currently both operations cost $.005 per 1000 requests, or 200,000 requests per $1 (US-East, June 2020).

List-objects can return up to 1000 items per request, but because we can’t assume any a priori knowledge about the distribution of keys, S3P’s divide-and-conquer algorithm must make educated guesses which aren’t perfect. S3P always requests 1000 items per list, but often requests overlap. In practice, S3P fetches a little more than 500 unique items per request overall.

Effectively, S3P costs about $.01 to scan one million items. To copy one million items costs about $5 (assuming they are all smaller than 5 gigabytes and staying within one AWS region). If each file was just under 5 gigabytes, that’s about $1 per petabyte.

In other words, though S3P uses approximately twice as many list-object calls to feed it’s massively parallel listing algorithm, that cost is small compared to the copying costs. S3P uses exactly the same number of copy requests AWS-cli or any other tool would. The only real cost danger is, since S3P is so fast, you might use it more, and that might drive up your S3 bill.

S3 pricing is based on how much you use, not how fast you use it.

Conclusion

S3P is an essential tool for working with large S3 buckets. The ability to scan buckets 15x faster and copy data 100x faster opens up all new capabilities. It allowed us to move half a petabyte of data between buckets in less than a day instead of more than a month. Perhaps even more important, it allowed us to inspect the client’s S3 buckets in detail to fully understand their data patterns and know with confidence what we needed to do to meet their requirements. Without S3P we would have been blindly doing large, slow aws-syncs, and often getting it wrong as we slowly learned about the data.

S3P’s key innovation is the ability to divide-and-conquer any arbitrary S3 bucket regardless of what keys were used (as long as they only use ASCII characters). This allowed us to break free of the standard, slow linear-list-object algorithm and open up a world of unbounded parallel possibilities.

• source code: github.com/generalui/s3p
• published: npmjs.com/package/s3p

Originally published at https://www.genui.com.

Modular Redux — An Example of Applied Modular-Design for Mastering Scalable, Shared State in React

Modular Design Part 2 — A step-by-step example of applying the 5 Essential Element of Modular Design to existing code in 3 levels of difficulty.
(Modular Design Part 1 — The 5 Essential Elements of Modular Design)

Summary: Applying Modular Design, using Redux as an example, the resulting “Modular Redux” is a new design pattern, not a new dependency. It only uses standard Redux and React. It takes less than half the code compared to traditional Redux/Redux-Toolkit and decreases complexity by almost 3x. Modular Redux is an easy, powerful and scalable way to manage shared state in React applications. This is largely due to the power of Modular Design.

The day of massive, single-page-applications is upon us. Websites are looking more and more like applications and less like collections of pages. The median JavaScript size for the most popular 20 websites of 2019 is 3.5 megabytes. All but two had at least 1 megabyte of JavaScript.

Single page apps require a new way of thinking about web design. Instead of storing client session state on the server, more and more is stored in the client. State is one of the trickiest things to manage in a growing code-base. It’s particularly tricky when the state is shared across multiple components in your client-side application. Thankfully there are tools that can help. Shared-state libraries help with understanding and debugging the state across multiple components.

In this article I’m going to focus on Redux, one of the most popular solutions to managing shared client side state. Redux really shines when it comes to understanding and debugging shared state:

• Redux’s serializable, immutable, shared state is easy to inspect, persist, and restore
• Redux’s serializable, atomic, dispatched updates are easy to inspect, record and playback

Redux is the de facto standard for React shared-state, and there’s good reason. With centralized serializable state, serializable actions, atomic updates in the form of reducers, not to mention great middleware, there’s a lot to recommend Redux. State predictability and debuggability is the name of the game, and Redux is a best-of-class solution for shared state management… almost.

Redux is a pain to use and doesn’t scale. Even on the smallest projects it takes an excessive amount of ‘boilerplate’ code to get up and running. On large projects, it only gets worse. Redux’s complexity scales poorly with project size. Except… it’s not actually Redux that’s the problem. It’s the way it is used. Both the traditional Redux and newer redux-toolkit design patterns are inherently complex and don’t scale.

Thankfully, Redux can be used in a way that is both easy and scales. We can have our cake and eat it too. It’s possible to have all the wonderful properties of Redux and do it with svelte, clean, scalable code. We just need a change of design pattern.

Redux is a best-of-class solution for shared state management

The current state-of-the-art redux-toolkit was a good start, but in this article I’ll show how to take your Redux code to the next level. I will show how to write Redux code that not only scales to very large applications but also how to reduce your Redux-related code by half. Best of all, I’m going to do it with no new dependencies. I’m only going to use plain React and Redux.

The key is modular design.

Outline

This article is organized in the following sections:

• The 5 Essential Elements of Modular Software Design
• What is Modular Redux?
• To-Do-List Tutorial Using Modular Redux
• Comparing Modular Redux with Traditional Redux
• Comparing Modular Redux with React Toolkit
• Streamlining Modular Redux with Hooks-for-Redux (H4R)
• Modular Redux Scales — It’s also Just Easier
• Where to Start with Modular Redux

The 5 Essential Elements of Modular Software Design

Modular Design Part 1 — Concepts

This article is a follow-up to my previous article on the elements of modular design. I’ll be referring back to it from time to time. I recommend reading it if you want to learn more about the principles of scalable, modular design:

The 5 Essential Elements of Modular Software Design

Where the previous article applied to software engineering in general, this article is all about the nitty-gritty of real-world React-Redux applications. My goal here is to show how to write better, more scalable Redux code, and also to use Redux as a real-world case-study of the benefits and techniques of modular design. I’ll compare three different real-world applications of increasing complexity showing for each in turn how Modular Redux is vastly superior to the status quo.

What is Modular Redux?

Modular Redux is the result of consistently applying good, modular design to applications using Redux. Modular Redux is a design pattern. It’s a re-usable solution to using Redux in real world applications. My focus is on React applications, but Modular Redux can apply to any platform.

From the point of view of modular design, there were four main opportunities for improving Redux design patterns:

• Encapsulate Redux modules with respect to each other: Reducers typically have access to all redux state and all dispatched actions. This means every part of the application that touches Redux is potentially dependent on any other part. This is why global variables are so bad, and why Redux, when used improperly, can get out of control. Redux is a global state, but it doesn’t have to be used that way. We can modularize Redux slices so each slice only has access to its own sub-state and actions.

Redux is a global state, but it doesn’t have to be used that way.

• Consolidate all code related to one Redux slice into one module: All reducers, action-types, subscriptions, getters and dispatchers can be consolidated for each Redux slice resulting in code reduction, stronger encapsulation and greatly reduced inter-dependencies.
• Isolate Redux from components: Components should not be directly dependent on Redux. If we need to refactor how we manage shared state, we shouldn’t have to rewrite every component that uses that shared state. Instead, we’ll implement the simplest possible API for each Redux-module, taking care not to expose the fact that we are using Redux as the underlying implementation.
• Minimize component shared-state dependencies: Often I see people adding Redux dependencies to a parent component only to do nothing with them other than pass them to children components via props. This can make sense in scenarios where the sub-component is reused with different state-sources. However, it’s overkill if the sub-component only accesses one slice of Redux state. In general, if a component doesn’t directly use or update shared state, it shouldn’t be dependent on it. (Build your components with ZEN — zero extra nuts.)

If a component doesn’t directly use or update shared state, it shouldn’t be dependent on it.

To see how apply modular design principles to Redux, and get the most out of the opportunities listed above, let’s do a tutorial.

To-Do-List Tutorial Using Modular Redux

I’ll use the classic ‘to-do’ app for this tutorial. First I’ll go through the source code and explain how each part works. In the next section I’ll compare and contrast Modular-Redux with the same app implemented using the traditional Redux design pattern. I’m assuming familiarity with React and React Hooks.

I’ve included all the JavaScript code here, but you can also run and explore the code in your browser on CodeSandbox:

• https://codesandbox.io/s/modular-redux-tutorial-t1mkz

To-Do-List Tutorial Files

This example consists of 5 files. These are the app’s modules. I’ll explain them one at a time:

• components/ToDo.js
• components/ToDoItem.js
• redux/list.js
• redux/store.js
• index.js

index.js

The root file of the app is about as simple as it can be. We are importing our root component, ToDo, and rendering it as a child of the ‘root’ element. Notice there is no Provider nor any other Redux-related dependencies. There is no need for index.js to be dependent on Redux, so it shouldn’t be. This is one of the essential elements of modular-design: minimize inter-dependencies.

ToDo.js

The main component shows a list of ToDo items returned from useList. It also presents a simple HTML form for adding new items via addItem. The call to useState and the createNewToDoItem function are used to capture the current value in the DOM’s input field.

Experienced Redux developers will notice an important difference here compared to other Redux design patterns. We are importing simple methods directly from our redux/list module. These are simple functions. There is no manual dispatching, action-creating or other complications. This component has no dependencies on Redux at all. It only accesses an API which provides exactly what the component needs to function and nothing more.

The redux/list module’s API is the simplest solution possible given the needs of the application:

• useList returns an array of items
• addItem takes a single item as input, an object with a single ‘text’ property

ToDoItem.js

ToDoItem is the only other component in this simple app. It, too, imports exactly what it needs from redux/list and nothing more: deleteItem. You might notice that we didn’t pass deleteItem into ToDoItem from ToDo as is common in other Redux patterns. Doing that makes two components dependent on deleteItem instead of only one.

redux/list.js

This module is where the magic happens. It solves a well defined problem, manage the to-do list, with the simplest possible solution: an API consisting of three functions with the simplest possible inputs and outputs. Defining the essential problem and the simplest solution possible are two essential elements of modular design.

If we stubbed the list module, it might look like this:

Now I’ll go through the actual implementation from top to bottom, explaining it chunk-by-chunk:

At the top of the file we define the basic logic of the list state:

• storeKey defines the name of the Redux store slice where the list will live
• getUniqueId takes a list of to-do items and returns an id not in the list
• initialState pre-populates the list with two to-do items
• reducers is a map from action-names to functions. These reducers take in the current state of the list-slice (importantly, not the whole Redux state), and a payload. Then they return a new list with the change applied.

Notice that reducers do not have access to the full Redux state, yet another essential element of modular-design: encapsulation.

Next are a few helper functions. These make it easy to interact with just our slice of the Redux state instead of the entire, global state.

• getState returns the current list
• subscribe takes a function as an input. That function gets called with the current list whenever the list changes. Subscribe also returns a function for unsubscribing.

Finally, we replace our stubs with working implementations:

• useList uses standard React hooks to re-render any component it is used in, returning the current list every time it’s called.
• addItem dispatches a Redux action to trigger the addItem reducer. Notice that the dispatcher only takes an item as input; that’s all it needs to know to do its job.
• deleteItem also dispatches a Redux action, this time to trigger the deleteItem reducer.

Notice that addItem and deleteItem, the dispatchers, only take exactly what they need as inputs — the item to add and the item to delete respectively. There is no need to pass in action-names nor construct action data-structures.

Last, we need to add our reducers to the redux store. This may seem a little backwards compared to the traditional Redux pattern. Usually you define all the reducers and then call createStore, passing them in. In order to minimize dependencies, we moved everything related to this Redux slice into one file. Therefor we need to have access to the store in the same file where we define the reducers. This is why in Modular-Redux, the Redux-modules register themselves with the store rather than the other way around.

As it turns out, this helps us not only further reduce dependencies by eliminating the need for the traditional Redux reducers file (see below), the self-registration method of defining Redux modules also allows us to do cool things like lazy loading and hot reloading.

Note: If an application needs control over which store a modular Redux slice is registered with, simply wrap the entire contents of this file in a function that takes the store as an input. Then invoke the function with the store you want to bind it to.

store.js

The last file of the to-do list example defines the Redux store. It creates a basic store with an injectReducer method. This pattern is described in detail on the Redux.js website:

Tutorial Wrap-Up

If you haven’t already, try the working tutorial example and explore the code: (same as linked above) https://codesandbox.io/s/modular-redux-tutorial-t1mkz

This example seems almost trivially simple. The key is consistently applying the 5 elements of modular design to each file: define a single, narrow and focused problem, create a well-defined, complete and minimal solution, ensure watertight encapsulation, implement a correct, performant and minimal foundation, and do it with minimal inter-dependencies.

Comparing Modular Redux with Traditional Redux

Here’s why modular Redux really matters: it can reduce the complexity of shared state by two to three times. Let’s compare two equivalent implementations. One is basically what we just finished in the tutorial — with styling added. The other is functionally identical but written in the traditional redux way:

• modular Redux:
  https://github.com/shanebdavis/modular-redux-todo-2020
  https://codesandbox.io/s/modular-redux-tutorial-t1mkz
• traditional Redux:
  https://github.com/sunil-sandhu/redux-hooks-todo-2019
  https://codesandbox.io/s/github/sunil-sandhu/redux-hooks-todo-2019

Modular redux slashes the dependencies between modules. The traditional implementation has 19 dependencies between modules:

Note: CSS, react, and react-dom imports are omitted since they are identical in both projects.

The solid arrows show explicit dependencies in the form of JavaScript import statements. The dashed arrows show implicit dependencies. A module is implicitly dependent if it needs updating when the implicitly-dependent-on module is updated. Let me explain the implicit dependencies one at a time:

• appActions depends on redux because it implements part of the redux dispatch API.
• appReducer similarly depends on Redux since it implements the redux-specific reducer API.
• further, App depends on Redux since it explicitly uses the Redux dispatch API.
  In all three of the above cases, if you swap out redux, you’d have to complete rewrite or replace these files.
• appActions depends on appReducer since every action in appActions must exactly parallel the reducers defined in appReducer. If you add or change your reducers, you’ll also have to update appActions.
• ToDoItem and ToDo both implicitly depend on appActions since they implicitly use appActions. If you change the signature of any of those actions, ToDoItem or ToDo will need to be updated.

Now, compare that with the Modular-Redux solution’s mere 7 dependencies:

There are zero indirect dependencies. Each module can be fully swapped out without needing to update other modules. The relationships are clear and easy to understand. Minimized dependencies maximize code scalability.

Modular Redux dramatically cuts down on the number of files and lines of code and has 63% less inter-dependencies.

• 2.7x improvement: 7 dependencies vs 19
• 1.8x improvement: 5 files vs 9
• 1.6x improvement: 104 lines of code vs 170

Comparing Modular Redux and React Toolkit

In the tutorial above I compared Modular Redux with the standard Redux pattern. I did this because it is the Redux pattern most people are familiar with. This is somewhat unfair, however, as the official Redux recommendation is to use the newer Redux Toolkit. Redux Toolkit does make good progress over the traditional approach, but let’s see how Modular Redux can bring things to the next level. The Redux Toolkit website provides a basic tutorial, an intermediate tutorial and an advanced tutorial. I’m going to skip over the basic tutorial and dive directly into a detailed comparison using the intermediate and advanced tutorials.

Intermediate Tutorial: To-Do List with Filter

This version of the To-Do list uses two Redux slices. In addition to adding and listing to-do items, you can also filter the item list.

First, let’s start with the Redux-Toolkit-based implementation. I recommend opening the CodeSandbox to get a frame of reference:

• interactive Redux Toolkit solution (codesandbox)
• Redux Toolkit solution source (github)
• Redux Toolkit Intermediate Tutorial Documentation

Let’s jump right into the Redux Toolkit dependency diagram:

There’s quite a bit going on compared to our first tutorial. There are three external dependencies on redux-related libraries instead of one: redux-toolkit, react-redux and redux itself. While there are a lot of interdependencies, there were only two implicit ones: Link and TodoList use the filterSlice and todoSlice APIs indirectly, respectively.

Redux-Toolkit does a good job of eliminating the need for multiple files to define each redux-slice. There is just one file for the to-do list and one file for the filter. However, Redux-related dependencies are still littered throughout the components.

Now let’s look at the Modular-Redux solution for the same To-Do-List-with-Filter:

• interactive Modular Redux solution (codesandbox)
• Modular Redux solution source (github)

The Modular Redux dependency diagram:

The above diagram shows what a dramatic improvement modular design can bring to a project. We cut the number of dependencies from 27 down to just 13.

You may notice there are less components in this version. As I was rewriting the code, simplifying it, additional simplifications suggested themselves. This is why it is so important to always strive to make things as simple as possible. Complexity breeds complexity. If a system is overly complex, it will tend to grow even more complex. The complexity itself necessitates ever more complexity to manage it.

Complexity breeds complexity.

When should a module not be a module? The main sign a module might not be necessary is when it has only one other module dependent on it. In that case, one needs to determine if the module is substantively different from the module that depends on it, and it the code is simpler or more complex with the sub-module.

In this tutorial, FilterLink and Link were intimately tied up with Footer and the filter options and weren’t being used anywhere else. I was able eliminate 38 lines of code, two files and only add 3 lines of code to Footer. If moving code into modules causes a 12x increase in code-size, it’s probably not the best solution.

I also added one module. There was some redundant code in the filter and todo modules. Since the code was used in two different places, I was able to create a new module, modularRedux, simplify both the original modules and reduce overall code-size.

• 2.1x improvement: 13 dependencies vs 27
• 2.0x improvement: 193 lines of code vs 393
• 30% improvement: 10 files vs 13
• 30% improvement: 134k minimized JavaScript build vs 170k

Advanced Tutorial: Github Issue Browser

The final Redux Toolkit tutorial is a styled Github issue browser. It adds asynchronous requests, TypeScript, limited testing, hot-reloading, styling and has four different redux-slices: issuesDisplay, repoDetails, issues and comments.

You can learn about the Redux Toolkit implementation here:

• interactive Redux-Toolkit Advanced Tutorial (codesandbox)
• Redux-Toolkit Advanced Tutorial source (github)
• Redux-Toolkit Advanced Tutorial Documentation

The module-dependency diagram is significantly more complex than anything we’ve looked at so far:

The Redux Toolkit Advanced Tutorial implementation had a ton more implicit dependencies. For example, showIssueComments is imported in App, but it is passed through IssuesListPage through IssuesList before it is finally actually used in IssuesListItem. It also contains a complicated set of inter-dependencies between the Redux modules themselves.

Now let’s look a the Modular Redux implementation:

As you might expect, Modular Redux dramatically decreases the number of inter-dependencies. The total dependencies decreased from 63 to 34, but that only tells part of the story. This time the React modules were already optimal, so the component dependency structure is identical in both implementations. All 28 eliminated dependencies came from improving how Redux is used. The Redux-related dependencies decreased by more than half from 48 to just 19.

The right-hand side is still pretty busy, but even without knowing the contents of the files, the diagram helps illuminate what’s going on in the application. For example comments, issues, and repoDetails all update when issuesDisplay changes, and that’s clearly reflected in the dependency diagram.

You’ll notice there is no dependency on redux-thunk. Each of the four Redux modules manage their own asynchronous loading internally. Subscriptions and promises sufficiently and elegantly manage the asynchronous requests. There is no need for the added complexity of thunks.

I also added TypeScript support for modularRedux. Typing works very well with the Modular Redux pattern. You’ll notice there is very little explicit typing in any of the Redux modules. Eliminating the pass-through, indirect dependencies in the components also cleaned up huge swaths of unnecessary typing code.

In the end, the most advanced, most challenging, and most up-to-date example of Redux Toolkit was outdone by using plain JavaScript and good software engineering in the form of modular design:

• 2.4x improvement: 20 redux-related dependencies, vs 48
• 1.8x improvement: 35 overall dependencies, vs 63
• 1.7x improvement: 685 lines of code, vs 1169
• 10% improvement: 244k minimized JavaScript build, vs 271k

Redux Toolkit’s Missed Modular Design Opportunities

H4R address all four of the Redux modularization opportunities I discussed above, but Redux Toolkit only addresses one and a half out of the four:

1. Redux Toolkit addressed: Encapsulating slices with respect to each other. Each reducer only handles the state of the slice, not the entire state. Reducer-routing is handled automatically, so a particular slice’s reducers aren’t allowed unfettered access to all actions.
2. Redux Toolkit partially addressed: Redux Toolkit does consolidate reducers, action-types and action-creators into one file, but it still leaves dispatching, subscribing and getting the current state to the individual components.
3. Redux Toolkit missed: Redux Toolkit still fully mingles Redux-specific code throughout every component in the project. This is such a disaster that, when rewriting the Advanced Example with Modular-Redux, I ended up rewriting it from scratch rather than refactoring the existing code. This is why it’s so important for modules to not expose their own dependencies. Redux-Toolkit’s slice-modules expose their Redux dependency making it near impossible to refactor on anything but trivial projects.
4. Redux Toolkit missed: Both the intermediate and advanced Redux Toolkit examples continue to repeat the broken pattern of importing dependencies in a parent component only to pass them to a child component without ever using them. This isn’t a direct failure of Redux Toolkit, but by repeating this error in the tutorials they are propagating bad design to the broader Redux community. The Advanced tutorial is especially guilty as the showIssueComments function is actually passed not to a child component, but all the way through to a great-grand-child component before it is actually used creating four dependencies when one would have sufficed.

While Redux-Toolkit is better than the traditional Redux design pattern, it comes at the expense of adding its own complexities. It adds new concepts but doesn’t successfully abstract away Redux base concepts. The result is you have to deeply understand both to successfully use the toolkit.

Streamlining Modular Redux with Hooks-for-Redux (H4R)

I said I was going to show how to use Modular Redux with no extra dependencies, and I have, but you’ll find a certain amount of reusable code across all three of the examples above. That code should go in its own NPM module. That NPM is hooks-for-redux.

If you decide to use the Modular Redux pattern, I recommend using hooks-for-redux. In some of the diagrams above there is a light-grey rectangle with ‘H4R’ in the lower right-hand corner. H4R is a drop-in replacement for those rectangles. It is a tiny library (currently just 90 lines of code). It implements the common parts of the Modular Redux design pattern in a well-tested and fully TypeScripted way.

I’ve implemented each of the tutorials above using H4R. You can view them here:

• To-Do Tutorial with Modular Redux and H4R:
  (github) (codesandbox)
• RTK Intermediate Tutorial with Modular Redux and H4R:
  (github) (codesandbox)
• RTK Advanced Tutorial with Modular Redux and H4R:
  (github) (codesandbox)

Read more about hooks for Redux:

• How I Eliminated Redux Boilerplate with Hooks-for-Redux (H4R)

Modular Redux Scales — It’s also Just Easier

Redux is a very simple idea. It lets you define shared state, methods for atomic updates and subscriptions to changes. Where Redux really shines is the wealth of middleware available you can attach to a redux store to persist, restore, playback and inspect your state. Because of all those reasons, Redux should scale very well, but it typically doesn’t. The problem isn’t Redux itself. The problem is the recommended ways of using Redux introduces a ton of unnecessary complexity.

The Modular Redux design pattern dramatically simplifies using Redux. While making things easier is great, Modular Redux shines most as you scale your application. Modular Redux kept the complexity under control for each of the three applications presented above even as they increased in size. This comes from the systematic application of the essential elements of modular software design. Modular Design helps ensure each module solves a focused problem, with a well-defined, complete and minimal solution, using watertight encapsulation, supported by a performant, correct and minimal foundational implementation, and most of all, with minimal inter-modular connections.

With Modular Design, Redux becomes a Best-of-Class solution to React shared state. You can have Redux’s centralized serializable state, serializable actions, atomic updates, great middleware and you can do it with svelte, clean, scalable, modular code.

With Modular Design, Redux becomes a Best-of-Class solution to React shared state

Where to Start with Modular Redux

To get started, I recommend either taking one of the examples above, or you can dive into the Hooks-for-Redux Tutorial which is a little more detailed and will guide you to towards the best way to use Modular Redux in your application.

Further Reading

• Part 1: The 5 Essential Elements of Modular Software Design
• More about H4R — recommended library for building modular-redux applications:
  How I Eliminated Redux Boilerplate with Hooks-for-Redux (H4R)

Originally published at https://www.genui.com.

The 5 Essential Elements of Modular Software Design (PIE-CI)

Why do we care about modular design? Let me tell you a story about my friend Mike Kelly. In 2016 he took a side job to build a simple course platform. Since it was a side job, and Mike was bored, he offered to do this job at a deeply discounted rate if he could retain rights to the source code. It was a brilliant move. That client had reach, and other people started asking about his work. “This is a really nice platform. It’s really easy to use. It’s fast, and it works well. What is it?”

Mike decided to test the waters. His plan was to package up the product so it would be easier to custom-install on each clients’ hardware. He quickly realized it was going to be a nightmare to maintain the software in each client’s environment. Instead, he moved towards a software-as-a-service solution, hosting each client’s course in one place for a monthly fee. Thus MemberVault.co was born.

MemberVault was a success. More and more people signed up. Mike quit his day job and, together with his wife, started working full time on marketing, sales, customer support, bugs and new features. There was just one problem. In the rush to get things out the door, he didn’t have time to change his original design: each client required their own database. Each new customer, free or paying, added another database to his MySQL server.

Fast forward to 2019 and MemberVault’s MySQL server had thousands of databases. MySQL is not designed to scale this way, and performance and memory problems were starting to creep up. Mike had to solve this problem soon. He was proverbially tied to the train-tracks and the locomotive was barreling down on him.

This is where I came into the story. I helped Mike think through the possibilities and we came up with a simple solution. We decided it was best to stick with MySQL, but refactor the code to use a centralized database. It would need a new schema with an added clientId field for most tables. The question was how to refactor the existing code-base, migrate old customers, minimize downtime, minimize development time, and avoid having to maintain two codebases for the different database schemas during the transition.

The solution ended up being astonishingly simple (the best solutions always are). Mike added the clientId field to the existing databases even though they didn’t need it. He could refactor the code in place, test it and deploy it incrementally before he introduced the new, centralized database. Once everything was working on the existing databases, he could add a small tweak to his existing database routing logic to route all new clients to the central database, which would have the exact same schema. He could worry about actually migrating the old clients later, at his leisure, and still only have to maintain one code-base.

The problem was Mike’s code was a mess. It wasn’t modular. It had grown organically as he struggled to keep up with all the needs of a successful, growing SAAS company. For example, instead of being wrapped up in a single, well-designed module, the code for adding users was duplicated and spread across multiple code-bases: member-user-signup, member-admin, membervault-admin, cron jobs, the API and other external integrations.

Modular design could have reduced the amount of code Mike needed to update by a factor of 5 and dramatically reduced the complexity of the overall refactor. Instead of taking months, the refactor could have been done in a couple weeks. That’s the power of modular design. Modular design allows code to remain agile in the face of ever-changing requirements.

Modular design allows code to remain agile in the face of ever-changing requirements.

Modular Design

Modular programming is a software design technique that emphasizes separating the functionality of a program into independent, interchangeable modules, such that each contains everything necessary to execute only one aspect of the desired functionality — wikipedia

As with all software engineering, the ultimate goal of modular design is to maximize developer productivity. We want to create maximum software value for minimum development cost. Modular design amplifies developer productivity by managing complexity.

Complexity is the primary killer of developer productivity. Software complexity can scale exponentially with size. Thankfully, modular design gives us complexity-fighting superpowers. It allows us to decompose big, seemingly complex systems into small, manageable parts. The strength of a project’s modular design will ultimately determine how large it can scale. Without modular design, it would be impossible to manage the complexity of all but the smallest, most incremental projects.

Modular design gives us complexity-fighting superpowers.

With good modular design, there is no limit to how high and how far we can go with software. Modules give us the super power of abstracting arbitrary complexity behind clean, simple interfaces. Modular design can, when wielded by masters of the craft, transform a project where complexity scales exponentially with size to a project where complexity scales sub-linearly. It is even possible for a project to get easier to add functionality as the project grows.

The 5 essential Elements of Modular Design

The goal of modular design is to manage complexity. To do this we must simultaneously minimize the complexity of each individual module and minimize the complexity of the overall system. Practically, the goal is to make each module as easy as possible to design, implement, test, deploy, upgrade and maintain. The 5 elements of modular design are essential to achieving this goal.

Make each module as easy as possible to design, implement, test, deploy, upgrade and maintain.

1. purpose: A module is an abstraction with purpose. Its purpose should be crystal-clear. It should have a single, exclusive responsibility. A module’s responsibility should be narrow and focused, and no two modules’ purpose should overlap.
   (single responsibility principle)
2. interface: A module’s interface should be easy to use, easy to understand and easy to ensure correctness. It should offer all this without needing to understand any of its implementation details.
   To achieve this, a module’s API should be well-defined and documented. The API should be complete and minimal. It should have exactly what is needed and nothing more. Last, it should be hard to misuse. The easiest way to use a module should also be the correct way.
   (Arnaud’s three principles of excellent API design)
3. encapsulation: A module’s implementation is private. Modules should expose as little as possible. They should not expose their functional structure, data-structure nor their own dependencies. Any implementation detail of a module should be changeable without affecting a single client.
   This is perhaps the most important element of modular design. The other four elements could all be expressed in terms of maximizing the isolation as much as possible of the internal implementation from the outside world. As an abstraction, each module should be as watertight as possible. Leaks become accidental, hidden parts of the public API. Without strong encapsulation, you end up with implicit dependencies which can be disastrous to scaling projects.
   (Joel’s law of leaky abstractions)
4. connection: In order to truly make our software scalable, it’s not enough to just minimize the complexity of an individual module. We must also minimize the complexity of the entire ecosystem in which the module functions. The dependencies between modules define a network which adds its own complexity. A well designed modular system minimizes the dependencies between all modules. To minimize the connections between modules, minimize each module’s dependencies.
5. implementation: In order to ensure the module is easy to use, it must work well. A module with the best-designed purpose, interface and encapsulation will still fail without good implementation. A module’s implementation should be correct, performant, tested and minimal. The worse the implementation, the leakier the abstraction.

Here’s an acronym I use to help me remember the 5 elements: PIE-CI. I pronounce it “pie-see-eye.”

What Shouldn’t Be a Module

As with any technique, modular design can go too far. If you factor a 100,000 line program into 20,000, 5-line modules, you may be creating just as much mess and headache as putting all 100,000 lines in one file.

Even if a chunk of code meets the five elements of modular design, it may still be overkill to factor it into its own module. There are two things to be wary of if you feel like you are making too many modules:

• mono-dependency: If a module is only used once in another module, a parent, it may not make sense to make it its own module. It’s ok, even good, to have modules which have only one dependency. These are often called sub-modules. As long as they solve a distinct sub-problem for their parent, as well as adhere to the other essential elements of good, modular design, they can be essential to managing the complexity of that parent.
  On the other hand, if the code is used two or more times, especially in different modules, it should almost certainly be a module in order to keep things DRY.
• mostly-ceremony: If you have a module that you think may be unnecessary, ask how much code it would save to merge it into its parent. If it is a mono-dependent module, you will almost always save a little code, but that alone isn’t a good reason to de-modularize. However, if you save something closer to 90% of the module’s code size just by folding it into the parent, that’s probably a code-smell you don’t want to ignore.

A module with two or more dependencies should almost certainly be a module in order to keep things DRY.

Subjective Reality of Modularization

The goals of modularization is always to decrease complexity and increase clarity. These are ultimately subjective judgements. Everything I’ve discussed in this article is secondary to your team’s well considered judgement for what works on your particular project.

Learn the rules well so you know how to break them properly — Dalai Lama

Benefits of Modularization

The primary benefit of modularization is mastering the art of managing software complexity. When done well, good modular design leads to some very powerful, practical benefits:

1. understandability: A well-modularized system is much easier to reason about, think about and communicate to others.
2. improvability: Strongly encapsulated modules maximizes your ability to fix or improve individual module implementations without needing to update any other, dependent modules.
3. refactorability: The less inter-dependencies in a project, the easier it is to make large changes across multiple modules.
4. reusability: Modules with the best-conceived purposes are fully reusable. Whenever you have the same problem again, you can simply reuse the old solution.
5. testability: Modules with good APIs and minimal inter-dependencies are easier to test. Well-designed APIs can be easily unit-tested. Modules with minimal inter-dependencies can be tested without the need for mocks or more difficult integration-testing. Modules allow you to write tests once, ensure correctness, then reuse the module without the need for further testing.
6. scalability: All of which adds up to the most important benefit of modules: They let our applications scale. It’s impossible to build large applications without good modularization. Without modules, complexity will destroy your productivity.

Without modules, complexity will destroy your productivity.

Higher, Further, Faster

Modules allow us to climb higher, go further and build faster. Modules are the building blocks of the virtual landscape. We’d be nowhere without modules. We can only build the amazing software possible today because of the amazing foundation of existing modules we can draw upon. A programming language is a module. Operating systems are collections of modules. The most successful languages have vast module libraries: module counts by language. Some modules have endured decades of use and continue to be key to our collective success. That is the power of good modular design.

Modules allow us to climb higher, go further and build faster.

So get out there! Take your modular design skills to the next level and build something awesome!

Higher, further, faster, baby! — Captain Marvel

Further Reading

In part 2 I apply the 5 elements of modular design to a real world problem — how to write scalable client-side applications with complex state:

• Part 2: Modular Redux — A Design Pattern for Mastering Scalable, Shared State

2021 Update: I could never remember the names I came up with for the five elements. That isn’t a good sign! So, I adjusted their names and now the five are much easier to remember with the acronym: PIE-C-I. Do you agree? — SBD

Originally published at https://www.genui.com.

Debuggers are Broken — Become a console.log-Samurai

This is a re-post of my reply to Please Stop Using console.log — It’s Broken by Luc Claustres.You can read his thoughtful response here.

Debuggers are useful, but logging has capabilities that debuggers can’t compete with: universality, temporality and turing-completeness. My title is entirely a tongue-in-cheek response to the source article maligning console.log. Both debuggers and logging are useful tools for solving problems. Personally though, over 30 years of coding, I’ve found logging to be dramatically more effective for finding tricky bugs. Here’s why:

Debuggers ARE Always Breaking; Logging is Universal

Across languages, debuggers are notoriously unreliable, particularly for compiled code. When they work, they can be very useful, but often they refuse to connect to your application, refuse to show you the value of an object, or dump you into an assembly-code listing. In interpreted languages this tends to be less of a problem, but if you need to do remote debugging, say of a web-page on a physical device, debuggers still fail to connect all the time.

On the other hand, logging is available in every language and every environment, and (usually) just works.

Too Much Information

Debuggers attempt to show you everything about the current state of the running application. It can be overwhelming and distracting. While it’s true when you log from your inner-loops you’ll get overwhelmed with output, log statements have another advantage. They are written in a turing-complete programming language. That means you can filter your logging to show exactly, and only what you need to see.

• Try wrapping your log statements in conditionals to only log information close to your error.
• One dumb, but effective technique is to count how many times the log statement is called and only output up to a certain count, or starting with a certain count… etc.
• Another technique is to put a unique string in your data — a tracer — and only log when that particular piece of data is being processed.

Too Little Information

Debuggers attempt to show you everything about the current state of the running application. In reality debuggers show less than 0.01% of your application’s current state. At most, they only show 1–2 kilobytes of information. Debuggers typically show the current local variables, the current call-stack, and let you inspect only one object at a time. Your application probably has megabytes if not gigabytes of state.

Debuggers show less than 0.01% of your application’s current state.

Debuggers are relatively fixed in their output. Logging, however, is under your full, unbounded control. Logging can perform arbitrary computation, summarizing and distilling avery aspect of the running state into exactly what you need to know to inspect your code.

• Leverage your language to extract and log just what you need to know.
• One trick I love in JavaScript-ES6 is, with the help of object-structuring, you can quickly log a labeled-list of the locals:
  console.log({data, i, j, sesson})

No Temporal Information

This is where debuggers really fall short. They only show you a snapshot of your running code. Runtime state is dynamic — it changes over time. Debugging with a debugger is like solving a whodunnit movie from a single frame.

“Wait!” you say, “You can step through your code with debuggers!” Of course you can. You can also watch a movie one frame at a time. It should only take about a week to find out if Professor Plum killed Mrs Scarlett in the ballroom with the candlestick.

Logging gives you temporal context. A well crafted log statement can summarize the plot for you. You can see the key transformations of your running state over the lifetime of your application.

Debugging with a debugger is like solving a whodunnit movie from a single frame.

• If you are trying to scan a lot of log-output for patterns, try column-aligning your output. The human eye is very good at finding anomalies in otherwise regular data:

• Change the color of your output to highlight important data or events.

Turing-Complete vs Fixed-Function

The key difference between logging vs using a debugger is the difference between a full, turing-complete language and a fixed-function calculator. Logging has no theoretical limitations. Debuggers are limited to only the functions their designers put into them.

In 30 years, I’ve found time and again that I solve my bugs faster with well-crafted logging. In part it’s because I’ve invested time to get good at logging. I’ve spent comparatively less time mastering debuggers, though I am fairly proficient at them. My point is you can get very good at logging. It is an exceptionally powerful tool for debugging.

Debuggers have fundamental limitations that logging does not:

• Logging is simple
• It works everywhere
• and most of all, you can use the full power of your programming language to inspect your state

Enhanced Logging

The techniques above work universally in every programming environment I’ve ever seen. If you can code it, you can log it. However, there are tools that can make logging even better. Several of them come built into your browser:

• collapsable logged-objects — web browsers let you collapse and expand logged objects to let you explore your log — getting the best of both worlds, debugger and logger
• inspection framework — languages like Ruby come with a concept of inspecting objects — objects have built-in methods to output their state cleanly and succinctly for loggers. I embrace this heavily whenever I can. If the language doesn’t have a predefined system of inspection, I’ll add my own inspect methods to every object I define.
  Pre-defining user-friendly views of objects throughout your code amplifies the effectiveness of logging. It means each time you add a new log statement, you get awesome output for free, even if you don’t know the types of the objects you are logging.
• dynamic filtering and search — web browsers also let you filter and search your log. I’ll often turn on all my logging, but also turn on a filter for just what I’m looking for. Once I find it, I might turn off filtering to see the related logs near-by.

Downsides of Logging

I have my own list of logging downsides — at least in contemporary languages:

• They clutter your code. There’s no easy way to add them, turn them on and off, without leaving litter all over. As a result I typically add them to debug, but then I remove them later to clean up. However, I sometimes have a similar bug later and I end up re-adding the same log statements. Some way to hide, remove and/or restore log statements at the source-level would be welcome.
• They can compound bugs. Your code is different when you add logs. Sometimes that affects the runtime sufficiently to make your bug disappear, or for new, spurious bugs to appear. Debuggers get a big win here. That said, as you improve your log-fu, you can master this shortcoming.
• They impact performance (and so do debuggers). Sometimes adding a logger can slow down the app so much it isn’t practical to run your tests. However, since you have turing-completeness at your side, you can solve this problem through careful logging.

No One-Size-Fits-All Solution

Ultimately, there is no one-size-fits-all solution. Debuggers have their time and place, and so does logging. It’s all about using the right tool for the job. Logging, however, is an extremely powerful tool:

• logging is always available and rarely breaks
• logs give you an evolution-over-time view of your application, not a single-frame, point-in-time snapshot.
• but most of all, turing-completeness means total control over filtering, presentation, and summarization of your application’s state

Written at GenUI.com — software engineers, designers and strategists for bringing visions to life

Redux is great. It lets you persist, restore, log, and investigate your application’s state in a reliable, predictable way. Further, there are lots of great libraries out there like react-redux, redux-devtools and redux-logger which make working with Redux a pleasure… almost.

There’s just one thing. Redux takes an excessive amount of boilerplate to code. The standard way to use Redux is a highly manual, redundant process of dispatching, routing and reducing your state. But it is just that: standard — and that means automatable.

After working on several projects using Redux, and noticing these patterns, I started designing a tool to automate it. I’m not the first person to observe this: reduce-react-boilerplate, reducing-boilerplate-in-redux and minimize-redux-boilerplate-with-4-lines-of-code, nor the first to try to address this: reduxless, redux-actions, redux-arc and many more. However, these previous attempts each fail in one of two ways:

1. They offer a non-standard solution that doesn’t integrate well with the Redux ecosystem
2. Or they fall short of removing all the redundancy of the standard Redux pattern.

So I created hooks-for-redux. I liked the way hooks cleaned up React, so I decided to use a similar API to clean up Redux. I’ll explain how H4R works with an example:

How I Used H4R to Eliminate 52% of the Code in the React-Hooks-Todo App

I recently read Sunil Sandhu’s great post detailing how well React-Hooks cleans up how you use Redux in components. I‘m going to use his result as my starting point:

• how-i-reduced-my-redux-code-using-redux-hooks (github)
• all 177 lines of JavaScript from Sunil’s code in one file

For those who don’t want to wait, you can see the final results here:

• github.com/shanebdavis/hooks-for-redux-todo-2019
• all 85 lines of the H4R JavaScript in one file

Essential Logic of the To-Do App

Before we dive into the code it is useful to understand the essential logic of the app. An app can be no simpler than its essential logic. That gives us a target to strive for as we attempt to clean up and reduce the code size of the To-Do app.

Starting with the user-interface, the React components pseudo-code might look like this:

As you can see, there are just two components. ToDo consists primarily of a list of ToDoItems, one for each item from useList. The ToDo component also has a text-box and a button for adding new ToDo items via addItem. ToDoItems show each item’s text and a deleteItem button.

This psuedo-code is almost functional except addItem and deleteItem aren’t getting passed the correct parameters.

The rest of the application is managing the todo list itself. Here’s the pseudo-code:

At the top of the pseudo-code you can see the structure of the list and its initial state. Lines 7 and 8 describe the logic behind our two list operations: addItem and deleteItem. These are expressed as reducers — they take the current list’s state, plus an item, and return a new list with the change applied.

Last, there are three stubs for the three functions used in the components above:

• useList returns the current value of the list and re-renders the component it is used in whenever the list changes
• deleteItem deletes the item from the list
• addItem adds the item to the list

The entire, near-working essential logic of the ToDo app is just 30 lines of code. Real apps have more code to manage styling and robustness, but the goal is to get as close to this minimal form as possible.

Let’s Get Started

I’m going to take it one file at a time starting with the root index.js. I’ll show the before and after and explain what changed.

I’m using lines to measure code size. Though it’s not my preferred method, it is easiest. More on my thoughts on counting code in tokens here.

src/index.js (22% less lines)

This is the root file for the application.

index.js before (14 lines):

index.js after (11 lines):

You’ll notice two changes to src/index.js. First, there is no call to configureStore, and second, there is a new Provider which doesn’t require a store parameter. Hooks-for-redux is designed to add reducers to an existing store rather than require all reducers be defined before the store is created. This allows us to use a default store, which is sufficient for many applications. That way you don’t have to explicitly create it or bind to it throughout your application. Of course, you can override the default store as needed.

src/App.js (67% less lines)

App is the root React Component for the application.

App.js before (15 lines):

App.js after (5 lines):

Simplifying App.js is pretty straight forward. App.js shouldn’t be dependent on the List state at all. Among other problems, that unnecessary dependency causes App to re-render whenever the list changes, which is wasteful. Only ToDo needs to re-render in that case. In general, always minimize dependencies between modules.

src/components/ToDo.js (42% less lines)

ToDo is the main component for presenting the todo-list. This is the biggest JavaScript file, and there are many changes. Most are small improvements in how React is used. Some of these are subjective, but my goal is always to reduce code without sacrificing clarity.

ToDo.js before: (72 lines)

ToDo.js after: (42 lines)

React code cleanup:

• I’m using export-const, not export-default — I generally avoid export-default. I thought it would save code, which undoubtedly is the point, but I’ve found it actually tends to increase code-length and fragility. One problem with export-default is when you import a default you can assign any name you want, and that opens up the possibility of inconsistent naming in a project. However, my biggest problem with export-default is it doesn’t work with “import *” or “export *” — which are very powerful constructs for reducing code redundancy.
• I inlined a few of the shorter functions. I find that while adding new, named things can help with clarity, excessive naming can also hurt readability. In general, I try to not name things that will only ever be used once. That way you can read them in-place where they are used and know exactly what’s going on. If you do name something, it’s important for it to add clarity. Names like ‘handleInput’ tell you nothing about what the function does while significantly decreasing readability. A better name would have been ‘captureTextUpdate’.

There are more substantial changes as well:

• I imported addItem and useList. As noted in the App.js changes, they shouldn’t be passed in as props, so now they are imported directly.
• I removed generateId. It is not the ToDo component’s responsibility to create and guarantee unique ids. It’s the responsibility of the list model.
• I removed deleteItem. ToDo.js never uses deleteItem, so it shouldn’t be dependent on it.

src/components/ToDoItem.js (35% less lines)

These changes are pretty straight forward:

ToDoItem.js before: (17 lines)

ToDoItem.js after: (11 lines)

• deleteItem is imported directly rather than coming from props
• export-const (same as ToDo.js)
• props-destructuring directly in the argument list allows us to make the entire render function an expression

redux/list.js (74% less lines, 80% less files!)

Now for the real magic. H4R’s useRedux eliminates all the boilerplate needed to manage your Redux store, and it does this without sacrificing any power or compatibility. It allows you to put all your Redux logic in one file.

Note, on larger projects I recommend creating one file per data-model. In this simple app, though, there is only one data-model: list.

5 redux-related files before: (61 lines)

Just list.js after: (16 lines)

getUniqueId is basically the same as generateId originally in ToDo.js. I moved it out of the React-view and into the Redux-model where it belongs.

The rest of the file is the call to useRedux. Each call to useRedux defines a slice of named Redux state. It takes three main arguments:

1. storeKey, a string, is the property name where your state will be stored in the Redux store.
2. initialState can be anything
3. reducers is an object of named reducers
   (the names are the action names used during dispatch)

And it returns, ready to use, an array containing the following:

1. A React hook for getting the current state and re-rendering the component whenever the state changes.
2. An object of named dispatchers matching the provided reducers.

Note: There are some additional inputs and outputs not shone in this example. Refer to the README for more.

This is how hooks-for-redux works. It takes the essential information needed to define your state — name, initial state and reducers, and it returns the essential methods for interacting with that state. Internally it’s doing exactly what you’d be doing manually yourself — in a clean, efficient and most importantly, well-tested way.

Finished! How Did We Do?

The original source was 177 lines of JavaScript. With hooks-for-redux, it streamlined down to just 85. That’s 48% the size of the original line-count. We also reduced the number of files from 9 down to 5.

How did we do compared to the 30-line essential solution? If you merge the H4R version into one-file you can remove most of the import and export lines. If you further remove all the styling, it slims down to just 45 lines. Compared to the essential, not-quite-working solution, the extra 15 lines account for handling text input, initializing the React app and a few other details required to make it actually work. I don’t think you can make it much simpler:

Hooks-for-Redux and Redux

Redux is a great extensible, client-side state management platform. I want to give plenty of credit to Dan Abramov and Andrew Clark. Redux’s extensibility is what makes H4R possible.

However, Redux falls short from the application-author point of view. The standard pattern for using Redux is highly repetitive. Not only does it take more time to write the same code over and over, but it also creates more surface area for bugs to occur. The standard pattern litters dependencies all over your code and encourages mixing your Redux logic with your React components.

Almost all good software engineering comes down to one thing: Don’t Repeat Yourself. This is what hooks-for-redux does for you. It eliminates the repetition without sacrificing any power. You still define your state. You still define your reducers, but the rest is taken care of for you.

Some more of my thoughts about DRY and writing less code can be found here.

Best of all, H4R is small. The core function, useRedux, is about 10 lines of code, and the entire library is just 90. That means, even with a tiny project like this, not only is your source code smaller and more robust, but your total code-size is also smaller. The savings only grow as your project grows.

What to Read Next

I have two follow-up articles to this original H4R announcement. I dive deeper into the good software engineering, modular design, and how to write better Redux with or without H4R.

• The 5 Essential Elements of Modular Software Design
• Modular Redux — a Design Pattern for Mastering Scalable, Shared State

Resources and Acknowledgments

• Get started with hooks-for-redux on npmjs.org
• Learn more about how hooks-for-redux works
• input source:
  github.com/sunil-sandhu/redux-hooks-todo-2019
  all 175 lines of JavaScript from Sunil’s code in one file
• output source:
  github.com/shanebdavis/hooks-for-redux-todo-2019
  all 85 lines of the H4R JavaScript in one file
• Thank you Sunil Sandhu for encouraging me to write this Medium post.
• Hooks-for-redux was developed at GenUI.co. I love that GenUI values contributing back to open source, and I’m grateful for the opportunity to create H4R. Thank you to my colleagues at GenUI who helped test and refine H4R.

First, the Options

There are four image-placement options in Medium. Your image must be at least 1000 px wide for all four options to show up.

• Full column-width images ( recommended >1400 px )
• Out-set images ( recommended > 2040 px )
• Screen-width images ( recommended > 2500 px )
• Out-set, float-left images ( medium doesn’t seem to have an official name for these)

Source: https://help.medium.com/hc/en-us/articles/215679797-Images

The options look like this (on Desktop browsers):

Full Column Width

This is the default option. As the name implies, the image is scaled to just fill the width of the column.

Screen-Width Image

These are dramatic, full-scree images inline with your text. Note the extra vertical padding given to them to add to their weight.

Out-set Image

This option is somewhere between column-width and screen-width. It is always at least a little wider than column-width with a max-width of screen-width. Usually it’s about an inch wider than column-width. Try adjusting the width of your browser to see. Also, note that it left-aligns with Out-set, Float-Left images.

Out-set images have slightly more vertical padding than column-width images.

Out-set, Float-Left Image

I couldn’t find the official Medium-name for this style, so I made one up. This is the only option that lets you place text alongside your image. It is also the smallest presentation option.

These images have the same vertical padding as the other out-set image option.

Atticus said to Jem one day, “I’d rather you shot at tin cans in the backyard, but I know you’ll go after birds. Shoot all the blue jays you want, if you can hit ’em, but remember it’s a sin to kill a mockingbird.” That was the only time I ever heard Atticus say it was a sin to do something, and I asked Miss Maudie about it. “Your father’s right,” she said. “Mockingbirds don’t do one thing except make music for us to enjoy. They don’t eat up people’s gardens, don’t nest in corn cribs, they don’t do one thing but sing their hearts out for us. That’s why it’s a sin to kill a mockingbird.

— Harper Lee, To Kill a Mockingbird