One useful strategy is Elimination by Aspects (EBA). This approach helps you narrow down options systematically until the choice becomes clear. The goal is to show how you can quickly get the “shape” of a problem, the rough contours of the solution space, without exhaustive analysis of every option.

What is Elimination by Aspects?

Elimination by Aspects is a decision-making heuristic introduced by psychologist Amos Tversky in 1972. The core idea is to gradually whittle down a list of alternatives by applying one criterion (aspect) at a time. Then you move to the next criterion, and so on, until you’re left with a manageable shortlist or even a single winner. In other words, you filter options sequentially by your must-have requirements.

Key characteristics of EBA:

• It’s an iterative, non-compensatory process. This means an option that fails one essential criterion is out, no matter how great it is on other factors.
• You’re effectively making a series of yes/no decisions on each aspect, rather than comparing everything against everything at once.

Example: Buying a Lawnmower with EBA

To illustrate the process, let’s use a non-software example first: buying a lawnmower.

1. Type of Mower: Is it a ride-on, robotic, or push mower?
2. Yard Size: How large is the area you need to mow?
3. Terrain: Do you need to mow on steep slopes (15° or more)?
4. Power & Maintenance: Do you prefer gas engine power or electric (battery/corded)? And how much maintenance are you okay with?
5. Budget Range: What’s your budget ceiling?

By sequentially applying these aspects, you might cut the options down from thousands to a handful. Essentially, you ask the most consequential questions first (those that divide the field the most). EBA works best when you choose aspects that maximize information gain early in the process.

I found that ChatGPT Thinking/Pro, or Gemini 2.5 Pro (both with web search enabled) generated pretty good EBA-style questions, the prompt I used was “What are the top 5 questions to narrow down my lawnmower search?”.

Example: Selecting a CSV Parser Library with EBA

Now let’s apply EBA to a software engineering decision. Suppose you need to choose a CSV parsing library for a project. Here’s how an elimination-by-aspects strategy might look:

1. Start with the Universe of Options: Do a broad search filtered by your programming language to list all CSV parser libraries that could be relevant. This is your initial pool.
2. Suitability for Production: Immediately discard any libraries that look obviously unsuitable for production use.
3. Basic Viability Check: Apply a few must-have sanity criteria to the remaining list:
   • Compilation/Installation: The library should at least compile/build or install cleanly.
   • Popularity/Community Usage: While not a perfect metric, check if the library has at least a minimal level of adoption, for instance, a few hundred stars on GitHub or a decent number of weekly downloads on NPM/PyPI.
   • Documentation: If there’s no README or documentation, that’s a huge red flag.
   • When was the package published, was it yesterday? If it’s been out for a while, then there is more time for people to report the package being malicious, etc.
4. Feature and Performance Requirements: With a shorter list in hand, introduce more specific criteria based on your project’s needs:
   • Performance: Do you need to parse very large CSV files or do streaming?
   • Features: Identify required features (e.g., does it handle quoted fields correctly? Can it parse into custom data types or handle different delimiters? Does it also support writing CSV, if you need that?).
   • Robustness: Consider how the library handles malformed data or edge cases (like newline characters within fields, missing values, etc.).
   • Dependencies: Does the library drag in huge external dependencies or native modules?
   • Maintenance: Is the library actively maintained?
5. Final Selection: By this point, you’ve likely narrowed it down to a handful (or even a single) candidate that meets all your aspects.

I applied elimination by aspects to 200 npmjs libraries tagged with “csv” by attaching all 200 readmes to a Gemini 2.5 Pro chat, here is how I narrowed them down:

Further refinement for “server-side, not browser-based user downloads” reduced the choices to 30. This iterative process allowed for increasingly specific filtering based on desired functionality. You can select multiple branches in parallel as well if you’re unsure which direction to take or skip questions.

If you’re not willing to download 200 readmes (I don’t blame you), the prompt I used for Gemini was “use elimination by aspects questions to tell me which library I should use based on my answers to your questions to select a csv parser library on npmjs.org.” and then it interviewed me and chose a library for me based on my answers.

Benefits of Using EBA in Tech Decisions

Employing elimination-by-aspects in software engineering decisions offers several benefits:

• Reduces Overwhelm: By focusing on one criterion at a time, you avoid the mental burnout of weighing every factor of every option simultaneously.
• Ensures Must-Haves Are Met: EBA forces you to identify and prioritize your non-negotiable requirements up front.
• Transparent and Defensible Process: The step-by-step nature of EBA makes your decision process transparent.
• Reduces Bias, Promotes Objectivity: Deciding on criteria before getting enamored with a particular option can mitigate knee-jerk bias toward a familiar or “shiny new” technology.

Caveats and Limitations to Watch For

No decision technique is perfect. Keep these caveats in mind when using elimination by aspects:

• Non-Compensatory = No Trade-Offs: Because EBA is non-compensatory, an otherwise great option will get tossed out if it fails on one chosen criterion. Tip: Choose your elimination aspects carefully and make sure each one really is a deal-breaker.
• Order Matters: The sequence in which you apply criteria can affect the outcome. It’s usually wise to start with the highest priority aspect, essentially, assert “if it doesn’t have X, nothing else matters” only for truly fundamental X’s.
• Requires Clear, Measurable Criteria: EBA works best when your aspects are well-defined. For instance, define scalability as “must handle >10k requests/sec” or security as “must have no critical vuln reports in the last year”, whatever fits your context.
• May Not Yield a Unique Winner: Sometimes you’ll go through your list of aspects and still end up with a tie or a few viable candidates. If multiple options survive all your filters, you can then switch to comparing them on secondary attributes or even doing a proof-of-concept with each.

Wrapping Up

Elimination by aspects is a handy tool in the decision-making toolbox for software engineers. Whenever you’re faced with a daunting list of technologies or design choices, think in terms of aspects: figure out your must-haves, and start chopping off options that don’t check those boxes.

In the fast-moving tech world, where new libraries and frameworks pop up weekly, this approach can help you and your team avoid analysis paralysis and make decisions with confidence.

Ultimately, elimination by aspects won’t guarantee a perfect choice (no method can), but it will give you a rational, repeatable process to arrive at a good choice that meets your needs. It provides practicality and efficiency in decision-making, even if it may not lead to the absolute optimal outcome in hindsight.

Note: rewritten 2025-12-28.

In Part 1 (List), we contrasted Map (Select) vs Bind (SelectMany) on List<T>, then built Maybe<T>.

If you read Part 1, you already know the shape: Bind/SelectMany chains steps, and Maybe decides whether the next step runs.

The Result monad¹ lets you compose computations that can fail. You return Ok(value) or Fail(error), then compose with Bind to propagate the first failure (later steps don’t run; the failure just flows through).² It’s useful for making expected failure explicit and composable.

Result<TSuccess, TError> has the same two-case shape as Maybe<T>, except the non-success case carries a reason (TError). Maybe models optionality; Result models failure with an explicit reason.

Prefer to compose with Bind until you need to branch, translate, or produce an output (often at a boundary/edge), then Match. ³

If you need to accumulate many errors (e.g., form validation) or you have several first-class outcomes, Result may not be the best fit.⁴

If you’re coming from FP, this is closest to an Either/Result-style type.⁵

TL;DR

What it looks like (implementations for User left out for brevity):

public record Error(string Code, string Message);
// Assume `repo` is in scope.

// Creating results:
Result<User, Error> okUser = Result<User, Error>.Ok(new User(/* ... */));
Result<User, Error> failed = Result<User, Error>.Fail(new Error("NotFound", "User not found"));

// Assume:
// Result<int, Error> ParseId(string inputIdFromRequest)
// Result<User, Error> DeactivateDecision(User user)

// Returning a Result from a function:
Result<User, Error> FindUserOrFail(IUserRepo repo, int id)
{
    User? user = repo.Find(id);
    if (user is null)
    {
        return Result<User, Error>.Fail(new Error("NotFound", $"User {id} not found"));
    }

    return Result<User, Error>.Ok(user);
}

Result<User, Error> result =                   // Result<User, Error>
    ParseId(inputIdFromRequest)                // Result<int, Error>, first in chain, always runs
        .Bind(id => FindUserOrFail(repo, id))  // Result<User, Error>, only runs if ParseId succeeded
        .Bind(DeactivateDecision);             // Result<User, Error>, only runs if FindUser succeeded

// Handle at the boundary (or when translating between layers):
string message = result.Match(
    ok:  _ => "User deactivated",
    err: e => $"Deactivate failed: {e.Code} - {e.Message}");

On success, Bind passes the inner value to the next step; on failure, it forwards the Error. Short-circuiting allows the failure to propagate: after the first Fail, later steps are bypassed and the error flows through to the end. Bind encodes the control flow so your business logic doesn’t have to repeat it.

Note: You may also see this described as “railway switching”, “bypassing”, “error propagation”, or “fail-fast”.

Yes, this example uses a repository (a very .NET thing) and mutates a User. That’s deliberate: I don’t want this post to imply you should replace exceptions everywhere with Result.

The core idea is simple: Bind chains successes and short-circuits on the first failure.

The problem: explicit vs. implicit

In C#, fallible work is often handled with exceptions, or with explicit branching (TryX/guard clauses/status checks) depending on whether failure is expected.

Option A: Implicit Control Flow (Exceptions)

Method signatures often don’t advertise failure when using exceptions, unlike TryX/bool-return patterns.³ DeactivateUser (below) returns void, so failures aren’t visible in the signature. In this style, it might throw for parsing/loading/saving and even for business-rule failures.

The following shows a style where expected failures are represented as exceptions (sometimes called “exceptions as control flow”; some may avoid this style). It’s intentionally heavy-handed.

Typical C# code is often closer to: parse with TryParse, call a repo/service that may throw, then catch once at the boundary.

// The implicit "User" entity used in the examples below
public class User
{
    public int Id { get; set; }
    public bool IsActive { get; set; }
}

User is mutable here to keep focus on Result. Prefer immutability where practical in domain code; some ecosystems (e.g., ORMs) still push you toward mutation.⁶

private readonly IUserRepo _repo;

public void DeactivateUser(string inputId)
{
    if (inputId is null) throw new ArgumentNullException(nameof(inputId));

    int id;
    try
    {
        id = int.Parse(inputId);
    }
    catch (Exception ex) when (ex is FormatException or OverflowException)
    {
        throw new InvalidOperationException("DeactivateUser failed at: parse id", ex);
    }

    User? user;
    try
    {
        user = _repo.Find(id);
    }
    catch (Exception ex)
    {
        throw new InvalidOperationException("DeactivateUser failed at: load user", ex);
    }

    if (user is null)
        throw new InvalidOperationException("User not found");

    if (!user.IsActive)
        throw new InvalidOperationException("User already inactive");

    user.IsActive = false;
    try
    {
        _repo.Save(user);
    }
    catch (Exception ex)
    {
        throw new InvalidOperationException("DeactivateUser failed at: save user", ex);
    }
}

Main point: in the code above, you’re responsible for null checks, initializing the user variable outside try/catch, and stopping (early return/throwing an exception). It’s noisy and easy to get wrong.

In larger apps, exceptions often surface far from where you want domain context, so you either catch at boundaries or add local try/catch when you truly need extra context.

Option B: Explicit Validation (Guard Clauses) To avoid throwing for expected failures, you often use TryX-style APIs, guard clauses, and early returns. It’s linear, but still noisy.

private readonly IUserRepo _repo;

public enum DeactivateUserResult
{
    Success,
    InvalidId,
    NotFound,
    AlreadyInactive,
    InfraError
}

public DeactivateUserResult DeactivateUser(string inputId)
{
    if (!int.TryParse(inputId, out var id)) return DeactivateUserResult.InvalidId;

    User? user;
    try
    {
        user = _repo.Find(id);
    }
    catch (Exception)
    {
        // logging omitted for brevity
        return DeactivateUserResult.InfraError;
    }
    if (user is null) return DeactivateUserResult.NotFound;

    if (!user.IsActive) return DeactivateUserResult.AlreadyInactive;

    // assuming an ORM that requires mutation
    user.IsActive = false;
    try
    {
        _repo.Save(user);
    }
    catch (Exception)
    {
        // logging omitted for brevity
        return DeactivateUserResult.InfraError;
    }
    return DeactivateUserResult.Success;
}

An enum gives you a status, but not a success payload. If you need to return data on success, you end up adding an out parameter, returning a tuple, or introducing a wrapper type. Conventions are easy to violate: nothing stops you from returning (user: null, error: null) or populating both.

Note: You can fix the “invalid combinations” problem with an OperationStatus hierarchy (e.g., OperationSuccess / OperationFailure) or a private constructor plus Success(...)/Failure(...) factories. That helps, but you still need good composition to avoid “check the status after every step.”

Result packages these conventions + combinators into a reusable shape (Ok(...)/Fail(...), Map/Bind).

The solution: the Result monad

Result returns expected failure as data (as long as your steps return Result rather than throwing), instead of using an exception jump. Unexpected exceptions still escape.

Now each step either produces the next value or propagates the first Fail(...). This is often described as short-circuiting.²

Non-LINQ syntax (plain method chaining):

Result<User, Error> result =
    ParseId(inputId)
        .Bind(FindUser)
        .Bind(DeactivateDecision);

In C#, string? is a nullable reference annotation; int? is Nullable<int>. In Rust, ? is an early-return operator for Result/Option (and other types that implement the Try pattern).

LINQ query syntax (Select/SelectMany) is in the appendix.

Procedural code detour

How to get out of a Result with Match

Translating between layers often means turning a Result into a different output shape via Match. In practice you’ll often also want MapError (or BindError) to translate error types between layers without ending the pipeline.

Result<int, Error> infraResult = ParseId(inputIdFromRequest);

string response = infraResult.Match(
        ok: id => $"OK: {id}",
        err: e => $"Bad request: {e.Code} - {e.Message}");

At boundaries (HTTP/CLI/public APIs), you typically translate into a DTO/status/ProblemDetails/exit code.

IMPORTANT! C# doesn’t force you to handle a returned Result. Use an analyzer.⁷

A tiny Result implementation

If you’re curious, try implementing Result yourself first.

This teaching implementation isn’t production-ready (use LanguageExt, Danom, or CSharpFunctionalExtensions) and is intentionally minimalist and unsafe around default/null. It stores default in the unused slot (don’t read it), doesn’t prevent Ok(null) / Fail(null), doesn’t guard against “returning null” from Bind, has no async support, no Equals/GetHashCode, and doesn’t catch exceptions – to keep the focus on the core shape.

Performance note: this sample uses a class for clarity, but in high-throughput / low-allocation scenarios you’d typically implement Result as a readonly struct (or record struct) to reduce GC pressure (often alongside careful API design to avoid copying).

Here’s the implementation for Result:

public sealed class Result<TSuccess, TError>
{
    private readonly TSuccess _value;
    private readonly TError _error;
    private readonly bool _isSuccess;

    private Result(TSuccess value, TError error, bool isSuccess)
    {
        _isSuccess = isSuccess;
        _value = value;
        _error = error;
    }

    // Unit
    public static Result<TSuccess, TError> Ok(TSuccess value)
    {
        return new Result<TSuccess, TError>(
            value,
            default,
            true);
    }

    public static Result<TSuccess, TError> Fail(TError error)
    {
        return new Result<TSuccess, TError>(
            default,
            error,
            false);
    }

    public Result<U, TError> Map<U>(Func<TSuccess, U> f)
    {
        if (_isSuccess)
        {
            return Result<U, TError>.Ok(f(_value));
        }

        return Result<U, TError>.Fail(_error);
    }

    public Result<U, TError> Bind<U>(Func<TSuccess, Result<U, TError>> f)
    {
        if (_isSuccess)
        {
            return f(_value);
        }

        return Result<U, TError>.Fail(_error);
    }

    public TResult Match<TResult>(Func<TSuccess, TResult> ok, Func<TError, TResult> err)
    {
        if (_isSuccess)
        {
            return ok(_value);
        }

        return err(_error);
    }
}

Where is Result.Unit(...)? In monad terms: for Result<_, TError>, Ok(...) is return/pure (the monadic “unit”). Fail(...) is the constructor for the error case. Practically, Result.Unit(...) = Result.Ok(...).

Where Result fits (and where it doesn’t)

Rule of thumb: T? for something that could be null, Maybe<T> for expected absence (no reason), Result<TSuccess, TError> for expected failures you’ll handle, and exceptions often for bugs or unrecoverable failures.⁸ Also: model only the failure detail callers can act on; if TError is part of a public API, keep it small and stable.

Prefer Result when:

• Failure is expected and recoverable: validation/business rules, not found, auth failures, parsing user input.
• You want refactor pressure (and sometimes exhaustiveness): refactors become visible because failure is in the return type.
• Failure is routine / on hot paths: avoid throwing for control flow; prefer TryParse/Result-style returns.

Prefer exceptions (or other types) when:

• It’s a bug / broken invariant: violated preconditions, “impossible states” → often exceptions (e.g., ArgumentNullException).
• Continuing is pointless / you need stack traces: misconfiguration, out-of-memory, “dead end” aborts → often exceptions / short-circuit, array out of bounds.
• You need accumulation: Bind is short-circuiting; use Validation<T>/applicatives (or a combine API) for independent validations.

Putting it together

Eventually you turn a Result into something your caller understands (HTTP response, CLI exit code, UI state, etc.). A boundary (the “edge” of the system) is a good place to Match.

Boundary/Application Layer: the boundary (or edge) is where your program interacts with something you don’t fully control (inputs, networks, storage, other systems).

See HandleDeactivateRequest below for a concrete example.

Example: deactivate a user

We want to deactivate a user given a user’s id from an HTTP request (received as a string, parsed to an int).⁹

This HTTP API/user service example is just a convenient boundary; the same idea applies to CLIs, jobs, message handlers, UI workflows, etc.

public class User
{
    public int Id { get; set; }
    public bool IsActive { get; set; }
}

public sealed class UserService
{
    private readonly IUserRepo _repo;
    public UserService(IUserRepo repo)
    {
        _repo = repo;
    }

    public Result<User, Error> DeactivateUser(string inputId)
    {
        // Workflow composition; the write happens at the boundary (see `HandleDeactivateRequest`).
        return ParseId(inputId)
            .Bind(FindUser)
            .Bind(DeactivateDecision);
    }

    public string HandleDeactivateRequest(string inputId)
    {
        Result<User, Error> result = DeactivateUser(inputId);

        return result.Match(
            ok: user =>
            {
                // If this throws, assume middleware/global handlers translate + log infra failures.
                _repo.Save(user);
                return "User deactivated";
            },
            err: e => $"Deactivate failed: {e.Code} - {e.Message}");
    }

    private static Result<int, Error> ParseId(string inputId) =>
        int.TryParse(inputId, out var id)
            ? Result<int, Error>.Ok(id)
            : Result<int, Error>.Fail(new Error("Parse", "Invalid ID format"));

    private Result<User, Error> FindUser(int id)
    {
        var user = _repo.Find(id);
        return user is null
            ? Result<User, Error>.Fail(new Error("NotFound", $"User {id} not found"))
            : Result<User, Error>.Ok(user);
    }

    private static Result<User, Error> DeactivateDecision(User user)
    {
        if (!user.IsActive)
            return Result<User, Error>.Fail(new Error("Domain", "User is already inactive"));

        user.IsActive = false;
        return Result<User, Error>.Ok(user);
    }
}

This computes Result<User, Error> internally, then Matches at the boundary (HandleDeactivateRequest) to produce the caller-facing output. In a real HTTP endpoint, you’d return IActionResult/IResult (not a string) and map Error to ProblemDetails/status codes.

Why is _repo.Save(user) inside Match?

Save is I/O. It can fail with domain-relevant outcomes (e.g., uniqueness conflicts) and unexpected infrastructure exceptions (timeouts, outages). If you want conflicts to be “expected failures,” catch and translate the specific exception into a Result error; otherwise let truly unexpected exceptions bubble to boundary handlers.

Why serializing Result directly can be awkward

In public contracts, serializing Result tends to leak an internal control-flow wrapper into your schema. Prefer Match into a DTO / HTTP status / ProblemDetails (unless using a custom converter).

Some Result types could expose public Value/Error/flags, which can make serialization even worse. If you serialize a Result-shaped class directly, you can end up with confusing “wrapper JSON” like this:

{
  "isSuccess": false,
  "value": null,
  "error": { "code": "DbError", "details": { /* ... */ } }
}

Now your public API couples clients to an internal control-flow wrapper and can leak internal shapes. Clients have to interpret isSuccess + value/error and your HTTP status code. What if isSuccess is true and error contains an error? It’s a weird situation.

Note on async: Task<Result<...>> nesting is where people get stuck

Once you mix Task and Result, you quickly end up with Task<Result<TSuccess, TError>>, and plain LINQ query syntax doesn’t compose that shape out of the box (because Task<Result<...>> doesn’t have the right SelectMany).

In practice you either:

• add async-aware combinators like BindAsync/MapAsync, or
• add LINQ helpers like SelectManyAsync to compose Task<Result<...>> without “await soup”.

Rather than reimplement those helpers here, use a library that provides them.

Recap

• Result<TSuccess, TError> makes expected failure explicit and composable.
• Use Bind for short-circuiting pipelines; Match to produce caller-facing output.
• For public contracts, unwrap Result into DTOs (rather than serializing it). Decide where you catch/translate exceptions.

Appendix: LINQ query syntax (Select/SelectMany)

If you want from/from/select query syntax to compile, add these extension methods (or add them directly to Result). Other keywords like where require additional methods.

public static class ResultLinqExtensions
{
    // Required for `select`
    public static Result<U, TError> Select<TSuccess, U, TError>(
        this Result<TSuccess, TError> result,
        Func<TSuccess, U> selector) =>
        result.Map(selector);

    // Required for multiple `from` clauses
    public static Result<V, TError> SelectMany<TSuccess, U, V, TError>(
        this Result<TSuccess, TError> result,
        Func<TSuccess, Result<U, TError>> bind,
        Func<TSuccess, U, V> project) =>
        result.Bind(val => bind(val).Map(next => project(val, next)));
}

Part 3 coming soon.

Note Sept 13 2025: this post has been revised based on the feedback from the Hacker News discussion.

The term “monad” is often invoked when describing patterns in functional programming. At the heart of monadic programming is sequencing computations, so each step can depend on the previous one while the monad threads context.

You may erroneously think all monads are containers, or burritos, or boxes. The simplest of monads can be idealized as a container (albeit a flawed metaphor). Monads are much more than just containers, and there isn’t the-one-and-only monad; instead it’s better to think about them as a programming pattern, recipe, factoring out control flow, or, in some cases, a deferred computation. It depends on which monad you’re talking about.

From a teaching perspective, to get the concept for what a monad is, we will start with the simplest of monads which will feel a lot like just a container but with some composable aspects. This provides the infrastructure to understand more complex monads later on.

List: Map & flatMap in Practice

To an OOP developer, monadic types (List<T>) might look just like generics. It’s a typical pitfall to think “we have generics, so we have monads,” which isn’t true by itself. Monads do usually involve generic types, but they require specific operations (Unit and flatMap) and the three monad laws on those types to ensure uniform behavior. This is key and is fundamental to working with monads.

A good example of a monad is List. You’re likely very familiar with lists and working with lists.

The monad Map operation is responsible for:

• Applying your function. For List, Map runs f (a function) on every element. For example, let’s define f as f(x) = x + 1. The list [0,1,2,3] becomes [1,2,3,4]. If the list doesn’t have any elements, then Map doesn’t call f. f doesn’t need to worry about that. Also, f doesn’t care if it’s List<T>; all f is, is just f(x) = x + 1. Map is responsible for running it.

• Managing sequencing and combination. The list context concatenates all results into one list (Map does not flatten any nested lists, flatMap is responsible for this). We don’t need to manually re-add elements via Add or otherwise manage the collection ourselves.

Notice that the monad in this case List<T> is responsible for running f. This shift means your business logic stays declarative and composable, you describe what happens to a single value, and the monad describes how and when it happens.

This is different from object-oriented and procedural programming because in those paradigms, if you want to process data, it is your responsibility to understand how to apply the function to your data. We have to use different control constructs to handle different types of data, and we’re also responsible for the “how”:

public string f(string input) {  
  return input + " -appended text";  
}

// 1. List<string>: you must foreach and build a new list

var fruits = new List<string> {  
  "apple",  
  "banana",  
  "cherry"  
};

var newFruits = new List<string>();

foreach (var fruit in fruits)  
{  
  newFruits.Add(f(fruit));  
}

// 2. Single string: you must check for null first, then concatenate

string userInput = GetUserInput(); // could be null

if (userInput != null)  
{  
  userInput = f(userInput);  
}

// userInput could still be null here, or it could be the concatenated result

// 3. Dictionary<string, string>: you must know it’s key/value pairs

var dict = new Dictionary<string, string>  
{  
  ["a"] = "alpha",  
  ["b"] = "beta",  
  ["c"] = "gamma"  
};

// can’t modify while iterating, so capture keys first  
foreach (var key in dict.Keys.ToList())  
{  
  dict[key] = f(dict[key]);  
}

In these examples, we are forced to know how to update each structure procedurally. For a List, we have to call Add; for the string we can update it in place; for the Dictionary, we have to iterate over keys and update each entry. We have to know it’s a List beforehand to know to use foreach. We have to know it’s just a string to append another string to it.

With monads, you delegate the control flow to the monad itself, the monad knows how to update its underlying value(s). Recall that even the simplest monads must implement two methods to be monads (Unit and flatMap) and must follow three monad laws.

Unit

Unit moves a raw value into the monadic context (this operation is sometimes called “lifting”, “identity”, “return”, “wrap”, or “promotion”, and in some libraries has names like liftM or liftA).

• In the list monad, Unit takes a single element and returns a list containing that element.

• For example, given the integer 1, Unit produces a list as follows:

Example (C#):

var list = new List<int> { 1 };

List<T> implements Unit because it allows moving a value into the mondaic context. Nothing about the value 1 changes, it’s simply wrapped in a List. If you access element 0 of that list, you get back 1. That’s it.

Map

Map applies a function to each value inside the monad.

In List, Map runs a function on every element and outputs a new list with that function applied to each element. Don’t overcomplicate it. For example, suppose we have a function that adds one, f(x) = x + 1. Passing this function to Map would simply add one to each element in the list. The list [0,1,2,3] would become [1,2,3,4].

Example (C#-ish):

var originalList = new List<int> { 0, 1, 2, 3, 4 };    
var mapped = originalList.Map(x => x + 1); // `Map` doesn’t exist in C# (use LINQ's `Select`), but assume this pseudocode

Example (C#, without monads):

var originalList = new List<int> { 0, 1, 2, 3, 4 };    
var mappedList = new List<int>();

foreach (int x in originalList)    
{    
    mappedList.Add(x + 1);    
}

How do you get the damn values out of the monads?

Ideally, you don’t want to pull the values out of a monad unless you absolutely have to. It’s possible to implement a GetValue() method that returns the underlying value, but once the value leaves the monadic context, we lose the benefits of that context and can no longer compose operations easily.

Think about List<T> as if you had never seen it before. You might say, “I don’t want my values trapped in this list, how am I supposed to use them?” and then manually extract each element into separate variables:

// Pretend it’s your first time with List<T>
var numbers = new List<int> { 1, 2, 3 };

// --- Manual extraction (values “trapped” in the list) ---
var a = numbers[0];
var b = numbers[1];
var c = numbers[2];

// Now call your function separately on each:
var r1 = AddOne(a);
var r2 = AddOne(b);
var r3 = AddOne(c);

But by doing so, you lose the advantages of using a list in the first place: the ability to store arbitrarily long sequences, to pass around all the values together, to concatenate with other lists, and to iterate easily. If you want to add one to each item, extracting them one by one and handling each separately is tedious and error-prone.

Up to this point, monads might just seem like “fancy containers” that have to implement two odd methods (Unit and flatMap). Let’s explore a slightly more complex monad to see why they’re more than just containers.

Maybe

Let’s consider a case where unwrapping the value may not always make sense. We’ll create a monad called Maybe (often also called an Option) which represents either an existing value or the absence of a value.

For simplicity, our MaybeMonad will hold an int internally (in a real library this would be a generic Maybe<T>). It’s not exactly a full monad yet, because we haven’t implemented flatMap on it.

public class MaybeMonad {    
    private int value;    
    private bool hasValue;  

    // Unit    
    public MaybeMonad(int value) {    
        this.value = value;
        this.hasValue = true;
    }  

    // Unit (no value)
    public MaybeMonad() {
        // hasValue remains false by default
    }  

    // Map    
    public MaybeMonad Map(Func<int, int> func) {
        if (hasValue) {
            return new MaybeMonad(func(value));
        }
        return this;
    }
}

Here, the Unit operation corresponds to calling one of the constructors, that’s how we lift a raw value into a MaybeMonad. The Map operation might feel a bit strange because we’re just dealing with a single value (or none), whereas you might be used to mapping over a list of many values.

For example, to add 1 to a MaybeMonad:

var age = new MaybeMonad(30);    
var newAge = age.Map(x => x + 1);    
// newAge now holds 31

Or if there was no value to begin with:

var age = new MaybeMonad();    
var newAge = age.Map(x => x + 1);    
// newAge is still “nothing”, `Map` didn’t call `f(x)` because there was no value

This looks verbose just to add 1 to a number. Why wrap 30 in a MaybeMonad and call Map when we could have just incremented 30 directly? The point is that age is a MaybeMonad, by definition it might or might not contain a value. In the case where there is no value, MaybeMonad’s Map simply does nothing. You’d have to write the same conditional logic yourself in a procedural style:

int? age = null;    
if (age != null) age++;

Or:

int? age = 30;    
if (age != null) age++;

Now we start to see why a monad is not simply a container to be unwrapped at will. How would you “unwrap” a MaybeMonad? If it has a value, you could return it, sure. But if it doesn’t, there’s nothing to return, the absence itself is a meaningful state. MaybeMonad essentially encodes the idea of “nothing” (no result) in a way that isn’t just null (because in many languages null is still a concrete value of sorts). With MaybeMonad, if there’s no value, any function passed into Map simply won’t execute. Unwrapping it and getting a raw value out isn’t always meaningful in this context.

Another benefit of monads is that you can chain computations that themselves produce monadic results. The limitation of only having Map is that you might end up with nested monads. For example, imagine a function that returns a Maybe<int>. If you call Map on a Maybe<int> with that function, the result would be a Maybe<Maybe<int>>, a nested container, because the Map wraps the function’s Maybe<int> result into yet another Maybe. We need a way to apply a function that returns a monad and avoid this unnecessary nesting when chaining operations.

flatMap

flatMap is like our Map, but it also flattens the result. flatMap provides the ability to chain computations that themselves produce monadic values, which is the defining feature of monads. For example, if you have a function that looks up a user and returns a Maybe<User>, but you want to pass it to another function that returns the user’s profile. Using Map would give you a Maybe<Maybe<UserProfile>>, an awkward nested container because the input would be a Maybe<UserProfile>. With flatMap, you both apply your lookup and collapse the layers in one go, so you can seamlessly sequence optional, error-handling, or asynchronous operations (e.g. promises/tasks) without ever wrestling with nested monadic types.

Here’s what flatMap looks like:

// Add this method inside MaybeMonad
public MaybeMonad FlatMap(Func<int, MaybeMonad> func)
{
    if (hasValue)
    {
        // Do not wrap again; let the callee decide whether to return a value or "nothing"
        return func(value);
    }
    // Propagate "no value"
    return this;
}

Use flatMap when your next step might also produce “no value,” and you want to keep chaining without ending up with Maybe<Maybe<int>>.

Maybe<User> lookupUser(string id)  
{  
    // Imagine this calls a database or external service and returns Maybe<User>  
    return GetUserFromDatabase(id);  
}

Maybe<string> userIdMaybe = GetUserId();

// Using Map would yield Maybe<Maybe<User>> (nested) because lookupUser returns a Maybe<User>.  
// This quickly becomes unwieldy and makes further processing difficult.  
var nested = userIdMaybe  
    .Map(lookupUser);

// Using flatMap collapses the result to a single Maybe<User>  
var user = userIdMaybe  
    .FlatMap(lookupUser);

flatMap is arguably more important than Map, in fact, flatMap is required to qualify as a monad, and given flatMap you can implement Map in terms of it.

What does this chaining look like procedurally? It would be similar to:

string userId = GetUserId(); // could be null  
if (userId == null) {  
  // e.g., return an error or stop here  
}

User user = GetUserFromDatabase(userId); // this could return null (no user found)  
if (user == null) {  
  // handle missing user  
} else {  
  // we have a valid user  
}

In the procedural version, we had to explicitly handle the control flow at each step (checking for null in this case). In the monadic version, the control flow is implicit in the monad. If userIdMaybe has no value, flatMap simply doesn’t call lookupUser at all, the “else do nothing” logic is built into Maybe.

In the monadic example, you could write:

Maybe<string> userIdMaybe = GetUserId();  
Maybe<User> userMaybe = userIdMaybe.FlatMap(lookupUser);

The monads handle the control flow for us. GetUserId() returns a Maybe because we’re acknowledging the user ID might not exist. We’ve defined the Maybe monad such that if there’s no value, any subsequent function (like lookupUser) won’t execute. There’s nothing mystical here, we explicitly designed Maybe to work that way.

This is why it makes sense to wrap values in monads and keep chaining within the monadic context: you can sequence operations (like getting a user ID, then looking up a user, then perhaps fetching their profile) without writing a single explicit if or loop for the control flow. Each monad step handles the logic of “if there’s no value, stop here” automatically.

If you prematurely yank a value out of a monad, you end up doing manual work that defeats this benefit. For instance, consider if we had a GetValue() method to extract the inner value (with null representing “no value”):

Maybe<string> userIdMaybe = GetUserId();  
var actualUserId = userIdMaybe.GetValue();  
if (actualUserId != null) {  
    // do something with actualUserId  
}

Eww. If we treat the monad as just a fancy wrapper to put a value in and then take it out immediately, it does feel like pointless ceremony. This is where many people give up on learning monads, it seems like you’re just putting a value in a box and taking it out again with extra steps. But the power of monads comes when you stay inside the monadic context and keep chaining operations. In Part 2, we’ll look at more advanced monads that aren’t just simple containers, and you’ll see how staying in the monadic pipeline pays off.

Closing the loop on Maybe

We’re making a few changes to the Maybe monad to give it a more official, ergonomic API. First, instead of letting callers construct the underlying representation directly, we’ll expose two factory methods: Some and None. Second, we’ll generalize map: instead of only mapping over integers, the monad will be generic so it can map any type. Finally, we’ll standardize the name to Maybe<T>. Together, these tweaks clean things up and make the monad easier to use across more scenarios.

public sealed class Maybe<T>
{
    private readonly bool _has;
    private readonly T _value;

    private Maybe(T value)
    {
        _has = true;
        _value = value;
    }

    private Maybe()
    {
        _has = false;
        _value = default(T);
    }

    public static Maybe<T> Some(T value)
    {
        return new Maybe<T>(value);
    }

    public static Maybe<T> None()
    {
        return new Maybe<T>();
    }

    public Maybe<U> Map<U>(Func<T, U> f)
    {
        if (_has)
        {
            return Maybe<U>.Some(f(_value));
        }
        return Maybe<U>.None();
    }

    public Maybe<U> Bind<U>(Func<T, Maybe<U>> f) // aka FlatMap
    {
        if (_has)
        {
            return f(_value);
        }
        return Maybe<U>.None();
    }
}

To wrap up Maybe: it’s perfect when you only need to model “value or no value.” Often, we also need to know why a value is missing (not found, invalid input, business‑rule violation). Maybe can’t carry that reason.

Monad Laws

To be a true monad, a type must not only provide Unit and flatMap operations, but also obey three simple laws that make sure these operations behave consistently:

1. Left Identity: Unit(x).flatMap(f) is the same as f(x). (Wrapping a value and then immediately applying a function to it is equivalent to just calling the function on the raw value.)
2. Right Identity: m.flatMap(Unit) is the same as m. (If you flatMap a monad with the Unit function, the monad should remain unchanged.)
3. Associativity: m.flatMap(f).flatMap(g) is the same as m.flatMap(x => f(x).flatMap(g)). (It doesn’t matter how you parenthesize nested flatMap operations, the outcome will be the same.)

You don’t need to memorize these laws, but they provide a mathematical guarantee that monadic operations will compose reliably. Our MaybeMonad adheres to these laws, making it a true monad.

As we’ve seen, monads provide a context for computation. By defining two core operations, Unit (to wrap a value) and flatMap (to sequence operations that produce a new context), we abstract away manual control flow like loops and null-checks. This lets us turn scattered procedural code into a single declarative pipeline.

The real power comes when we apply this pattern to different contexts. In Part 2, we’ll explore other useful monads, like Either for more descriptive error handling, and see how to combine monads to manage multiple concerns at once.

Exercise for reader: I’d encourage opening up your IDE, without any AI assistance, and implementing the Maybe monad from scratch (no cheating.)

Part 2

When we go into the day-to-day business side of things, things become less clear. For example, a customer’s email is a string–sure, if my function expects the String type, then I’ll get a string. But we have to do defensive validation, cluttering up our code with validation checks everytime we pass this mysterious customer email address string, that might not be valid.

In C#, we can simulate this type safetyness by using the static factory pattern. In this example, the ValidatedObject cannot be instantiated via new(), it must be through the factory. This gives a few advantages, namely, the object (or record in this case) cannot exist unless it is valid. Therefore, since it is also sealed, and immutable (as it is a record type), we don’t have to constantly re-validate it.

Additonally, we get the bonus of being type safe. That is, say a method accepts two strings: a customer name and email. If it’s typed as a CustomerEmail, then it is not possible to compile your code or get customer name and email mixed up in the method params.

public static partial class ValidatedObjectCreator
{
    internal sealed record ValidatedObject : IValidatedObject 
    {
        public string Data { get; }

        private ValidatedObject(string data) => this.Data = data;

        public static IValidatedObject Create(string data)
        {
            if (!IsValid(data)) throw new ArgumentException("Invalid input data.");

            return new ValidatedObject(data);
        }

        // Basic validation logic (can be customized)
        private static bool IsValid(string data) => data.Length > 3;

        public override string ToString() => Data;
    }

    internal interface IValidatedObject
    {
        static IValidatedObject Create(string data) => throw new NotImplementedException();
        // expose properties here as needed
    }
}


public static class Program
{
    public static void Main()
    {
        var validatedObject = ValidatedObjectCreator.ValidatedObject.Create("Test data");
        // this will fail
        // var validatedObject = new ValidatedObject("Test data");
    }
}

I went through about 360GB (gzipped) of journal article keywords from The General Index. One of the datasets in the The General Index is a dataset of keywords extracted from 107 million journal articles.

I made a quick shell script to parse the data and collect all keywords that ended in “Ops” case sensitively. I deleted some that were completely non-sensical (composed of punctuation) and also highlighted some that I thought were particularly interesting.

The following table is some of the -Ops terms.

It was a hot summer day inside my apartment. At least not for long. I just got an air conditioner from my landlord and it was time to fire it up.

A few weeks later

I was in the middle of writing a program to decompress some gzip files in C# and I got a strange exception that indicated that the archive was corrupted:

Unhandled exception. System.IO.InvalidDataException: The archive entry was compressed using an unsupported compression method.

System.IO.Compression.Inflater.Inflate(FlushCode flushCode)
     at System.IO.Compression.Inflater.ReadInflateOutput(Byte* bufPtr, Int32 length, FlushCode flushCode, Int32& bytesRead)
     at System.IO.Compression.Inflater.ReadOutput(Byte* bufPtr, Int32 length, Int32& bytesRead)
     at System.IO.Compression.Inflater.InflateVerified(Byte* bufPtr, Int32 length)
     at System.IO.Compression.DeflateStream.ReadCore(Span`1 buffer)
     at System.IO.Compression.DeflateStream.Read(Byte[] array, Int32 offset, Int32 count)
     at System.IO.StreamReader.ReadBuffer()
     at System.IO.StreamReader.ReadLine()
     at MyApp.Program.ReadAllZippedLines(String filename)+MoveNext()
     at System.Linq.Enumerable.EnumerablePartition`1.MoveNext()
     at MyApp.Program.Main(String[] args)
     at MyApp.Program.<Main>(String[] args)

This was very strange, because the files were not normally corrupted. This was very concerning because I would have lost a lot of data if that were the case. Out of irrationality, I re-ran the program, and it worked? Weird, I thought. Maybe Windows had the file open with an anti-virus scanner for a second and I caught it at a bad time.

But nevertheless, the error returned again a few minutes later. I decided to turn off Windows Defender, rebooted my laptop, and tried again. No luck. The strange thing was the error was not predictable, sometimes it would work, other times not.

When I re-ran it again, Visual Studio waited on the line (due to an exception handler.) I was able to inspect the code at that point (located in C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.1.16\System.IO.Compression.dll):

internal int ReadCore(Span <byte> buffer) {

  this.EnsureDecompressionMode();

  this.EnsureNotDisposed();

  this.EnsureBufferInitialized();

  int start = 0;

  while (true) {

    do {

      int num = this._inflater.Inflate(buffer.Slice(start));

      start += num;

      if (start == buffer.Length || this._inflater.Finished() && (!this._inflater.IsGzipStream() || !this._inflater.NeedsInput()))

        goto label_7;

    }

    while (!this._inflater.NeedsInput());

    int count = this._stream.Read(this._buffer, 0, this._buffer.Length);

    if (count > 0) {

      if (count <= this._buffer.Length) < this conditional was false

      this._inflater.SetInput(this._buffer, 0, count);

      else

        break;

    } else

      goto label_7;

  }

  throw new InvalidDataException(SR.GenericInvalidData);

  label_7:

    return start;

}

This was strange, why was the buffer length greater than the count, but only sometimes? This would indicate that the buffer’s length or the count variable were being updated to an incorrect value. The file is the same and hasn’t changed (according to its SHA1 hash.) Except, I couldn’t get the hash the second time due to a strange internal exception error in the plugin for File Explorer I was using.

Could I have discovered a race condition? It didn’t look like the gzip decompressor was multi-threaded. I continued to dig deeper down the callstack,

private ZLibNative.ErrorCode Inflate(ZLibNative.FlushCode flushCode) {

  ZLibNative.ErrorCode errorCode;

  try {

    errorCode = this._zlibStream.Inflate(flushCode);

  } catch (Exception ex) {

    throw new ZLibException(SR.ZLibErrorDLLLoadError, ex);

  }

  switch (errorCode) {

  case ZLibNative.ErrorCode.BufError:

    return errorCode;

  case ZLibNative.ErrorCode.MemError:

    throw new ZLibException(SR.ZLibErrorNotEnoughMemory, "inflate_", (int) errorCode, this._zlibStream.GetErrorMessage());

  case ZLibNative.ErrorCode.DataError:

    throw new InvalidDataException(SR.UnsupportedCompression);

  case ZLibNative.ErrorCode.StreamError:

    throw new ZLibException(SR.ZLibErrorInconsistentStream, "inflate_", (int) errorCode, this._zlibStream.GetErrorMessage());

  case ZLibNative.ErrorCode.Ok:

  case ZLibNative.ErrorCode.StreamEnd:

    return errorCode;

  default:

    throw new ZLibException(SR.ZLibErrorUnexpected, "inflate_", (int) errorCode, this._zlibStream.GetErrorMessage());

  }

}

It was returning a DataError or a StreamError, sometimes. This meant that there was something wrong with the data or the stream (yes, a bit obvious.)

To rule out if it was a coding error, I tried to decompress the gzip file with bash via gzip -dc file. bash threw a strange error, “can’t seek file descriptor” when trying to read the file. This error is emitted from bash.c here. I also tried decompressing several other gzip files, but they were unable to be decompressed.

At this point, I strongly suspected data corruption. I quickly checked CrystalDiskInfo for any reallocated sectors but could not find any. I then ran sfc.exe /SCANNOW, and I believe it did find errors and corrected them. I ran it again, and again, and again, and it kept fixing errors. At some point (maybe five times) it didn’t report any more errors.

Over the next few weeks, I began to have very strange issues with my laptop. Apps would take forever to load (and sometimes not at all), icons would be missing, and text would appear garbled. I figured my laptop was failing at this point. But one mystery remained: there was no incidents of strange behavior when on battery power. Only when plugged in. And why was my monitor, that I recently purchased, also having issues?

Time to go deeper

What caught my attention was that the wrist-rest was very uncomfortable. It was an aging 2012 MacBook Pro, running Boot Camp, but there was no signs of wear on the aluminum. Again, the same pattern came up: the wrist-rest was only uncomfortable when running on AC power.

Aside: at this point, I had to repair Visual Studio about two or three times due to strange ungoogle-able errors.

It all clicked when I got a very bad shock when I touched my laptop’s frame: could it be an electrical issue?

Electricity going where it shouldn’t

Let’s recap on what I found out so far:

- There was a strange green and red noise on my brand new monitor.

- There was a strange green and red noise on my old monitor.

- The trackpad was spicy.

- There (appeared to be) data corruption.

- I got shocked by my laptop.

- Sometimes my apartment lights turned on a little bit by themselves too after I turned them off.

It seems strange that, just out of coincidence, my old and new monitor both exhibited the exact same issue. I also upgraded my laptop’s battery earlier in the year, so perhaps I could have made a mistake?

What could cause things that shouldn’t conduct electricity to conduct it? There could be many issues:

- A bad device that’s wired incorrectly.

- Grounding issues.

- Potentially more issues.

To rule out if it’s caused by another device, as everything was connected using a power strip, I unplugged everything and plugged my laptop directly into a GFCI outlet in the bathroom and turned on all devices and ran prime95 for about 15 minutes. It seemed to be ok. I then plugged everything else in and didn’t have any issues.

And then the issues came back. But only sometimes, or so I thought. The issues were correlated with if it was raining outside or if it was humid. Could it be too much humidity causing a short-circuit?

Apple says “You should also use your Mac notebook where the relative humidity is between 0% and 95% (noncondensing)” and the ambient temperature was in-spec. Just to make sure, I turned on my air conditioning at full blast to lower the temperature and humidity. But that just made things worse.

Could it be that the air was too dry? I tried to increase the air conditioner’s temperature but that didn’t help.

But what I did notice was that the issues correlated with having the air conditioner on, which was correlated with humid conditions (because it was hot, so I turned on the AC.)

At this point I suspected an electrical issue. I contacted my landlord to send out an electrician. A service personnel came out and suggested I get an AVR UPS (auto-voltage regulator.)

These are fancy UPSes that automatically adjust the voltage to a desired level if it dips below a threshold (in my case, 120V.) This was because I noticed that the lights browned out sometimes, and he suspected my technical issues might be because of the voltage dips causing issues.

So, I got one. And I was pretty happy, because it was beeping a lot (which meant it was detecting issues and fixing them.)

It was beeping too much, actually. Beeping during calls, beeping during lunch, beeping during trying to sleep. It just kind of snuck up on me, drinking a glass of water and then…beeeep! Thankfully, I was just drinking water.

I decided to silence the beeps as they were getting distracting and waking me up in the night. I checked the stats and it fixed at least 20 power incidents in only a few days. Wow, it’s working pretty hard I thought.

Unfortunately, after a few more days, the issues reappeared.

Grounding

The next order of business was to check if there was a grounding issue, according to my research. The “ground” is a path to earth to direct excess or unwanted electrical energy safely away. If there is an issue with the ground, then the unwanted electricity might not be dissipating (I probably butchered this term, I’m not an electrician, but if you have any suggestions on how to improve this let me know.)

A quick check with a $10.00 outlet tester confirmed that there was a grounding issue. The tester confirmed that there was no ground.

Aside: At this point, my laptop was bordering on unusable, even on battery power. Random shutdowns and the screen would turn black. Many events appeared in the Event Log, related to APIC power issues. I bought a new laptop in the meantime.

Armed with my new knowledge, I asked my landlord to send in an electrician this time. They asked lots of questions, and then recommended that my air conditioner stay on a different outlet because there were two circuits. I tried this for a few days, but the issues still reappeared.

I asked them to come again, and this time they installed a completely new outlet and the outlet was correctly grounded using my water heater as ground, according to my outlet tester.

At last, although one laptop was down, there were no more issues. No issues with strange colors on my monitor, no spicy trackpad, and no shocks.

Sometimes, debugging enters the real world.

However, changing the ZSTD_CLEVEL environment variable allows you to modify the compression level.

For example, in your GitHub actions YAML file, add the env stanza:

- name: Cache
uses: actions/cache@v2.1.6
env:
ZSTD_CLEVEL: 19
with:
[...]

…To change the compression level.

Does this actually work?

As of zstd v1.5.0 on September 20th, 2021 on the ubuntu-latest image, it works. If I cache a 100mb test file from Cachefly, GitHub says the cache size is 416162 B. Consequently, after setting the environment variable to 19, GitHub says the cache size is only 38709 B.

The higher-compressed file is 10.75 times smaller than not setting the environment variable, which means that the environment variable did have an effect.

Levels higher than 19

This doesn't work with levels higher than 19 because the –ultra flag wasn't specified. If you try to do it anyway, you'll get a message saying "Warning : compression level higher than max, reduced to 19".

It doesn't appear to be possible to manipulate the –ultra flag through the GitHub cache action, but is "possible" if you create your own caching mechanism (i.e. upload the file directly to the cache.)

If you must use GitHub's cache action and still want to compress higher than 19, you can first pre-compress the file using –ultra and level 21, then set the ZSTD_CLEVEL to 1 for the GitHub cache action. This is just a workaround as there is overhead when compressing a file twice, and in some cases could make the file slightly larger.

It is hard to determine how much data is deleted because file systems usually do not separate zeros from overwritten areas.

Both Testdisk and PhotoRec can restore deleted files (to estimate how much data has been lost but not yet reallocated); however, they can only do this if the file system's index about that file is intact. Also, it operates block by block, so while it is slow as an approximation, it's fantastic for recovering deleted files. Additionally, it takes time to recover other parts of the file, such as the filename, which, again, is valuable for recovering files, but not necessary for an estimate.

One might be concerned with how many files can be undeleted, when sending a disk over a network, and when reassigning those sectors in a server or maintenance environment.

Terminology

In this post, a “zeroed block” is a block which just contains zeros on the file system that we're using. In this case, we're mounting an XFS filesystem file, so a “zero block” is one which is pure zeros on that virtual file system; we're not concerned about the physical file system which contains that file.

A free block (defined by XFS) is a block that the file system can use to write data to. It might contain zeros, or it might have deleted file data.

Getting started

Here's an XFS filesystem, approximately 256GB in size, and contains about 80 million files. Let's find out how much data is deleted.

First, let's get the block size.

$ xfs_info xfspart.img

meta-data=xfspart.img isize=512 agcount=4, agsize=16777088 blks

= sectsz=4096 attr=2, projid32bit=1

= crc=1 finobt=1, sparse=1, rmapbt=0

= reflink=1

data = bsize=4096 blocks=67108352, imaxpct=25

= sunit=0 swidth=0 blks

naming =version 2 bsize=4096 ascii-ci=0, ftype=1

log =internal log bsize=4096 blocks=32767, version=2

= sectsz=4096 sunit=1 blks, lazy-count=1

realtime =none extsz=4096 blocks=0, rtextents=0

The data’s block size is 4096.How many free blocks are there?

$ xfs_db -r -f -c 'freesp -s' xfspart.img

from to extents blocks pct

1 1 2546152 2546152 7.49

2 3 911501 2041822 6.01

4 7 135254 623605 1.83

8 15 17445 190110 0.56

16 31 77858 2142232 6.30

32 63 14970 552755 1.63

64 127 1390 123298 0.36

128 255 750 126097 0.37

256 511 231 73905 0.22

512 1023 40 26820 0.08

1024 2047 25 35063 0.10

2048 4095 16 48302 0.14

4096 8191 20 128915 0.38

8192 16383 42 524281 1.54

16384 32767 43 1075196 3.16

32768 65535 46 2342649 6.89

65536 131071 41 3947378 11.61

131072 262143 7 1108540 3.26

262144 524287 5 2231234 6.56

524288 1048575 1 687478 2.02

4194304 8388607 3 13416832 39.47

total free extents 3705840

total free blocks 33992664

average free extent size 9.17273

About 50% of the disk is free, which means 50% of the space is reserved. According to the du command, this is correct. This means that “total free blocks” refer to blocks that are available for reallocation, not necessarily zeroed ones.

We could look at the free space extents:

$ xfs_db -r -f -c 'freesp -d' xfspart.img | wc -l

3703532

We would need to convert the agbno's into blocks, and then write a script to sample the intervals uniformly. Is there an alternative way that isn't XFS specific?

Let's start at the block level. We'll print out a few blocks from our file:

$ xfs_logprint -D -f xfspart.img

[...]

BLKNO: 1505

0 a5355de4 e325a5f5 54cd9df6 6506fc89 ca52b9ff 6ace79d7 ef9eb46f 4c7de7b6

8 426c52d0 439ace42 7dcce422 8042584f ffd90e7e 3c645921 49c1e0e1 92788e42

10 9923bc96 8ec69d91 62e7d648 1a08a9ac a496ba08 b650c6d cbf97496 486d0fee

18 caad8832 8bb79812 773960e9 6fb5aa62 dfcc301b 3ea437f2 28d47707 8caddb2b

20 c3ef6603 e7408c96 d6cf8054 33fded0e fb7ddd8d 4ebb95a0 fb23bf04 93b891c

28 fee01d7b b7e665b5 57fcb8b6 730cd271 c63304d1 abc746be 97384437 c94802f3

30 af38b5d5 888f264f a7a164d2 6665cdbb 49a600f3 368435a4 db25a652 e58cab38

38 1e55f7fb 44708a45 9be66a16 619afffd 6c7c859f 9eb8a9f0 541df29c a3f83150

40 257fe38f 9fce9fa8 be74d00 24b7e1af 7d6666af a8ffc4bb 67e2771a 64810a3e

48 2aa638d1 2e19e242 a0a2d696 3aac42b5 fb05685c bd61ec5c 4fdc662f 856f106c

50 67992c1b 8b2e7579 75adf95 61d1162 2fafd42c b737a2c3 ff23c6aa 39b1f81e

58 b6629743 e0a477bd f07c9065 3776fd0 1ddb6213 61eb5aa3 ff3f736e af1c2fd9

60 d48e6a72 c6ad2e6b cc7ca116 9a5d52d7 c25029c4 7c9f03a4 4bd71895 6bbbfb0e

68 5fc5f9a5 8ff25f73 312d3af3 659cf372 7af223ec 85cca734 1c45febe 99017b68

70 3693035 b3097096 e14e5950 56c6a840 20ca7ab3 db70f6e9 236334c4 ad2a6100

78 fc760869 6ebafb95 a496146 9724741d b00f359a c8ade661 417ef62f 10765443

[...]

Let's make sure that this maps to a physical block[0]:

$ dd if=xfspart.img bs=512 count=1 skip=1505 | od -t x4 -w32

0000000 a5355de4 e325a5f5 54cd9df6 6506fc89 ca52b9ff 6ace79d7 ef9eb46f 4c7de7b6

0000040 426c52d0 439ace42 7dcce422 8042584f ffd90e7e 3c645921 49c1e0e1 92788e42

0000100 9923bc96 8ec69d91 62e7d648 1a08a9ac a496ba08 0b650c6d cbf97496 486d0fee

0000140 caad8832 8bb79812 773960e9 6fb5aa62 dfcc301b 3ea437f2 28d47707 8caddb2b

0000200 c3ef6603 e7408c96 d6cf8054 33fded0e fb7ddd8d 4ebb95a0 fb23bf04 093b891c

0000240 fee01d7b b7e665b5 57fcb8b6 730cd271 c63304d1 abc746be 97384437 c94802f3

0000300 af38b5d5 888f264f a7a164d2 6665cdbb 49a600f3 368435a4 db25a652 e58cab38

0000340 1e55f7fb 44708a45 9be66a16 619afffd 6c7c859f 9eb8a9f0 541df29c a3f83150

0000400 257fe38f 9fce9fa8 0be74d00 24b7e1af 7d6666af a8ffc4bb 67e2771a 64810a3e

0000440 2aa638d1 2e19e242 a0a2d696 3aac42b5 fb05685c bd61ec5c 4fdc662f 856f106c

0000500 67992c1b 8b2e7579 075adf95 061d1162 2fafd42c b737a2c3 ff23c6aa 39b1f81e

0000540 b6629743 e0a477bd f07c9065 03776fd0 1ddb6213 61eb5aa3 ff3f736e af1c2fd9

0000600 d48e6a72 c6ad2e6b cc7ca116 9a5d52d7 c25029c4 7c9f03a4 4bd71895 6bbbfb0e

0000640 5fc5f9a5 8ff25f73 312d3af3 659cf372 7af223ec 85cca734 1c45febe 99017b68

0000700 03693035 b3097096 e14e5950 56c6a840 20ca7ab3 db70f6e9 236334c4 ad2a6100

0000740 fc760869 6ebafb95 0a496146 9724741d b00f359a c8ade661 417ef62f 10765443

Looks like we're on the right track.

We can print out if a block is empty via checking if the hex representation of it is just zeros. We're temporarily using the 512 byte block size (for inodes) so that we can cross-check our work with xfs_db:

$ dd if=xfspart.img bs=512 count=1 skip=1505 status=none | hexdump -v -e '/4 "%02x"' | grep -c '^[0]\*$'

0

Let's manually go to a block we know is free (block 15) and see if grep returns 1.

$ dd if=xfspart.img bs=512 count=1 skip=15 status=none | hexdump -v -e '/4 "%02x"' | grep -c '^[0]\*$'

1

As we want to total all of the blocks that are zeros, we return one if there is a match, and zero otherwise.

Recall that we have 67108352 blocks in total, so for a conf. interval of 99% and a margin of error of 1%, we have to sample 16637 blocks. The sample size is approximately 68.15MB.

While the amount of data isn't particularly large for an HDD, random accesses slow down the data transfer. Using CrystalDiskMark, we measure the performance of random 4KiB reads. Q1T1 (one queue, one thread to simulate how the data will be accessed) indicates we can read at 0.41MB/s.

As a best case scenario, this will take about a minute and 37 seconds. Since we're running it through WSL2, performance might be impacted further, however.

Let's use 4096 as our data block size, and sample 16637 random blocks:

$ for i in $(shuf -i0-67108351 -n16637 | sort -n); do dd if=xfspart.img bs=4096 count=1 skip=$i status=none | hexdump -v -e '/4 "%02x"' | grep -c '^[0]\*$'; done | pv -l -s 16637 | awk '{s+=$1} END {printf "%.0f", s}'

4759

We don't need PV, but it gives us a progress bar and ETA. To make reading more sequential, I'm sorting the sectors in increasing order.

It took two minutes and 46 seconds, which was not close to what we expected. As well as running WSL2, I had a lot of apps and backup software running, which could have caused the estimation to be off.

So, out of 16637 blocks, 4759 were zeros, which is about 28.60% of zero blocks or 78.61GB.

Since zero blocks are free space[1], then we can subtract them from the reported free space to get the non-zeroed out blocks. We know that 33992664/67108352 blocks or 50.653% is reported as free by XFS from our xfs_info command earlier.

So, reported free space - zero blocks = 50.653% - 28.60% = 22.053% of the disk is non-zeroed blocks.

This means that 22.053% of 67108352 blocks is about 60.62GB of deleted files. So, our free space is composed of 60.62GB of deleted files and 78.61GB of zeroed blocks, or 139.23GB of free space total.

Is this accurate?

Checking our work

Cross checking with zeroed out filesystem

I happened to have a copy of the exact filesystem, except that I've zeroed out the free space, defragged it, then re-zeroed out the free space again. Let's rerun the script on that filesystem.

We get 8493, which means that out of 16637 blocks, 8493 were free, or about 51.04%.

So, reported free space - zero blocks = 50.653% - 51.04% = -0.387% of our disk contains deleted files.

Why is it negative? Well, it's in our 1% margin of error and 99% confidence interval. For all intents and purposes, this is essentially zero.

We're expecting the zeroed out volume to not have any recoverable deleted files so this matches up with our expectations.

What does PhotoRec recover?

After running PhotoRec on the partition, it recovered 45GB of data (“du” apparent size.) This is the same order of magnitude of what we estimated, however, it is less because we are working at a file system block level rather than a file level. This means that even if a file occupied only one byte of data, we would have counted it as 512 bytes.

Compressing both disks

Compressing the zeroed out disk using zstd with default options gives us a ratio of 16.84 (15.2GB), while the non-zeroed out disk is 6.26 (40.9GB); each disk was 256GB on-disk.

The difference between these two compressed files is 25.7GB. We have 60.62GB of deleted files, so let's go to the first disk.

The non-zeroed out disk had 128GB of data, plus 60.62GB of deleted data which gives 188.62GB of data to compress. Consequently, the zeroed out disk only had 128GB of data to compress.

The adjusted compression ratio for the compressed disk is 188.62GB/40.9GB is 4.61, while the zeroed out disk is 128GB/15.2GB = 8.421.

This doesn't tell us a whole lot, but confirms that the disk with deleted files doesn't compress as well as the one with just zeros. Since the deleted file was a subsection of the files (potentially gzipped), it makes sense that the compression ratio would be smaller.

Areas for improvement

• 1.2x-2x faster 4k random reads using multiple threads from CrystalDiskMark (Q32T16.) Unclear if this will be a performance improvement under WSL2.

• The sampling method also includes the internal log, which is 32767 blocks or about 17MB. We probably shouldn't include the log in our sampling because that's not considered a deleted file. While the internal log is very small compared to our entire disk, larger disks might have a larger log and could affect sampling.

• I zeroed the disk using cat /dev/zero \> zeros then deleted the file; this means that each block doesn't have a special header because if it did, then that header should be intact for all of the blocks. Since deleting the file was instantaneous, a header couldn't have been on every block, and this would have caused our free block count to be close to zero. This means that a single large file with just zeros that was deleted would not count towards the deleted files.

• Consequently, if a file has zero blocks that align with the filesystem's blocks, then that part of the file won't be counted towards the deleted data total.

• fstrim could improve the performance of discarding unused blocks https://man7.org/linux/man-pages/man8/fstrim.8.html. It has a minimum option that could be used in conjunction with the free space extents histogram discussed earlier to determine a tradeoff between how many unused blocks will be discarded compared to how long sequential writes will take.

Areas to explore in the future

A drive does not have to contain pure zeros for it to have been wiped. It is possible for a drive to contain any amount of random data, a random pattern, or a predictable pattern, as long as it is not sensitive information. The method described in this post can't be adapted to work in the general case because it works on blocks of content rather than finding sensitive information.

While these are interesting questions, they are outside the scope of this post.

Conclusion

Using block-level analysis, we determined how many unused blocks there are compared to those that have already been discarded. We were able to estimate how much data had been deleted but had not yet been reallocated.

We found out that our first filesystem had about 60.62GB ± 1% of deleted data, and the identical filesystem which was zeroed out and defragged had about 0.26MB ± 1% of deleted data. While I don't know for certain whether these are actually deleted files, the two measurements are different enough to warrant a conclusion.

While the apparent size of the recovered files by “du” was about 45GB, it gives us the same order of magnitude as to how many files were deleted. Additionally, it is unclear if photorec does not recover files that don't have a valid file header (as it uses file types to determine what files to recover.)

This measurement is important because of:

• Potential security concerns (deleted files can be salvaged.) This method should not be used in and of itself for checking if any deleted files remain because it is stochastic, but it is helpful in preemptively prioritizing what drives have many deleted files or large quantities of deleted data.
• Determining when or what threshold to zero out a disk's free space. Doing it too often is wasteful, but not often enough means the underlying storage may be occupying unnecessary data. This is especially relevant for dynamic disks.
• How compressible your disk is if you're sending it over a network. Getting the average compressibility of all files may not be a good estimator of disk compressibility because the deleted files may still take up space.

Footnotes

0: This data is from /dev/urandom.

1: In the context of this post, zero blocks are free space.

There’s two main ways that files are changed together:

• Through refactoring, renaming, or logic changes.

• Renaming a string or term.

Renaming a string or a team in a string won’t be covered by this post, and this approach isn’t that great for solving it. This is because renaming strings may encompass many files at once but only once, and so there won’t be enough data to correlate them together.

Similar research

There has been research similar to this, including [future-defective]{.ul} files and [Who Should Review My Code? A File Location-Based Code-Reviewer Recommendation Approach for Modern Code Review]{.ul}. What we need is the ability to find out what files are normally changed with other files, then flag those if they aren’t changed as part of a PR. This article [Temporal correlation in Git repositories]{.ul} shows a sample program written in PHP that correlates file paths with other file paths if they are changed together.

While that article is focused around avoiding [Shotgun Surgery]{.ul} (finding classes which are correlated together too much) it can also be used to find which files should be modified together. Sometimes it’s not possible to fully decouple files (e.g. when adding a translation string when modifying the UI.)

Getting started

Let’s experiment. Clone [Angular]{.ul}, create a directory called “commits”, cd into angular, and run this command:

while read hash; do git show --pretty='format:' --name-only "$hash" > ../commits/$RANDOM$RANDOM$RANDOM.txt; done < <(git log --oneline | cut -d " " -f 1)

What this command does (it’s from the original article) is get a list of all of the file paths that changed for each commit and create a new file with those file paths in it. The command isn’t exactly optimal; the random file names aren’t guaranteed to be perfectly random and so there might be a chance that one of the files gets overwritten. For demo purposes this is ok, as there’s over 20k files so if one gets overwritten it’s not the end of the world. I could use a counter next time.

Whipping up a quick C# program, we can group the files by the other files that they’ve changed with. I don’t want to release the source code just yet as it’s an O(N\^2) algorithm and the code is super messy. Essentially what we’re doing is reading each file with the file paths, creating a key-value pair with each path together with each other path (permutations), and then adding those pairs to a list.

With those pairs, we group by the first element and then group the values together. This is what we get:

Type = "packages/router/src/url_tree.ts" -> [0] = { File = "packages/router/src/router_state.ts", Count = 15 }, [1] = { File = "packages/router/src/shared.ts", Count = 15 }, [2] = { File = "packages/router/src/router.ts", Count = 14 }, …

Parsing the output

How to read: out of all the PRs that the file at the path packages/router/src/url_tree.ts was changed, the file at the path packages/router/src/router_state.ts was also changed in the same PR 15 times this file was changed, the file at the path packages/router/src/shared.ts was changed 15 times, etc. If we change url_tree.ts without changing router_state.ts, then there might be a bug, because router_state.ts is usually changed when url_tree.ts is changed.

Note that there are some caveats with this approach. I’m not checking when url_tree.ts changes just by itself, and I don’t know the percentage of how often router_state.ts changes in regards to the other files. Also, I’m not keeping track of renaming.

If we repeat this for all of the commits, we can get this structure for each file path. I did the first 100 files on Angular and got a 12MB JSON file; there’s certainly a better way to store the database, however.

What this means for you

What this means is that if you have a PR with file X changed, you can look it up in a dictionary whose value is a list of the other popular files that are usually changed with this file. If those files have a high probability of being changed (say, 90%), then you could warn the user that these files are usually changed together and that it was “missing” from the PR. Of course, the files aren’t usually inextricably linked but it’s good to do a double check.

Why are E2E tests ignored? Well, they’re a bit of a pain to get running in general, especially for newer users. (The command npm run test should be called npm run test-components, because it’s just running the component tests, by default.)

Just a simple E2E test that checks a page title is incredibly useful. Sadly, it does need a lot of setup to just do a simple E2E test so oftentimes it goes ignored.

Starting from scratch

Let’s make a brand new Angular project. From scratch. From ng new my-app. Let’s run npm install for good measure, then run npm run e2e.

ng new my-app

npm install

git init

git add .

git commit -m "Initial commit"

npm run e2e

You may think that Protractor is part of Angular. It’s not--it’s a separate project. This means that the command npm run e2e doesn’t work out of the box (yes, I have the Google Chrome browser installed):

\[19:33:27\] I/launcher - Running 1 instances of WebDriver

\[19:33:27\] I/direct - Using ChromeDriver directly\...

DevTools listening on ws://127.0.0.1:56681/devtools/browser/829c0ca6-bc1d-4f9c-aa8f-ba379fca9279

\[50252:41332:0309/193337.329:ERROR:device_event_log_impl.cc(214)\] \[19:33:37.329\] USB: usb_device_handle_win.cc:1056 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)

\[50252:41332:0309/193337.362:ERROR:device_event_log_impl.cc(214)\] \[19:33:37.363\] USB: usb_device_handle_win.cc:1056 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)

\[50252:41332:0309/193337.386:ERROR:device_event_log_impl.cc(214)\] \[19:33:37.386\] USB: usb_device_handle_win.cc:1056 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)

Jasmine started

workspace-project App

× should display welcome message

\- Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

internal/timers.js:549:17

jasmine-spec-reporter: unable to open \'internal/timers.js\'

Error: ENOENT: no such file or directory, open \'internal/timers.js\'

internal/timers.js:492:7

jasmine-spec-reporter: unable to open \'internal/timers.js\'

Error: ENOENT: no such file or directory, open \'internal/timers.js\'

\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*

\* Failures \*

\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*

1\) workspace-project App should display welcome message

\- Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

Executed 1 of 1 spec (1 FAILED) in 35 secs.

\[19:34:36\] I/launcher - 0 instance(s) of WebDriver still running

\[19:34:36\] I/launcher - chrome \#01 failed 1 test(s)

\[19:34:36\] I/launcher - overall: 1 failed spec(s)

\[19:34:36\] E/launcher - Process exited with error code 1

npm ERR! code 1

npm ERR! path C:\\Users\\yorke\\Desktop\\angular-tour-of-heroes-master\\angular-tour-of-heroes-master

npm ERR! command failed

npm ERR! command C:\\WINDOWS\\system32\\cmd.exe /d /s /c ng e2e

npm ERR! A complete log of this run can be found in:

npm ERR! C:\\Users\\yorke\\AppData\\Local\\npm-cache\\\_logs\\2021-03-10T00_34_37_674Z-debug.log

Side-note: ok, so in this case I was “lucky” that it failed. Turns out if you have Docker running and a few other things running, your computer might get a bit slow (surprise surprise.) This causes the test to timeout. This is still an error though. Sometimes it works, sometimes it doesn’t. On the CI though it was consistently failing, so we’ll go with that.

Hmm, ok, maybe our setup is weird. Let’s run it via GitHub actions instead (and add it manually to the build + test stage, since the first few in Google’s search results don’t contain E2E tests by default.) What happens now?

\[00:40:26\] I/launcher - Running 1 instances of WebDriver

\[00:40:26\] I/direct - Using ChromeDriver directly\...

\[00:40:29\] E/runner - Unable to start a WebDriver session.

\[00:40:29\] E/launcher - Error: WebDriverError: unknown error: Chrome failed to start: exited abnormally.

(unknown error: DevToolsActivePort file doesn\'t exist)

(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

(Driver info: chromedriver=89.0.4389.23 (61b08ee2c50024bab004e48d2b1b083cdbdac579-refs/branch-heads/4389@{\#294}),platform=Linux 5.4.0-1040-azure x86_64)

at Object.checkLegacyResponse (/home/runner/work/angular-tour-of-heroes/angular-tour-of-heroes/node_modules/selenium-webdriver/lib/error.js:546:15)

at parseHttpResponse (/home/runner/work/angular-tour-of-heroes/angular-tour-of-heroes/node_modules/selenium-webdriver/lib/http.js:509:13)

at /home/runner/work/angular-tour-of-heroes/angular-tour-of-heroes/node_modules/selenium-webdriver/lib/http.js:441:30

at processTicksAndRejections (internal/process/task_queues.js:97:5)

\[00:40:29\] E/launcher - Process exited with error code 100

npm ERR! code ELIFECYCLE

npm ERR! errno 1

npm ERR! angular-tour-of-heroes\@0.0.0 e2e: \`ng e2e\`

npm ERR! Exit status 1

npm ERR!

npm ERR! Failed at the angular-tour-of-heroes\@0.0.0 e2e script.

npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

At this point you’re likely to want to fix the error, so you’ll do one or some of the following (potentially, unless you know where I’m going with this):

Option one

Delete the E2E tests or delete the E2E test runner.

It can be tempting to snip the test that looks like it’s just checking if one equals one...

it('should display welcome message', async () => {

await page.navigateTo();

expect(await page.getTitleText()).toEqual('<%= relatedAppName %> app is running!');

});

...Just remove the bolded statement and all of the errors magically disappear, except we’re not running the E2E tests. But who cares? All it’s checking is the page title. Right?

Option two

Try to get the E2E tests working or die trying. I’m being a bit sarcastic, but I’ll work through the problem as if I sequentially Googled each error message. It is surprisingly difficult.

Search Google for “Error: WebDriverError: unknown error: Chrome failed to start: exited abnormally.” then…

• Click on [https://github.com/angular/webdriver-manager/issues/444]{.ul}, but that doesn’t go anywhere

• Click on [https://github.com/actions/virtual-environments/issues/41]{.ul}, which goes to [https://github.com/actions/virtual-environments/issues/9]{.ul}, which suggests to do “apt-get install chromium-chromedriver” first, so you squeeze in a “apt-get update && apt-get install chromium-chromedriver” before the npm build

• That gives you the same error.

• You go to [https://github.com/heroku/heroku-buildpack-google-chrome/issues/56]{.ul} which has seven ways to try to fix it with varying degrees of success.

• You then search “E2E google chrome github actions”.

• The first five results are unrelated to anything to do with github actions.

• You then find [https://stackoverflow.com/questions/63651059/]{.ul} which says to run the same command that you tried except prefixed with sudo.

• You then copy-and-paste the exact 12-line YAML part/blob of text of the E2E test from the answer into your workflow.

• It doesn’t work because it says permission denied.

• You try to prefix every apt-get with sudo.

• You get an error like “E: This command can only be used by root.”

• At this point you may delete the E2E test and just forget it. Or you could keep going and wonder if E2E tests were designed to be run.

• You find [https://askubuntu.com/questions/845534/this-command-can-only-be-used-by-root]{.ul} and then prefix “apt-key” with sudo

• Try it again.

• You get a new error, “/home/runner/work/_temp/878817f4-7ca2-4e31-8c02-092beb2da616.sh: line 5: /etc/apt/sources.list.d/google.list: Permission denied”

• You add sudo to wget and to echo just in case.

• Try again.

• Get the same error.

• Oops. Shouldn’t have added sudo in front of echo.

• Try again.

• Still with me? Want to give up now?

• Oh no, same error.

• You do some more searching and find out that >> isn’t as root. You then use “sudo tee -a” (because >> doesn’t append as root). Try again.

• The same error again this time around, “Error: WebDriverError: unknown error: Chrome failed to start: exited abnormally.”

• Somehow you add this to your protractor.conf.js:

chromeOptions: {
args: ['--headless']
}

All of a sudden it works!

\[20:31:04\] W/element - more than one element found for locator By(css selector, app-root .content span) - the first result will be used

workspace-project App

√ should display welcome message

Executed 1 of 1 spec SUCCESS in 2 secs.

\[20:31:05\] I/launcher - 0 instance(s) of WebDriver still running

\[20:31:05\] I/launcher - chrome \#01 passed

Recap and final script

Let’s recap what we did. Here’s part of the original script, modified a little and commented a lot.

# Update our apt repos, then install wget. You don’t need wget (you could use curl); it doesn’t matter though; you just need to download a signing key.

sudo apt-get update

sudo apt-get install -y wget

# To get Google Chrome, you have to add the PPA for it. Google Chrome isn’t in the default Ubuntu repo. You can run apt-get install wget because wget is in the repos.

wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -

echo "deb http://dl.google.com/linux/chrome/deb/ stable main" | sudo tee -a /etc/apt/sources.list.d/google.list

# After adding in a repo, you have to tell apt to fetch the manifests. This includes Google Chrome.

sudo apt-get -y update

# Install Chrome.

sudo apt-get install -y google-chrome-stable

# “Run” Google Chrome, but with the version flag. This produces a version with a lot of verbose output. We just care about the first line and we can use a regex to extract just the version number.

VERSION=`google-chrome --version | egrep -o '[0-9]+.[0-9]+' | head -1'

The script goes on to install webdriver-manager and a chrome package. Turns out that we don’t need them as it “uses Chromedriver directly.” It might have been for older Chrome versions. The test passes perfectly fine without them.

Just this isn’t enough. We have to tell Protractor and Chrome that we’re running in headless mode, which means that we can’t run a UI. We also need to update Protractor. Let’s add,

capabilities: {
browserName: 'chrome',
chromeOptions: {
args: ['--headless']
}
},

[To protractor.conf.js]{.ul}. Pay attention to the “args” that we’ve added. We’re adding these arguments because Chrome doesn’t know we’re running in a container. This means it tries to set up a UI and such and it can’t do it because there isn’t a UI.

Why don’t we need --no-sandbox? Some of the older issues mention it, but it seems there were some changes to chrome which allows chrome to run with the sandboxing enabled within a container. This stemmed from an [error on Travis CI]{.ul} about “The SUID sandbox helper binary was found, but is not configured correctly”. Using --no-sandbox is a quick band aid patch. The correct way is through [setting the owner of the binary to root and the permissions correctly.]{.ul}

You don’t need --disable-dev-shm-usage anymore either. You do need --headless though, as Chrome doesn’t know we’re on a CI.

Now, --no-sandbox, --disable-dev-shm-usage and some other flags may be required for older versions of Chrome. Keep that in mind while testing.

Why are E2E tests important?

Maybe this should have been the first section. If we were to snip our first E2E test that appeared as though it wasn’t doing anything, we’d be missing out on a lot of testing opportunities. The first test isn’t a tautology. This point two fold:

• A single E2E test causes the afterEach function to run, which fails the tests if the console contains any errors. The afterEach function will not run if there are no tests. The console can contain errors (and still pass the component tests.) Try adding some bad code to the main.po.ts file. The component tests will run fine, but your app will just be a blank page. npm run test isn’t enough to check if your app works.

• “Breaks in” E2E testing. If you want to write another E2E test, go for it! You know that everything is in a somewhat sane state because you’re able to run a single test and it works fine. Granted, there might be other problems down the line if you’re working with flakier tests but otherwise the dependencies are installed.

What if I want to test with extensions, including Chrome Web Store ones?

Chrome with the --headless flag doesn’t support extensions yet. You [have to use xvfb]{.ul}, which is a virtual frame buffer. You may also want to pass in the “-a” flag if you’re [running xvfb-run]{.ul} with multiple concurrent instances on the same machine.

It’ll want a path to your extension, so [you’ll have to download it first]{.ul} to your computer from the Chrome Web Store, then add it to your Git repo.