Having access to multiple parallel CPU cores isn't a new thing by any means, people have been programming in parallel for half a century now, but recent years we've found ourselves at an inflection point. Moore's law is dying, beefy single cores are no longer keeping up. Modern computers come with multiple CPU cores, so exploiting parallel compute is more important than ever. Given how long it's been an area of research we can naturally expect that effective tools have taken root and that synchronizing threads is trivial now right...?

Unfortunately this has not been my experience, and I'm willing to bet it hasn't been yours either. Managing shared state across threads is hard, and the most commonly used tools: mutexes and semaphores, simply haven't evolved much since their inception.

The words that follow will dig into the problems inherent to mutexes and synchronizing shared mutable state. Afterwards we'll look into other avenues which should prove more helpful.

The Problem with Shared State

Let's begin by crafting a simple software system which needs synchronization in the first place.

I'll present a commonly used example: the task of managing bank account balances correctly in spite of parallel transfer requests.

Of course real banks don't store all their account balances in RAM, so I'll hope that the reader can apply the concepts from this pedagogical example to a their own domain as necessary, it serves as a stand-in for any sufficiently complex system which requires ad-hoc synchronization of arbitrary data between multiple threads.

Here's some golang'ish pseudo-code (please don't try to actually compile it) for a simple bank account and the operations upon it. I'm focused on the synchronization problems here, so forgive me for skipping the double-entry accounting, input validation, and other real-world complexities.

struct Account {
  balance int,
}

// Deposit money into an account
func (a *Account) deposit(amount int) {
  a.balance += amount
}

// Withdraw money from an account, or return false if there are insufficient funds
func (a *Account) withdraw(amount int) bool {
  if (a.balance <= amount) {
    return false
  } else {
    balance -= amount
    return true
  }
}

Great! This defines our Account type and some methods for withdrawing and depositing money into such an account. Now let's add a function to transfer money between accounts:

func transfer(from *Account, to *Account, amount int) bool {
  if (from.withdraw(amount)) {
    to.deposit(amount)
    return true
  } else {
    return false
  }
}

Looks good, but now what happens when we start handling multiple requests concurrently?

struct TransferRequest {
  from *Account,
  to *Account,
  amount int,
}

func main() {
  // loop forever, accepting transfer requests and processing them in goroutines
  for {
    req := acceptTransferRequest()
    go transfer(req.from, req.to, req.amount)
  }
}

Things may work well in your tests if you're (un)lucky, and might even work well in production for a while, but sooner or later you're going to lose track of money and have some confused and angry customers.

Do you see why? This brings us to our first synchronization problem to solve, Data Races.

Data races

Most programming languages are imperative with mutable data structures [citation needed], so passing pointers to multiple threads leads to shared mutable data, and shared mutable data necessarily causes data races.

A data race occurs any time two threads access the same memory location concurrently and non-deterministically when at least one of the accesses is a write. When a data race is present two runs of the same code with the same state may non-deterministically have a different result.

We're passing accounts by reference here, so multiple threads have access to modify the same account. With multiple transfer go-routines running on the same account, each could be paused by the scheduler at nearly any point during its execution. This means that even within this simple example we've already introduced a data race. Take another look at the withdraw function, I'll point it out:

// Withdraw money from an account, or return false if there are insufficient funds
func (a *Account) withdraw(amount int) bool {
  hasFunds := a.balance >= amount 
  // HERE! The scheduler could pause execution here and switch to another thread
  if (hasFunds) {
    balance -= amount
    return true
  } else {
    return false
  }
}

If two threads are withdrawing $100 from Alice's account, which only has $150 in it, it's possible that thread 1 checks the balance, sees there's enough money, then gets paused by the scheduler. Thread 2 runs, checks the balance, also sees there's enough money, then withdraws $100. When thread 1 later resumes execution after the check it withdraws its $100 too, Alice's account ends up with a negative balance of -$50, which is invalid even though we had validation!

This sort of concurrency error is particularly insidious because the original withdraw method is perfectly reasonable, idiomatic, and correct in a single-threaded program; however when we decide to add concurrency at a completely different place in the system we've introduced a bug deep within existing previously correct code. The idea that a perfectly normal evolution from a single-threaded to a multi-threaded program can introduce critical system-breaking bugs in completely unrelated code without so much as a warning is quite frankly completely unacceptable. As a craftsman I expect better from my tools.

Okay, but now that we've lost thousands if not millions of dollars, how do we fix this?

Traditional knowledge points us towards Mutexes.

Mutexes

Okay, we've encountered a problem with our shared mutable state, the traditional approach to solving these problems is to enforce exclusive access to the shared data in so-called "critical sections". Mutexes are so-named because they provide mutual exclusion, meaning only a single thread may access a given virtual resource at a time.

Here's how we can edit our program to fix the data race problems using a mutex:

struct Account {
  mutex Mutex,
  balance int,
}

func (a *Account) deposit(amount int) {
  a.mutex.lock()
  defer a.mutex.unlock()
  a.balance += amount
}

func (a *Account) withdraw(amount int) bool {
  a.mutex.lock()
  defer a.mutex.unlock()
  hasFunds := a.balance >= amount 
  if (hasFunds) {
    balance -= amount
    return true
  } else {
    return false
  }
}

Now every Account has a mutex on it, which acts as an exclusive lock.

It's much like a bathroom key in a busy restaurant. When you want to use the bathroom, you take the key, there's only one key available for each bathroom, so while you've got hold of it nobody else can use that bathroom. Now you're free to do your business, then you return the key to the hook on the wall for the next person.

Unlike a bathroom key however, mutexes are only conceptual locks, not real locks, and as such they operate on the honor system.

If the programmer forgets to lock the mutex the system won't stop them from accessing the data anyways, and even then there's no actual link between the data being locked and the lock itself, we need to trust the programmers to both understand and respect the agreement. A risky prospect on both counts.

In this case, we've addressed the data-race within withdraw and deposit by using mutexes, but we've still got a problem within the transfer function.

What happens if a thread is pre-empted between the calls to withdraw and deposit while running the transfer function? It's possible that money will been withdrawn from an account, but won't have yet been deposited in the other. This is an inconsistent state of the system, the money has temporarily disappeared, existing only in the operating memory of a thread, but not visible in any externally observable state. This can (and will) result in very strange behaviour.

As a concrete way to observe the strangeness let's write a report function which prints out all account balances:

func report() {
    for _, account := range accounts {
        account.mutex.lock()
        fmt.Println(account.balance)
        account.mutex.unlock()
    }
}

If we run a report while transfers are ongoing we'll likely see that the count of the total amount of money that exists within the system is incorrect, and changes from report to report, which should be impossible in a closed system like this! This inconsistency occurs even if we obtain the locks for each individual account before checking the balance.

In larger systems this sort of inconsistency problem can cause flaws in even simple logic, since choices may be made against inconsistent system states. The root of this issue is that the transfer function requires holding multiple independent locks, but they're not grouped in any way into an atomic operation.

Composing Critical Sections

We need some way to make the entire transfer operation atomic, at least from the perspective of other threads who are respecting our mutexes.

Okay, well no problem, we can just lock both accounts, right?

func transfer(from *Account, to *Account, amount int) bool {
  from.mutex.lock()
  to.mutex.lock()
  defer from.mutex.unlock()
  defer to.mutex.unlock()

  if (from.withdraw(amount)) {
    to.deposit(amount)
    return true
  } else {
    return false
  }
}

I'm sure some readers have already seen a problem here, but have you seen two problems here?

The first is obvious when you point it out, remember that withdraw and deposit also lock the mutex on the account, so we're trying to acquire the same lock twice in the same thread.

transfer won't even begin to run in this state, it will block forever inside withdraw when it tries to lock the from.mutex for the second time.

Some systems, like re-entrant locks and Java's synchronized keyword do some additional book-keeping which allow a single thread to lock the same mutex multiple times, so using a re-entrant lock here would solve this particular problem. However other systems, like golang, avoid providing re-entrant locks on a matter of principle.

So what can we do? I suppose we'll need to pull the locks out of withdraw and deposit so we can lock them in transfer instead.

func (a *Account) deposit(amount int) {
  a.balance += amount
}

func (a *Account) withdraw(amount int) bool {
  hasFunds := a.balance >= amount 
  if (hasFunds) {
    balance -= amount
    return true
  } else {
    return false
  }
}

func transfer(from *Account, to *Account, amount int) bool {
  from.mutex.lock()
  to.mutex.lock()
  defer from.mutex.unlock()
  defer to.mutex.unlock()

  if (from.withdraw(amount)) {
    to.deposit(amount)
    return true
  } else {
    return false
  }
}

Ugh, a correct transfer function should conceptually just be the composition of our well encapsulated withdraw and a deposit functions, but defining it correctly has forced us to remove the locking from both withdraw and deposit, making both of them less safe to use. It has placed the burden of locking on the caller (without any system-maintained guarantees), and even worse, we now need to remember to go and add locking around every existing withdraw and deposit call in the entire codebase. Even if we try to encapsulate everything within the module and only export "safe" operations we've caused duplication since we now need synchronized and unsynchronized versions of our withdraw and deposit operations. And we'd still need to expose the mutexes if we want to allow callers to synchronize operations with other non-Account data.

What I'm getting at is that mutexes don't compose! They don't allow us to chain multiple critical sections into a single atomic unit, they force us to break encapsulation and thrust the implementation details of mutexes and locking onto the caller who shouldn't need to know the details about which invariants must be maintained deep within the implementation. Adding or removing access to synchronized variables within an operation will also necessitate adding or removing locking to every call site, and those call sites may be in a completely different application or library. This is an absolute mess.

All that sounds pretty bad, but would you believe those aren't the only problems here? It's not just composition that's broken here though, in fixing transfer to make it an atomic operation we've managed to introduce a new, extra-well-hidden deadlock bug.

Deadlocks/Livelocks

Recall that in our main loop we're accepting arbitrary transfer requests and spawning them off in goroutines. What happens in our system if we have two transfer requests, Alice is trying to Venmo Bob $25 for the beanbag chair she just bought off him, meanwhile Bob remembers he needs to Venmo Alice the $130 he owes her for Weird Al concert tickets.

If by sheer coincidence they both submit their requests at the same time, we have two transfer calls:

• transfer(aliceAccount, bobAccount, 25)
• transfer(bobAccount, aliceAccount, 130)

Each of these calls will attempt to lock their from account and then their to account. If Alice and Bob get very unlucky, the system will start the first transfer and lock Alice's account, then get paused by the scheduler. When the second transfer call comes in, it first locks Bob's account, then tries to lock Alice's account, but can't because it's already locked by the first transfer call.

This is a classic deadlock situation. Both threads will be stuck forever, and worse, both Alice and Bob's accounts will be locked until the system restarts.

This is a pretty disastrous consequence for a problem which is relatively hard to spot even in this trivially simple example. In a real system with dozens or hundreds of methods being parallelized in a combinatorial explosion of ways it's very difficult to reason about this, and can be a lot of work to ensure locks are obtained in a safe and consistent order.

Golang gets some credit here in that it does provide some runtime tools for detecting both dead-locks and data-races, which is great, but these detections only help if your tests encounter the problem; they don't prevent the problem from happening in the first place. Most languages aren't so helpful, these issues can be very difficult to track down in production systems.

Assessing the damage

What a dumpster fire we've gotten ourselves into...

While it may be no accident that the example I've engineered happens to hit all of the worst bugs at once, in my experience, given enough time and complexity these sorts of problems will crop up any system eventually. Solving them with mutexes is especially dangerous because it will seem to be an effective solution at first. Mutexes work fine in small localized use-cases, thus tempting us to use them, but as the system grows organically we stretch them too far and they fail catastrophically as the complexity of the system scales up, causing all sorts of hacky workarounds. I'm of the opinion that crossing your fingers and hoping for the best is not an adequate software-engineering strategy.

So, we've seen that architecting a correct software system using mutexes is possible, but very difficult. Every attempt we've made to fix one problem has spawned a couple more.

Here's a summary of the problems we've encountered:

• Data races causing non-determinism and logic bugs
• Lack of atomicity causing inconsistent system states
• Lack of composition causing
  • Broken encapsulation
  • Code duplication
  • Cognitive overload on callers
• Deadlocks/livelocks causing system-wide freezes
• New features may require changes to every call-site

In my opinion, we've tried to stretch mutexes beyond their limits, both in this blog post and in the industry as a whole. Mutexes work great in small, well-defined scopes where you're locking a single resource which is only ever accessed in a handful of functions in the same module, but they're too hard to wrangle in larger complex systems with many interacting components maintained by dozens or hundreds of developers. We need to evolve our tools and come up with more reliable solutions.

Cleaning up the Chaos

Thankfully, despite an over-reliance on mutexes, we as an industry have still learned a thing or two since the 1960s. Particularly I think that enforcing immutability by default goes a long way here. For many programmers this is a paradigm shift from what they're used to, which usually causes some uneasiness. Seatbelts, too, were often scorned in their early years for their restrictive nature, but over time it has become the prevailing opinion that the mild inconvenience is more than worth the provided safety.

More and more languages (Haskell, Clojure, Erlang, Gleam, Elixir, Roc, Elm, Unison, ...) are realizing this and are adopting this as core design principle. Obviously not every programmer can switch to an immutable-first language over night, but I think it would behoove most programmers to strongly consider an immutable language if parallelism is a large part of their project's workload.

Using immutable data structures immediately prevents data-races, full-stop. So stick with immutable data everywhere you can, but in a world of immutability we'll still need some way to synchronize parallel processes and for that most of these languages do still provide some form of mutable reference. It's never the default, and there's typically some additional ceremony or tracking in the type system which acts as an immediate sign-post that shared-mutable state is involved; here there be dragons.

Even better than mutable references, decades of research and industrial research have provided us with a swath battle-tested high-level concurrency patterns which are built on top of lower-level synchronization primitives like mutexes or mutable references, typically exposing much safer interfaces to the programmer.

Concurrency Patterns

Actor systems and Communicating Sequential Processes (CSP) are some of the most common concurrency orchestration patterns. Each of these operate by defining independent sub-programs which have their own isolated states which only they can access. Each actor or process receives messages from other units and can respond to them in turn. Each of these deserves a talk or blog post of their own so I won't dive too deeply into them here, but please look into them deeper if this is the first you're hearing of them.

These approaches work great for task parallelism, where there are independent processes to run, and where your parallelism needs are bounded by the number of tasks you'd like to run. As an example, I used an actor-based system when building Unison's code-syncing protocol. There was one actor responsible for loading and sending requests for code, one for receiving and unpacking code, and one for validating the hashes of received code. This system required exactly 3 workers to co-operate regardless of how many things I was syncing. Actor and CSP systems are great choices when the number of workers/tasks we need to co-ordinate is statically known, i.e. a fixed number of workers, or a pre-defined map-reduce pipeline. These patterns can scale well to many cores since each actor or process can run independently on its own core without worrying about synchronizing access to shared mutable state, and as a result can often scale to multiple machines as well.

However, there are also problems where the parallelism is dynamic or ad-hoc, meaning there could be any number of runtime-spawned concurrent actors that must co-ordinate well with each other. In those cases these systems tend to break down. I've seen consultants describe complex patterns for dynamically introducing actors, one-actor-per-resource systems, tree-based actor resource hierarchies and other complex ideas but in my opinion these systems quickly outgrow the ability of any one developer to understand and debug.

So how then do we model a system like the bank account example? Even if we were to limit the system to a fixed number of transfer-workers they'd still be concurrently accessing the same data (the bank accounts) and need some way to express atomic transfers between them, which isn't easily accomplished with actors or CSP.

What's a guy to do?

A new (old) synchronization primitive

In the vast majority of cases using a streaming system, actors or CSP is going to be most effective and understandable. However in cases where we must synchronize individual chunks of data across many workers, and require operations to affect multiple chunks of data atomically, there's only one name in town that gets the job done right.

Software Transactional Memory (STM) is a criminally under-utilized synchronization tool which solves all of the problems we've encountered so far while providing more safety, better compositionality, and cleaner abstractions. Did I mention they prevent most deadlocks and livelocks too?

To understand how STM works, think of database transactions; in a database transaction isolation provides you with a consistent view of data in spite of concurrent access. Each transaction sees an isolated view of the data, untampered by other reads and writes. After making all your reads and writes you commit the transaction. Upon commit, the transaction either succeeds completely and applies ALL the changes you made to the data snapshot, or it may result in a conflict. In cases of a conflict the transaction fails and rolls back all your changes as though nothing happened, then it can retry on the new data snapshot.

STM works in much the same way, but instead of the rows and columns in a database, transactions operate on normal in-memory data structures and variables.

To explore this technique let's convert our bank account example into Haskell so we can use STM instead of mutexes.

data Account = Account {
  -- Data that needs synchronization is stored in a 
  -- Transactional Variable, a.k.a. TVar
  balanceVar :: TVar Int
}

-- Deposit money into an account.
deposit :: Account -> Int -> STM ()
deposit Account{balanceVar} amount = do
  -- We interact with the data using TVar operations which
  -- build up an STM transaction.
  modifyTVar balanceVar (\existing -> existing + amount)

-- Withdraw money from an account
-- Everything within the `do` block
-- is part of the same transaction.
-- This guarantees a consistent view of the TVars we 
-- access and mutate.
withdraw :: Account -> Int -> STM Bool
withdraw Account{balanceVar} amount = do
  existing <- readTVar balanceVar
  if existing <= amount
    then (return False)
    else do
      writeTVar balanceVar (existing - amount)
      return True

-- Transfer money between two accounts atomically
transfer :: Account -> Account -> Int -> STM Bool
transfer from to amount = do
  -- These two individual transactions seamlessly
  -- compose into one larger transaction, guaranteeing
  -- consistency without any need to change the individual
  -- operations.
  withdrawalSuccessful <- withdraw from amount
  if successful
    then do
      deposit to amount
      return True
    else 
      return False

Let's do another lap over all the problems we had with mutexes to see how this new approach fares.

Data Races

Data races are a problem which I believe are best solved at the language level itself. As mentioned earlier, using immutable data by default simply prevents data races from existing in the first place. Since data in Haskell is all immutable by default, pre-emption can occur at any point in normal code and we know we won't get a data race.

When we need mutable data, it's made explicit by wrapping that data in TVars. The language further protects us by only allowing us to mutate these variables within transactions, which we compose into operations which are guaranteed a consistent uncorrupted view of the data.

Let's convert withdraw to use STM and our balaceVar TVar.

-- Withdraw money from an account
withdraw :: Account -> Int -> STM Bool
withdraw Account{balanceVar} amount = do
  existing <- readTVar balanceVar
  if existing <= amount
    then (return False)
    else do
      -- No data races here!
      writeTVar balanceVar (existing - amount)
      return True

We can see that the code we wrote looks very much like the original unsynchronized golang version, but while using STM it's perfectly safe from data races! Even if it the thread is pre-empted in the middle of the operation, the transaction-state is invisible to other threads until the transaction commits.

Deadlock/Livelock

STM is an optimistic concurrency system. This means that threads never block waiting for locks. Instead, each concurrent operation proceeds, possibly in parallel, on their own independent transaction log. Each transaction tracks which pieces of data it has accessed or mutated and if at commit time it is detected that some other transaction has been committed and altered data which this transaction also accessed, then the latter transaction is rolled back and is simply retried.

This arrangement is fundamentally different from a lock-based exclusive access system. In STM, you don't deal with locks at all, you simply read and write data within a transaction as necessary. Our transfer function reads and writes two different TVars, but since we're not obtaining exclusive locks to these vars, we don't need to worry about deadlock at all. If two threads happen to be running a transfer on the same TVars at the same time, whichever commits first will atomically apply its updates to both accounts and the other transaction will detect this update at commit-time and will retry against the new balances.

This can cause some contention and possibly even starvation of any single transaction if many threads are trying to update the same data at the same time, but since a conflict can only occur if some other transaction has been committed, it does still have the guarantee that the system will make progress on at least some work. In Haskell, STM transactions must be pure code, and can't do IO, so most transactions are relatively short-running and should proceed eventually. This seems like a downside, but in practice it only surfaces as a rare annoyance and can usually be worked around without too much trouble.

Composition

It may not be immediately obvious from the types if you're not used to Haskell code, but all three of withdraw, deposit, and transfer are all functions which return their results wrapped in the STM monad, which is essentially a sequence of operations which we can ask to execute in a transaction using the atomically function.

We can call out to any arbitrary methods which return something wrapped in STM and it will automatically be joined in as part of the current transaction.

Unlike our mutex setup, callers don't need to manually handle locks when callingwithdraw and deposit, nor do we need to expose special synchronized versions of these methods for things to be safe. We can define them exactly once and use that one definition either on its own or within a more complex operation like transfer without any additional work. The abstraction is leak-proof, the caller doesn't need to know which synchronized data is accessed or lock or unlock any mutexes. It simply runs the transaction and the STM system happily handles the rest for you.

Here's what it looks like to actually run our STM transactions, which we do using the atomically function:

main :: IO ()
main = do
  forever $ do
    req <- acceptTransferRequest
    -- Run each transfer on its own green-thread, in an atomic transaction.
    forkIO (atomically (transfer req.from req.to req.amount)

If we'd like to compile a report of all account balances as we did previously, we can do that too. This time however we won't get a potentially inconsistent snapshot of the system by accident, instead the type-system forces us to make an explicit choice of which behaviour we'd like.

We can either:

• Access and print each account balance individually as separate transaction which means accounts may be edited in-between transactions, leading to an inconsistent report like we saw earlier.
• Or, we can wrap the entire report into a single transaction, reading all account balances in a single transaction. This will provide a consistent snapshot of the system, but due to the optimistic transaction system, the entire transaction will be retried if any individual transfers commit and edit accounts while we're collecting the report. It's possible that if transfers are happening very frequently, the report may be retried many times before it can complete.

This is a legitimate tradeoff that the developer of the system should be forced to consider.

Here's what those two different implementations look like:

-- Inconsistent report, may see money disappear/appear
reportInconsistent :: [Account] -> IO ()
reportInconsistent accounts = do
  for_ accounts $ \Account{balanceVar} -> do
    balance <- atomically (readTVar balanceVar)
    print balance

-- Consistent report, may be retried indefinitely 
-- if transfers are happening too frequently
reportConsistent :: [Account] -> IO ()
reportConsistent accounts = do
  balances <- atomically do 
    for accounts $ \Account{balanceVar} -> do
      readTVar balanceVar
  -- Now that we've got a snapshot we can print it out
  for_ balances print

Smart Retries

One last benefit of STM which we haven't yet discussed is that it supports intelligent transaction retries based on conditions of the synchronized data itself. For instance, if we have a task to withdraw $100 from Alice's account but it only has $50 in it, the mutex-based system has no choice to but fail the withdrawal entirely and return the failure up the stack. We can wrap that call with code to try again later, but how will we know when it's reasonable to try again? This would once again require the caller to understand the implementation details, and which locks the method is accessing.

STM, instead, supports failure and retrying as a first-class concept. At any point in an STM transaction you can simply call retry, this will record every TVar that the transaction has accessed up until that point, then will abort the current transaction and will sleep until any of those TVars has been modified by some other successful transaction. This avoids busy-waiting, and allows writing some very simple and elegant code.

For example, here's a new version of our withdraw function which instead of returning a failure will simply block the current thread until sufficient funds are available, retrying only when the balance of that account is changed by some other transaction's success.

-- Withdraw money from an account, blocking until sufficient funds are available
withdraw :: Account -> Int -> STM ()
withdraw Account{balanceVar} amount = do
  existing <- readTVar balanceVar
  if existing <= amount
    then retry
    else do
      writeTVar balanceVar (existing - amount)

You typically wouldn't use this to wait for an event which may take days or weeks to occur like in this example; but it's a very elegant and efficient solution for waiting on a channel, waiting for a future to produce a result, or waiting on any other short-term condition to be met.

Here's an example utility for zipping together two STM queues. The transaction will only succeed and produce a result when a value is available on both queues, and if that's not the case, it will only bother retrying when one of the queues is modified since readTQueue calls retry internally if the queue is empty.

zipQueues :: TQueue a -> TQueue b -> STM (a, b)
zipQueues q1 q2 = do
  val1 <- readTQueue q1
  val2 <- readTQueue q2
  return (val1, val2)

Nifty!

Conclusion

We've covered a lot in this post, if there's only one thing you can take away from it, I hope that you've taken the time to consider whether mutexes with shared mutable state are providing you with utility which outweighs their inherent costs and complexities. Unless you need peak performance, you may want to think twice about using such dangerous tools. Instead, consider using a concurrency pattern like actors, CSP, streaming, or map-reduce if it matches your use-case.

If you need something which provides greater flexibility or lower-level control, Software Transactional Memory (STM) is a fantastic choice if it's available in your language of choice, though note that not all languages support it, or if they do, may not be able to provide sufficient safety guarantees due to mutable variables and data structures.

If you're starting a new project for which concurrency or parallelism is a first-class concern, consider trying out a language that supports STM properly, I can recommend Unison or Haskell as great starting points.

Hopefully you learned something 🤞! Did you know I'm currently writing a book? It's all about Lenses and Optics! It takes you all the way from beginner to optics-wizard and it's currently in early access! Consider supporting it, and more posts like this one by pledging on my Patreon page! It takes quite a bit of work to put these things together, if I managed to teach your something or even just entertain you for a minute or two maybe send a few bucks my way for a coffee? Cheers! 🍻

Last time, we explored common methods of sequencing effects into little programs. If you haven't read it yet, I'd recommend starting with that, but you can probably manage without it if you insist.

We examined Applicatives, Monads, and Selective Applicatives, and each of these systems had its own trade-offs. We dug into how all approaches exist on the spectrum between being expressive or analyzable and at the end of the post we were unfortunately left wanting something better. Monads reign supreme when it comes to expressiveness as they can express any possible programs we may want to write, but they offer essentially no ability to analyze program they represent without executing it.

On the other hand, Applicatives and Selective Applicatives offered reasonable program analysis, but are unable to express complex programs. They can't even encode programs in which downstream effects materially depend on the results of upstream effects.

These approaches are all based on the same Functor-Applicative-Monad hierarchy, in this post we'll set that aside and rebuild on an altogether different foundation to see if we can do even better.

Setting the goal posts

Before putting in the work let's think critically about the what we felt was missing from the Monad hierarchy and what we wish to gain from a new system.

Here's my wish-list:

• I want to be able to list out every effect that program might perform without executing anything.
• I want to understand the dependencies between the effects including the flow of data between them.
• I want to be able to express programs in which downstream effects can fully utilize the results of upstream effects.

Looking at these requirements, the biggest problem with the Monadic effects system is that it's far too rough-grained in how it handles the results of previous effects. We can see this by reviewing the signature of bind:

(>>=) :: Monad m => m a -> (a -> m b) -> m b

We can see that the result from the previous effect is passed to an arbitrary Haskell function whose job is to return the entire continuation of the program! This permits that function to swap out the entire rest of the program on any particular run, which I'd argue is way more power than the vast majority of reasonable programs require. This is quite frankly a dangerous amount of expressive power, what sort of programs are you writing where you can't even statically identify the possible code paths that might be taken? Even more complex flows like branching, looping and recursion can be expressed in a more structured way without resorting to this sledgehammer level of dynamism.

This tells us we have some room to constrain our programs a bit, and if we're economical about how we do it we can trade that power for the benefits we desire.

We still need to utilize these past results, but we want to avoid opening Pandora's box. That is, we must be careful not to allow the creation of new effects by running arbitrary Haskell functions at execution time. So, in order to use results without a continuation-building function like Monads use, we must meaningfully include the inputs and outputs for our effects in the structure of our effect system itself. We also know that we need to be able to chain these effects together, so we'll need some way to compose them.

If it's not obvious already, this is a great fit for the Category typeclass:

class Category k where
  id :: k a a
  (.) :: k b c -> k a b -> k a c

This already gives us a lot of what we want. Unlike Monads which bake outputs into the continuation of the program using function closures, the Category structure routes inputs and outputs explicitly as part of its structure. Unsurprisingly, it's quite a natural fit; after all, it's called Category Theory, not Monad Theory...

Rebuilding on Categories

Now let's begin to re-implement the examples from the previous post using this new Category-based effect system. In order to save some time, we're actually going to jump up the hierarchy a bit all the way to Arrows.

The Arrow class, if you're not familiar with it, looks like this:

class Category a => Arrow (a :: Type -> Type -> Type) where
  arr :: (b -> c) -> a b c
  (***) :: a b c -> a b' c' -> a (b, b') (c, c')

There are a few other methods we get for free, but this is a minimal set of methods we need to define.

Notice that it has a Category superclass, so we'll use identity and composition from there. We can leverage arr to lift pure Haskell functions into our Category structure. I know we just said we wanted to avoid arbitrary Haskell functions, but note that in this case, just like Applicatives, the function is pure, we can't determine any effects or structure of the effects within the function. No problems here.

We'll re-visit (***) in just a minute.

To get started, how about we re-implement the program we wrote using Applicative in the previous post?

I'll save you from clicking over, here's a refresher on what we did before:

import Control.Applicative (liftA3)
import Control.Monad.Writer (Writer, runWriter, tell)

class (Applicative m) => ReadWrite m where
  readLine :: m String
  writeLine :: String -> m ()

data Command
  = ReadLine
  | WriteLine String
  deriving (Show)

-- | We can implement an instance which runs a dummy interpreter that simply records the commands
-- the program wants to run, without actually executing anything for real.
instance ReadWrite (Writer [Command]) where
  readLine = tell [ReadLine] *> pure "Simulated User Input"
  writeLine msg = tell [WriteLine msg]

-- | A helper to run our program and get the list of commands it would execute
recordCommands :: Writer [Command] String -> [Command]
recordCommands w = snd (runWriter w)

-- | A simple program that greets the user.
myProgram :: (ReadWrite m) => String -> m String
myProgram greeting =
  liftA3
    (\_ name _ -> name)
    (writeLine (greeting <> ", what is your name?"))
    readLine
    (writeLine "Welcome!")

-- We can now run our program in the Writer applicative to see what it would do!
main :: IO ()
main = do
  let commands = recordCommands (myProgram "Hello")
  print commands

-- [WriteLine "Hello, what is your name?", ReadLine, WriteLine "Welcome!"]

The key aspects of this Applicative version were that we could analyze any program which required only an Applicative constraint to get the full list of sequential effects that the program would perform.

Here's the same program, but this time we'll encode the effects using Arrow constraints instead.

But first, a disclaimer: writing Arrow-based programs looks ugly, but don't worry, bear with me for a bit and we'll address that later.

Just like the Applicative version, we'll define a typeclass as the interface to our set of ReadWrite effects, but this time will assume an Arrow constraint:

import Control.Arrow
import Control.Category
import Prelude hiding (id)

class (Arrow k) => ReadWrite k where
  -- Readline has no interesting input, so we use () as input type.
  readLine :: k () String

  -- We track the inputs for the writeLine directly in the Category structure.
  writeLine :: k String ()

-- Helper for embedding a static Haskell value directly into an Arrow
constA :: (Arrow k) => b -> k a b
constA b = arr (\_ -> b)

-- | A simple program which uses a statically provided message to greet the user.
myProgram :: (ReadWrite k) => String -> k () ()
myProgram greeting =
  constA (greeting <> ", what is your name?")
    >>> writeLine
    >>> readLine
    >>> constA "Welcome!"
    >>> writeLine

Great, that should feel pretty straight-forward, it's trivial to convert sequential Applicative programs like this.

In order to run it, we still need to use the IO monad, since that's just how base does IO, but we can use the nifty Kleisli newtype wrapper which turns any monadic computation into a valid Arrow by embedding the monadic effects into the Arrow structure.

Here's how we implement the ReadWrite instance for Kleisli IO:

instance ReadWrite (Kleisli IO) where
  readLine = Kleisli $ \() -> getLine
  writeLine = Kleisli $ \msg -> putStrLn msg

run :: Kleisli IO i o -> i -> IO o
run prog i = do
  runKleisli prog i

And it runs just fine:

>>> run (myProgram "Hello") ()
Hello, what is your name?
Chris
Welcome!

Let's look a little closer at Kleisli:

newtype Kleisli m a b = Kleisli { runKleisli :: a -> m b }

Look familiar? It's just the continuation function from monadic bind hiding in there.

There's a difference though, now that arbitrary function is part of our implementation, not our interface!

This is important, because it means we can invent a different implementation of our ReadWrite interface that just tracks the effects that doesn't have to deal with arbitrary binds like this.

Let's implement a command-recorder that does exactly that.

data Command
  = ReadLine
  | WriteLine
  deriving (Show)

-- Just like the applicative we create a custom implementation of the interface which for static analysis.
-- The parameters are phantom, we won't be running anything, so we only care about
-- the structure of the effects for now.
data CommandRecorder i o = CommandRecorder [Command]

-- We need a Category instance since it's a pre-requisite for Arrow:
instance Category CommandRecorder where
  -- The identity command does nothing, so it records no commands.
  id = CommandRecorder []

  -- Composition of two CommandRecorders just collects their command lists.
  (CommandRecorder cmds2) . (CommandRecorder cmds1) = CommandRecorder (cmds1 <> cmds2)

-- Now the Arrow instance.
instance Arrow CommandRecorder where
  -- We know this function must be pure (barring errors), so we don't
  -- need to track any effects from it.
  arr _ = CommandRecorder []

  -- Don't worry about this combinator yet, we'll come back to it.
  -- For now we'll collect the effects from both sides.
  (CommandRecorder cmds1) *** (CommandRecorder cmds2) = CommandRecorder (cmds1 <> cmds2)

-- | Now implementing the ReadWrite instance is just a matter of collecting the commands
-- the program is running.
instance ReadWrite CommandRecorder where
  readLine = CommandRecorder [ReadLine]
  writeLine = CommandRecorder [WriteLine]

-- | A helper to run our program and get the list of commands it would execute
recordCommands :: CommandRecorder i o -> [Command]
recordCommands (CommandRecorder cmds) = cmds

-- | Here's a helper for printing out the effects a program will run.
analyze :: CommandRecorder i o -> IO ()
analyze prog = do
  let commands = recordCommands prog
  print commands

We can analyze our program and it'll show us which effects it will run if we were to execute it:

>>> analyze (myProgram "Hello")
[WriteLine,ReadLine,WriteLine]

Okay, we've achieved the ability to analyze and execute our program at parity with the Applicative version, but isn't it silly that we're asking the user their name and simply ignoring it? As it turns out, our Arrow interface is quantifiably more expressive: we can use results of past effects in future effects! Since we're now allowing writeLine to take it's input dynamically we no longer track the output in the structure of the command itself. This bit might seem like a step back, but if you still wanted the old version you could of course still define it: writeLineStatic :: String -> k () (). Arrows allow us the flexibility to choose which we prefer. We'll chat a bit more about this later in the article.

Here's something we couldn't do with the Applicative version, we can rewrite the program to greet the user by the name they provide. While we're at it, why not receive the greeting message as an input too?

-- | This program uses the name provided by the user in the response.
myProgram2 :: (ReadWrite k) => k String ()
myProgram2 =
  arr (\greeting -> greeting <> ", what is your name?")
    >>> writeLine
    >>> readLine
    >>> arr (\name -> "Welcome, " <> name <> "!")
    >>> writeLine

Composing arrows lets us route data from one effect to the next, and arr let's us map over values to change them just like fmap does for Functors. The structure of the effects are still statically defined, so even when routing input we can still analyze the entire program ahead of time:

>>> analyze myProgram2
[WriteLine, ReadLine, WriteLine]

>>> run myProgram2 "Hello"
Hello, what is your name?
Chris
Welcome, Chris!

Nifty!

Levelling Up

We're off to a great start, the ability to use the results of past effects is already better than we could get from Selective Applicative, without sacrificing any of the analysis capabilities we had in the Applicative version.

However, at the moment our programs are all still just linear sequences of commands. What happens if we want to route results from an earlier effect down to one far later in the program?

We need a bit more power, time to call back to that (***) we ignored earlier, and while we're at it, let's look at (&&&) too, which we get for free when we implement (***).

(***) :: Arrow k => k a b -> k c d -> k (a, c) (b, d)
(&&&) :: Arrow k => k a b -> k a c -> k a (b, c)

These operators allow us to take two independent programs in our arrow interface and compose them in parallel to one another, rather than sequentially. What parallel means is going to be up to the implementation (within the scope of the Arrow laws), but the key part is that these two sides don't depend on each other, which is distinct from the normal sequential composition we've been doing with (>>>).

With these we can write a now write a slightly more complex program which routes values around, and can forward values from earlier effects to later ones.

import UnliftIO.Directory qualified as Directory

-- The effects we'll need for this example
class (Arrow k) => FileCopy k where
  readLine :: k () String
  writeLine :: k String ()
  copyFile :: k (String, String) ()

data Command
  = ReadLine
  | WriteLine
  | CopyFile
  deriving (Show)

-- Here's the real executable implementation
instance FileCopy (Kleisli IO) where
  readLine = Kleisli $ \() -> getLine
  writeLine = Kleisli $ \msg -> putStrLn msg
  copyFile = Kleisli $ \(src, dest) -> Directory.copyFile src dest

-- Helper prompting the user for input.
prompt :: (FileCopy cat) => String -> cat a String
prompt msg =
  pureC msg
    >>> writeLine
    >>> readLine

fileCopyProgram :: (FileCopy k) => k () ()
fileCopyProgram =
  ( prompt "Select a file to copy"
      &&& prompt "Select the destination"
  )
    >>> copyFile

This program prompts the user for a source file and a destination file, then copies the source file to the destination. Notably, each prompt is independent of one another, that is, they don't have any data-dependencies on one another. But, copyFile takes two arguments, the results of each prompt. (&&&) allows us to express this.

Let's run it:

>>> run fileCopyProgram ()
Select a file to copy
ShoppingList.md
Select the destination
ShoppingList.backup

Uhh, okay so you can't see the result, but trust me it works! Kleisli's implementation of (***) just runs the left side, then the right side; but if, for other applications, you wanted real parallel execution you could write your implementation which runs each pair of parallel operations using Concurrently or something like it and your program will magically become as parallel as your data-dependencies allow! Caveat emptor, but at least having the option is nice, we don't get that from the Monadic interface where data-dependencies are hidden from us.

Now for the analysis.

We could, of course, still collect and print out the list of effects that would be run, but I'm bored of that, so let's level that up too. Now that we have both sequential and parallel composition, our programs are a tree of operations, so our analysis tools should probably follow suite.

Here's a rewrite of our CommandRecorder which tracks the whole tree of effects:

-- | We can represent the effects in our computations as a tree now.
data CommandTree eff
  = Effect eff
  | Identity
  | Composed (CommandTree eff {- >>> -}) (CommandTree eff)
  | -- (***)
    Parallel
      (CommandTree eff) -- First
      (CommandTree eff) -- Second
  deriving (Show, Eq, Ord, Functor, Traversable, Foldable)

data CommandRecorder eff i o = CommandRecorder (CommandTree eff)

instance Category (CommandRecorder eff) where
  -- The identity command does nothing, so it records no commands.
  id = CommandRecorder Identity

  -- I collapse redundant 'Identity's for clarity.
  -- The category laws make this safe to do.
  (CommandRecorder Identity) . (CommandRecorder cmds1) = CommandRecorder cmds1
  (CommandRecorder cmds2) . (CommandRecorder Identity) = CommandRecorder cmds2
  (CommandRecorder cmds2) . (CommandRecorder cmds1) = CommandRecorder (Composed cmds1 cmds2)

instance Arrow (CommandRecorder eff) where
  -- We don't bother tracking pure functions, so arr is a no-op.
  arr _f = CommandRecorder Identity

  -- Track when we fork into parallel execution paths as part of the tree.
  (CommandRecorder cmdsL) *** (CommandRecorder cmdsR) = CommandRecorder (Parallel cmdsL cmdsR)

-- | The interface implementation just tracks the commands
instance FileCopy (CommandRecorder Command) where
  readLine = CommandRecorder (Effect ReadLine)
  writeLine = CommandRecorder (Effect WriteLine)
  copyFile = CommandRecorder (Effect CopyFile)

analyze :: CommandRecorder Command i o -> IO ()
analyze prog = do
  let commands = recordCommands prog
  putStrLn $ renderCommandTree commands

Now we can build the tree of effects, let's take advantage of that and render it as a tree too!

Here's a function that renders any program tree down into a flow-chart description using the mermaid diagramming language.

Don't judge me for the implementation of my mermaid renderer... In fact, if you have a nicer one please send it to me :)

(It's not terribly important, so feel free to skip it)

diagram :: CommandRecorder Command i o -> IO ()
diagram prog = do
  let commands = recordCommands prog
  putStrLn $ commandTreeToMermaid commands

-- | A helper to render our command tree as a flow-chart style mermaid diagram.
commandTreeToMermaid :: forall eff. (Show eff) => CommandTree eff -> String
commandTreeToMermaid cmdTree =
  let preamble = "flowchart TD\n"
      (outputNodes, links) =
        renderNode cmdTree
          & flip runReaderT (["Input"] :: [String])
          & flip evalState (0 :: Int)
   in preamble
        <> unlines
          ( links
              <> ((\output -> output <> " --> Output") <$> outputNodes)
          )
  where
    newNodeId :: (MonadState Int m) => m Int
    newNodeId = do
      n <- get
      put (n + 1)
      return n
    renderNode :: CommandTree eff -> ReaderT [String] (State Int) ([String], [String])
    renderNode = \case
      Effect cmd -> do
        prev <- ask
        nodeId <- newNodeId
        let cmdLabel = show cmd
            nodeDef = show nodeId <> "[" <> cmdLabel <> "]"
            links = do
              x <- prev
              pure $ x <> (" --> " <> nodeDef)
        pure ([nodeDef], links)
      Identity -> do
        nodeId <- newNodeId
        prev <- ask
        let nodeDef = show nodeId <> ("[Identity]")
        let links = do
              x <- prev
              pure $ x <> (" --> " <> nodeDef)
        pure ([nodeDef], links)
      Composed cmds1 cmds2 -> do
        (leftIds, leftNode) <- renderNode cmds1
        (rightIds, rightNode) <- local (const leftIds) $ renderNode cmds2
        pure (rightIds, leftNode <> rightNode)
      Parallel cmds1 cmds2 -> do
        prev <- ask
        nodeId <- newNodeId
        let nodeDef = show nodeId <> ("[Parallel]")
        (leftIds, leftNode) <- local (const [nodeDef]) $ renderNode cmds1
        (rightIds, rightNode) <- local (const [nodeDef]) $ renderNode cmds2
        let thisLink = do
              x <- prev
              pure $ x <> (" --> " <> nodeDef)
            links =
              thisLink
                <> leftNode
                <> rightNode
        pure (leftIds <> rightIds, links)

Here's what the diagram output for our fileCopyProgram looks like:

>>> diagram fileCopyProgram
flowchart TD
Input --> 0[Parallel]
0[Parallel] --> 1[WriteLine]
1[WriteLine] --> 2[ReadLine]
0[Parallel] --> 3[WriteLine]
3[WriteLine] --> 4[ReadLine]
2[ReadLine] --> 5[CopyFile]
4[ReadLine] --> 5[CopyFile]
5[CopyFile] --> Output

And rendered:

Pretty cool eh?

Diagramming is just one thing you can do with our CommandTree, it's just data, you can fold over it to get all the effects, analyze which effects depend on which others, all sorts of things. This provides more clarity into what's happening than Selective's Over and Under newtypes.

This was a very simple example, but I promise you, with combinations of arr, (***) and first/second you can do any possible routing of values that you might like.

What you can't do yet, however, is to branch between possible execution paths, then run only one of them.

Let's add that.

Branching with ArrowChoice

Luckily for us, adding branching is pretty straight-forward. There's an aptly named ArrowChoice in base that we'll go ahead and implement.

ArrowChoice adds a new combinator:

(+++) :: ArrowChoice k => k a b -> k c d -> k (Either a c) (Either b d)

Similar to how (***) lets us represent two parallel and independent programs and fuse them into a single arrow which runs both, (+++) lets us introduce a conditional branch to our program, only one path will be executed based on whether the input value is a Left or a Right.

By implementing (+++) we also get the similar (|||) for free:

(|||) :: ArrowChoice k => k a c -> k b c -> k (Either a b) c

Let's add a Branch case to our CommandTree and implement ArrowChoice for our CommandRecorder.

data CommandTree eff
  = Effect eff
  | Identity
  | Composed (CommandTree eff {- >>> -}) (CommandTree eff)
  | Parallel
      (CommandTree eff) -- First
      (CommandTree eff) -- Second
  | Branch
      (CommandTree eff) -- Left
      (CommandTree eff) -- Right
  deriving (Show, Eq, Ord, Functor, Traversable, Foldable)

instance ArrowChoice (CommandRecorder eff) where
  (CommandRecorder cmds1) +++ (CommandRecorder cmds2) = CommandRecorder (Branch cmds1 cmds2)

No problem. As a reminder, here's the branching program we expressed using Selective Applicatives last time:

-- | A program using Selective effects
myProgram :: (ReadWriteDelete m) => m String
myProgram =
  let msgKind =
        Selective.matchS
          -- The list of values our program has explicit branches for.
          -- These are the values which will be used to crawl codepaths when
          -- analysing your program using `Over`.
          (Selective.cases ["friendly", "mean"])
          -- The action we run to get the input
          readLine
          -- What to do with each input
          ( \case
              "friendly" -> writeLine ("Hello! what is your name?") *> readLine
              "mean" ->
                let msg = unlines [ "Hey doofus, what do you want?"
                                  , "Too late. I deleted your hard-drive."
                                  , "How do you feel about that?"
                                  ]
                 in writeLine msg *> deleteMyHardDrive *> readLine
              -- This can't actually happen.
              _ -> error "impossible"
          )
      prompt = writeLine "Select your mood: friendly or mean"
      fallback =
        (writeLine "That was unexpected. You're an odd one aren't you?")
          <&> \() actualInput -> "Got unknown input: " <> actualInput
   in prompt
        *> Selective.branch
          msgKind
          fallback
          (pure id)

This example was always a bit forced just because of how limited Selective Applicatives are, but let's copy it over into our Arrow setup anyways.

First we'll implement ArrowChoice for our CommandRecorder.

-- Define our effects
class (Arrow k) => ReadWriteDelete k where
  readLine :: k () String

  writeLine :: k String ()

  deleteMyHardDrive :: k () ()

-- New commands for the new effects
data Command
  = ReadLine
  | WriteLine
  | DeleteMyHardDrive
  deriving (Show)

-- Track the effects
instance ReadWriteDelete CommandRecorder where
  readLine = CommandRecorder (Pure ReadLine)
  writeLine = CommandRecorder (Pure WriteLine)
  deleteMyHardDrive = CommandRecorder (Pure DeleteMyHardDrive)

-- Here's the runnable implementation
instance ReadWriteDelete (Kleisli IO) where
  readLine = Kleisli $ \() -> getLine
  writeLine = Kleisli $ \msg -> putStrLn msg
  deleteMyHardDrive = Kleisli $ \() -> putStrLn "Deleting hard drive... Just kidding!"

And here's our program which uses ArrowChoice:

branchingProgram :: (ReadWriteDelete k, ArrowChoice k) => k () ()
branchingProgram =
  pureC "Select your mood: friendly or mean"
    >>> writeLine
    >>> readLine
    >>> mapC
      ( \case
          "mean" -> Left ()
          "friendly" -> Right ()
          -- Just default to friendly
          _ -> Right ()
      )
    >>> let friendly =
              pureC "Hello! what is your name?"
                >>> writeLine
                >>> readLine
                >>> mapC (\name -> "Lovely to meet you, " <> name <> "!")
                >>> writeLine
            mean =
              pureC
                ( unlines
                    [ "Hey doofus, what do you want?",
                      "Too late. I deleted your hard-drive.",
                      "How do you feel about that?"
                    ]
                )
                >>> writeLine
                >>> deleteMyHardDrive
         in mean ||| friendly

Notice again, this version is actually more expressive than the Selective Applicative version, it actually greets the user by the name they provided, how kind.

I'll elide the edits to the mermaid renderer, Branch is very similar to the implementation of Parallel.

Let's make a mermaid chart like before:

>>> diagram branchingProgram
flowchart TD
Input --> 0[WriteLine]
0[WriteLine] --> 1[ReadLine]
1[ReadLine] --> 2[Branch]
2[Branch] --> 3[WriteLine]
3[WriteLine] --> 4[DeleteMyHardDrive]
2[Branch] --> 5[WriteLine]
5[WriteLine] --> 6[ReadLine]
6[ReadLine] --> 7[WriteLine]
4[DeleteMyHardDrive] --> Output
7[WriteLine] --> Output

See how it's now clear that the effects on one branch differ from another?

And of course we can run it just as you'd expect:

>>> run branchingProgram
Select your mood: friendly or mean
friendly
Hello! what is your name?
Joe
Lovely to meet you, Joe!

>>> run branchingProgram
Select your mood: friendly or mean
mean
Hey doofus, what do you want?
Too late. I deleted your hard-drive.
How do you feel about that?

Deleting hard drive... Just kidding!

Okay, so the syntax of that last example was starting to get pretty hairy, if only there was something like do-notation, but for arrows...

Arrow Notation

By enabling the {-# LANGUAGE Arrows #-} pragma we can use a form of do-notation with arrows. It will automatically route your inputs wherever you need them using combinators from the Arrow class and will even translate if and case statements into ArrowChoice combinators, it's very impressive.

I won't explain Arrow Notation deeply here, so go ahead and check out the GHC Manual for a more detailed look.

Here's what our branching program looks like when we translate it:

branchingProgramArrowNotation :: (ReadWriteDelete k, ArrowChoice k) => k () ()
branchingProgramArrowNotation = proc () -> do
  writeLine -< "Select your mood: friendly or mean"
  mood <- readLine -< ()
  case mood of
    "mean" -> mean -< ()
    "friendly" -> friendly -< ()
    _ -> friendly -< ()
  where
    friendly = proc () -> do
      writeLine -< "Hello! what is your name?"
      name <- readLine -< ()
      writeLine -< "Lovely to meet you, " <> name <> "!"

    mean = proc () -> do
      writeLine
        -<
          unlines
            [ "Hey doofus, what do you want?",
              "Too late. I deleted your hard-drive.",
              "How do you feel about that?"
            ]
      deleteMyHardDrive -< ()

It takes a bit of getting used to, but it's not so bad.

Here's the diagram, so we can get an idea of how it's being translated:

It's not quite as pretty, the translation introduces a lot of unnecessary calls to Parallel where it's just inserting Identity on the other side, this is perfectly valid, since the Category laws require that the Identity won't affect behaviour, but in our case it's messy and is clogging up our diagram, so let's clean it up.

The command tree we build as an intermediate step is just a value, so we can transform it to clean it up no problem.

If you derive Data and Plated for our Command and CommandTree types then we can do this with a simple transform on the tree. transform will rebuild the tree from the bottom up removing any redundant Identity nodes as it goes.

unredundify :: (Data eff) => CommandTree eff -> CommandTree eff
unredundify = transform \case
  Parallel Identity right -> right
  Parallel left Identity -> left
  Composed Identity right -> right
  Composed left Identity -> left
  other -> other

Diagramming the unredundified version looks much cleaner:

We can see here that the with multiple arms are getting collapsed into a sequence of binary branches, which is perfectly correct of course, but if you wanted to diagram it as a single branch you could rewrite the Branch constructor to have a list of options and collapse them all down with another rewrite rule. Same for Parallels of course. You can really do whatever is most useful for your use-case.

Arrow notation has its quirks, but it's still a substantial improvement over doing argument routing completely manually.

Static vs Dynamic data

It's worth a quick note on the difference between static and dynamic data with Arrows. With Applicatives, all the data needed to define an effect's behaviour was static, that is, it must be known at the time the program was constructed, though this might still be at runtime for the greater Haskell program.

With Arrows it's possible to interleave static and dynamic data, it's up to the author of the interface.

For example, if one were constructing a build-system they might have an interface like this:

class (Arrow k) => Builder k where
  dynamicReadFile :: k FilePath String
  staticReadFile :: FilePath -> k () String

dynamicReadFile takes its FilePath as a dynamic input, so we won't know which file we're going to read until execution time, however staticReadFile takes its FilePath as a static input. You pass it a single FilePath as a Haskell value when you construct the program. In this case we can embed the FilePath into the structure of the effect itself so that it's available during analysis.

While this is a bit more of an advanced use-case, it can be very useful. In the build-system case you could provide any statically known dependency files using staticReadFile and the build-system could check if those files have changed since the last run and safely replace some subtrees of the build with cached results if no dependencies in that subtree have changed.

This sort of thing takes careful thought and design, but provides a lot of flexibility which can unlock whole new programming techniques.

Folks may well have heard of Haxl, it's a Haskell library for analyzing programs and batching and caching requests to remote data sources. The implementation and interface for Haxl is moderately complex, and is limited in what it can do by the fact that it uses Monads. I'm curious how effective an Arrow-based version could be.

What's next?

We explored enough classes to enable most basic programs here. At this point you can branch, express independence between computations, and route input anywhere you need it. In case you're still hankering for a bit more expressive power we'll do a lightning quick tour of a few more classes.

There's ArrowLoop which encodes fixed-point style recursion.

class Arrow a => ArrowLoop a where
  loop :: a (b, d) (c, d) -> a b c

Interestingly, this is actually just another name for Costrong, as you can see by comparing with Costrong from the profunctors package.

If you really really need to be able to completely restructure your program on the fly you can do so using the ArrowApply class, which enables applying arbitrary runtime-created arrows.

class Arrow a => ArrowApply a where
    app :: a (a b c, b) c

This gives you the wildly expressive power to define entirely new code-paths at runtime. I'd still argue that reasonable programs that actually need to do this are pretty rare, but sometimes it's a useful shortcut to avoid some tedium. Note that if you use app, any effects within the dynamically applied arrow will be hidden from analysis, but you can still analyze the non-dynamic parts.

There are a few additional interesting classes which are strangely missing from base; but they have counterparts in profunctors. One example would be an arrow counterpart to Cochoice, which, if it existed, would look something like this:

class (Arrow k) => ArrowCochoice k where
  unright :: k (Either d a) (Either d b) -> k a b
  unleft :: k (Either a d) (Either b d) -> k a b

While the behaviour ultimately depends on the implementation, you can use this to implement things like recursive loops and while-loops, which avoids one of the more common needs for ArrowApply while preserving analysis over the contents of the loop.

There's some other good stuff in profunctors so I'd recommend just browsing around over there, (Thanks Ed). Traversing lets you apply a profunctor to elements of a Traversable container, Mapping does the same for Functors.

Anyways, you can see that most behaviours you take for granted when writing Haskell code with arbitrary functions in do-notation binds can generally be decomposed into some combination of Arrow typeclasses which accomplish the same thing. Using the principal of least-power is a good rule of thumb here. Generally you should use the lowest-power abstraction you can reasonably encode your program with, that will ensure you'll have the strongest potential for analysis.

In Summary

We've discovered that by switching from the Functor-Applicative-Monad effect system to a Category and Arrow hierarchy we can express significantly more complex and expressive programs while maintaining the ability to deeply introspect the programs we create.

We learned how we can collect additional typeclasses to gain more expressive power, and how we can implement custom instances to analyze and even diagram our programs.

Lastly we took a look at Arrow notation and how it improves the burden of syntax for writing these sorts of programs.

So, should we all abandon Monads and write everything using Arrows instead? Truthfully, I do believe they comprise a better foundation; so while the current Haskell ecosystem is all-in on Monads, if you the reader happen to be designing the effects system for a brand new functional programming language, why not give Arrows a try?

Hopefully you learned something 🤞! Did you know I'm currently writing a book? It's all about Lenses and Optics! It takes you all the way from beginner to optics-wizard and it's currently in early access! Consider supporting it, and more posts like this one by pledging on my Patreon page! It takes quite a bit of work to put these things together, if I managed to teach your something or even just entertain you for a minute or two maybe send a few bucks my way for a coffee? Cheers! 🍻

Okay, so you and I both know monads are great, they allow us to sequence effects in a structured way and are in many ways a super-power in the functional-programming toolkit. It's likely none of us would have even heard of Haskell without them.

It's my opinion, though, that monads are actually too powerful for their own good. Or to be more clear, monads are more expressive than they need to be, and that we're paying hidden costs to gain expressive power that we rarely, if ever, actually use.

In this post we'll take a look at how different approaches to effects lie on the spectrum between expressiveness and strong static analysis, and how, just like Dynamic vs Statically typed programming languages, there's a benefit to limiting the number of programs you can write by adding more structure and constraints to your effects system.

The Status Quo

A defining feature of the Monadic interface is that it allows the dynamic selection of effects based on the results of previous effects.

This is a huge boon, and is what allowed the construction of real programs in Haskell without compromising on its goals of purity and laziness. This ability is what allows us to express normal programming workflows like fetching input from a user before deciding which command to run next, or fetching IDs from the database and then resolving those IDs with subsequent database calls. This form of choice is necessary for writing most moderately complex programs.

Alas, as it turns out, this expressiveness isn't free! It exists on a spectrum. As anyone who's maintained any relatively complex JavaScript or Python codebase can tell you, the ability to do anything at any time comes at a cost of readability, perhaps more relevant to the current discussion, at the cost of static analysis.

Allow me to present, in all its glory, the Expressiveness Spectrum:

Strong Static Analysis <+---------+---------+> Embarrassingly Expressive Code

As you can clearly see, as you gain more expressive power you begin to lose the ability to know what the heck your program could possibly do when it runs.

This has fueled a good many debates among programming language connoisseurs, and it turns out that there's a similar version of the debate to be had within the realm of effect systems themselves.

In their essence, effect systems are just methods of expressing miniature programs within your programming language of choice. These mini programs can be constructed, analysed, and executed at runtime within the framework of the larger programming language, and the same Expressiveness Spectrum applies independently to them as well. That is, the more programs you allow your effect system to express, the less you can know about any individual program before you run it.

In the effect-system microcosm there are similar mini compile time and run time stages. As an example here's a simple Haskell program which constructs a chain of effects using a DSL:

-- The common way to express effects in Haskell 
-- is with a Monadic typeclass interface.
class Monad m => ReadWrite m where
  readLine :: m String
  writeLine :: String -> m ()

-- We can write a little program builder which depends on 
-- input that may only be known at runtime.
greetUser :: ReadWrite m => String -> m () 
greetUser greeting = do
  writeLine (greeting <> ", what is your name?")
  name <- readLine
  writeLine ("Hello, " <> name <> "!")

-- We can, at run time, construct a new mini-program 
-- that the world has never seen before!
mkSimpleGreeting :: ReadWrite m => IO (m ())
mkSimpleGreeting = do 
  greeting <- readFile "greeting.txt"
  pure (greetUser greeting)

In this simplified example we clearly see that we can use our host languages features arbitrarily to construct a smaller program within our ReadWrite DSL. Our simple program here just reads a line of input from the user and then greets them by name.

This is all well and good in such a simple case, however if we expand our simple ReadWrite effect slightly by adding a new effect:

class Monad m => ReadWriteDelete m where
  readLine :: m String
  writeLine :: String -> m ()
  deleteMyHardDrive :: m ()

Well now, if we're constructing or parsing programs of the ReadWriteDelete effect type at runtime, we probably want to be able to know whether or not the program we're about to run contains a call to deleteMyHardDrive before we actually run it.

We could of course simply abort execution or ignore requests to delete everything when we're running the effects in our host language, which is nice, but the fact remains that if our app is handed an arbitrary ReadWriteDelete m => m () program at runtime, there's no way to know whether or not it could possibly contain a call to deleteMyHardDrive without actually running the program, and even then, there's no way to know whether there's some other possible execution path that we missed which does call deleteMyHardDrive.

We'd really love to be able to analyse the program and all of its possible effects before we run anything at all.

The Benefits of Static Analysis

Most programmers are familiar with the benefits of static analysis when applied to regular everyday programming languages. It can catch basic errors like type-mismatches, incorrect function calls, and in some cases things like memory unsafety or race conditions.

We're typically after different kinds of benefits when analysing programs in our effect systems, but they are similarly useful!

For instance, given enough understanding of an effectful program we can perform code transformations like removing redundant calls, parallelizing independent workflows, caching results, and optimizing workflows into more efficient ones.

We can also gain useful knowledge, like creating a call graph for developers to better understand what's about to happen. Or perhaps analyzing the use of sensitive resources like the file system or network such that we can ask for approval before even beginning execution.

But as I've already mentioned, we can't do most of these techniques in a Monadic effect system. The monad interface itself makes it clear why this is the case:

class Applicative m => Monad m where
  (>>=) :: m a -> (a -> m b) -> m b
  return :: a -> m a

We can see from Bind (>>=) that in order to know which effects (m b) will be executed next, we need to first execute the previous effect (m a) and then we need the host language (Haskell) to execute an arbitrary Haskell function. There's no way at all for us to gain insight about what the results of that function might be without running it first.

Let's move a step towards the analysis side of the spectrum and talk about Applicatives...

The origin of Applicatives

Applicatives are another interface for expressing effectful operations.

As far as I can determine, the first widespread introduction of Applicatives to programming was in Applicative Programming with Effects, a 2008 paper by Conor McBride and Ross Paterson.

Take note that this paper was written after Monads were already in widespread use, and Applicatives are, by their very definition, less expressive than Monads. To be precise, Applicatives can express fewer effectful programs than Monads can. This is shown by the fact that every Monad implements the Applicative interface, but not every Applicative is a Monad.

Despite being less expressive Applicatives are still very useful. They allow us to express programs with effects that aren't valid monads, but they also provide us with the ability to better analyse which effects are part of an effectful program before running it.

Take a look at the Applicative interface:

class Functor f => Applicative f where
  pure :: a -> f a
  (<*>) :: f (a -> b) -> f a -> f b

Notice how the interface does contain an arrow f (a -> b), but this arrow can only affect the pure aspect of the computation. Unlike monadic bind, there's no way to use the a result from running effects to select or build new effects to run.

The sequence of effects is determined entirely by the host language before we start to run the effects, and thus the sequence of effects can be reliably inspected in advance.

This limitation, if you can even call it that, gives us a ton of utility in program analysis. For any given sequence of Applicative Effects we can analyse it and produce a list of all the planned effects before running any of them, then could ask the end-user for permission before running potentially harmful effects.

Let's see what this looks like for our ReadWrite effect.

import Control.Applicative (liftA3)
import Control.Monad.Writer (Writer, runWriter, tell)

-- | We only require the Applicative interface now
class (Applicative m) => ReadWrite m where
  readLine :: m String
  writeLine :: String -> m ()

data Command
  = ReadLine
  | WriteLine String
  deriving (Show)

-- | We can implement an instance which runs a dummy interpreter that simply records the commands
-- the program wants to run, without actually executing anything for real.
instance ReadWrite (Writer [Command]) where
  readLine = tell [ReadLine] *> pure "Simulated User Input"
  writeLine msg = tell [WriteLine msg]

-- | A helper to run our program and get the list of commands it would execute
recordCommands :: Writer [Command] String -> [Command]
recordCommands w = snd (runWriter w)

-- | A simple program that greets the user.
myProgram :: (ReadWrite m) => String -> m String
myProgram greeting =
  liftA3
    (\_ name _ -> name)
    (writeLine (greeting <> ", what is your name?"))
    readLine
    (writeLine "Welcome!")

-- We can now run our program in the Writer applicative to see what it would do!
main :: IO ()
main = do
  let commands = recordCommands (myProgram "Hello")
  print commands

-- [WriteLine "Hello, what is your name?",ReadLine,WriteLine "Welcome!"]

Since this interface doesn't provide us with a bind, we can't use results from readLine in a future writeLine effect, which is a bummer. It's clear that Applicatives are less expressive in this way, but we can run an analysis of a program written in the Applicative ReadWrite to see exactly which effects it will run, and which arguments each of them are provided with, before we execute anything for real.

I hope that's enough ink to convince you that it's not a simple matter of "more expressive is always better", but rather that expressiveness exists on a continuum between ease of program analysis and expressiveness.

Expressive power comes at a cost, specifically the cost of analysis.

Closer to the Sweet Spot

So clearly Applicatives are nice, but they're a pretty strong limitation and prevent us from writing a lot of useful programs. What if there was an interface somewhere on the spectrum between the two?

Selective Applicatives fit nicely between Applicatives and Monads.

If you haven't heard of them, this isn't a tutorial on Selective itself, so go read up on them here if you like.

The interface for Selective Applicatives is similar to Applicatives, but they allow us to specify a known set of branching codepaths that our program may choose between when executing. Unlike the monadic interface, these branching paths need to be known and enumerated in advance, we can't make them up on the fly while running our effects.

This interface gets us much closer to matching the level of expressiveness we actually need for everyday programming while still granting us most of the best benefits of program analysis.

Here's an example of what it looks like to analyse a ReadWriteDelete program using Selective Applicatives:

import Control.Monad.Writer
import Control.Selective as Selective
import Data.Either
import Data.Functor ((<&>))

-- We require the Selective interface now
class (Selective m) => ReadWriteDelete m where
  readLine :: m String
  writeLine :: String -> m ()
  deleteMyHardDrive :: m ()

data Command
  = ReadLine
  | WriteLine String
  | DeleteMyHardDrive
  deriving (Show)

-- | "Under" is a helper for collecting the 
-- *minimum* set of effects we might run.
instance ReadWriteDelete (Under [Command]) where
  readLine = Under [ReadLine]
  writeLine msg = Under [WriteLine msg]
  deleteMyHardDrive = Under [DeleteMyHardDrive]

-- | "Over" is a helper which collects *all* possible effects we might run.
instance ReadWriteDelete (Over [Command]) where
  readLine = Over [ReadLine]
  writeLine msg = Over [WriteLine msg]
  deleteMyHardDrive = Over [DeleteMyHardDrive]

-- | A "real" IO instance
instance ReadWriteDelete IO where
  readLine = getLine
  writeLine msg = putStrLn msg
  deleteMyHardDrive = putStrLn "Deleting hard drive... Just kidding!"

-- | A program using Selective effects
myProgram :: (ReadWriteDelete m) => m String
myProgram =
  let msgKind =
        Selective.matchS
          -- The list of values our program has explicit branches for.
          -- These are the values which will be used to crawl codepaths when
          -- analysing your program using `Over`.
          (Selective.cases ["friendly", "mean"])
          -- The action we run to get the input
          readLine
          -- What to do with each input
          ( \case
              "friendly" -> writeLine ("Hello! what is your name?") *> readLine
              "mean" -> 
                let msg = unlines [ "Hey doofus, what do you want?"
                                  , "Too late. I deleted your hard-drive."
                                  , "How do you feel about that?"
                                  ]
                 in writeLine msg *> deleteMyHardDrive *> readLine
              -- This can't actually happen.
              _ -> error "impossible"
          )
      prompt = writeLine "Select your mood: friendly or mean"
      fallback =
        (writeLine "That was unexpected. You're an odd one aren't you?")
          <&> \() actualInput -> "Got unknown input: " <> actualInput
   in prompt
        *> Selective.branch
          msgKind
          fallback
          (pure id)

allPossibleCommands :: Over [Command] x -> [Command]
allPossibleCommands (Over cmds) = cmds

minimumPossibleCommands :: Under [Command] x -> [Command]
minimumPossibleCommands (Under cmds) = cmds

runIO :: IO String
runIO = myProgram

-- | We can now run our program in the Writer applicative to see what it would do!
main :: IO ()
main = do
  let allCommands = allPossibleCommands myProgram
  let minimumCommands = minimumPossibleCommands myProgram
  putStrLn "All possible commands:"
  print allCommands
  putStrLn "Minimum possible commands:"
  print minimumCommands

-- All possible commands:
-- [ WriteLine "Select your mood: friendly or mean"
-- , ReadLine
-- , WriteLine "Hey doofus, what do you want?\nToo late. I deleted your hard-drive.\nHow do you feel about that?"
-- , DeleteMyHardDrive
-- , ReadLine
-- , WriteLine "Hello! what is your name?"
-- , ReadLine
-- , WriteLine "That was unexpected. You're an odd one aren't you?"
-- ]
--
-- Minimum possible commands:
-- [ WriteLine "Select your mood: friendly or mean"
-- , ReadLine
-- ]

Okay, so now you've read a program which uses the full power of Selective applicative to branch based on the results of previous effects.

We can branch on user input to select either a friendly or mean greeting style, so it's clearly more expressive than the Applicative version, but it's also pretty obvious that this is the clunkiest option available. It's a bit tricky to write, and is also pretty tough to read.

We can now branch on user input, but since we need to pre-configure an explicit branch for every possible input we want to handle, we can't even write a simple program which echos back whatever the user types in, or even one that greets them by name. There are clearly still some substantial limitations on which programs we can express here.

However, let's look on the bright side for a bit, similar to our approach with Applicatives we can analyse the commands our program may run. This time however, we've got branching paths in our program.

The selective interface gives us two methods to analyse our program:

• The Under newtype will let us collect the minimum possible sequence of of effects that our program will run no matter what inputs it receives.
• The Over newtype instead collects the list of all possible effects that our program could possibly encounter if it were to run through all of its branching paths.

This isn't as usful as receiving, say, a graph representing the possible execution paths, but it does give us enough information to give users a warning aobut what a program might possibly do, we can let them know that hey, I don't know exactly what will cause it, but this program has the ability to delete your hard-drive.

You can of course write additional Selective interfaces, or use the Free Selective to re-write Selective computations in order to optimize or memoize them as you wish just like you can with Applicatives.

It's clear at this point that Selectives are another good tool, but the limitations are still too severe:

• We can't use results from previous effects in future effects.
• We can't express things like loops or recursion which require effects
• Branching logic like case-statements are expressible, but very cumbersome.
• The syntax for writing programs using Selective Applicatives is a bit rough, and there's no do-notation equivalent.

In search of the true sweet spot

This isn't a solved problem yet, but don't worry, there are yet more methods of sequencing effects to explore!

It may take me another 5 years to finally finish it, but at some point we'll continue this journey and explore how we can sequence effects using the hierarchy of Category classes instead. Perhaps you've wondered why Arrows don't get more love, we'll dive into that too! We'll seek to find a more tenable middle-ground on our Expressiveness Spectrum, a place where we can analyze possible execution paths without sacrificing the ability to write the programs we need.

I hope this blog post helps others to understand that while Monads were a huge discovery to the benefit of functional programming, that we should keep looking for abstractions which are a better fit for the problems we generally face in day-to-day programming.

Hopefully you learned something 🤞! Did you know I'm currently writing a book? It's all about Lenses and Optics! It takes you all the way from beginner to optics-wizard and it's currently in early access! Consider supporting it, and more posts like this one by pledging on my Patreon page! It takes quite a bit of work to put these things together, if I managed to teach your something or even just entertain you for a minute or two maybe send a few bucks my way for a coffee? Cheers! 🍻

This one will be quick.

Imagine this, you get a report from your bug tracker:

Sophie got an error when viewing the diff after her most recent push to her contribution to the @unison/cloud project on Unison Share

(BTW, contributions are like pull requests, but for Unison code)

Okay, this is great, we have something to start with, let's go look up that contribution and see if any of the data there is suspicious.

Uhhh, okay, I know the error is related to one of Sophie's contributions, but how do I actually find it?

I know Sophie's username from the bug report, that helps, but I don't know which project she was working on, or what the contribution ID is, which branches are involved, etc. Okay no problem, our data is relational, so I can dive in and figure it out with a query:

> SELECT 
  contribution.* 
  FROM contributions AS contribution
  JOIN projects AS project 
    ON contribution.project_id = project.id
  JOIN users AS unison_user 
    ON project.owner = unison_user.id
  JOIN users AS contribution_author 
    ON contribution.author_id = contribution_author.id
  JOIN branches AS source_branch 
    ON contribution.source_branch = source_branch.id
  WHERE contribution_author.username = 'sophie'
    AND project.name = 'cloud'
    AND unison_user.username = 'unison'
  ORDER BY source_branch.updated_at DESC

-[ RECORD 1 ]--------+----------------------------------------------------
id                   | C-4567
project_id           | P-9999
contribution_number  | 21
title                | Fix bug
description          | Prevent the app from deleting the User's hard drive
status               | open
source_branch        | B-1111
target_branch        | B-2222
created_at           | 2025-05-28 13:06:09.532103+00
updated_at           | 2025-05-28 13:54:23.954913+00
author_id            | U-1234

It's not the worst query I've ever had to write out, but if you're doing this a couple times a day on a couple different tables, writing out the joins gets pretty old real fast. Especially so if you're writing it in a CLI interface where's it's a royal pain to edit the middle of a query.

Even after we get the data we get a very ID heavy view of what's going on, what's the actual project name? What are the branch names? Etc.

We can solve both of these problems by writing a bunch of joins ONCE by creating a debugging view over the table we're interested in. Something like this:

CREATE VIEW debug_contributions AS
SELECT 
  contribution.id AS contribution_id,
  contribution.project_id,
  contribution.contribution_number,
  contribution.title,
  contribution.description,
  contribution.status,
  contribution.source_branch as source_branch_id,
  source_branch.name AS source_branch_name,
  source_branch.updated_at AS source_branch_updated_at,
  contribution.target_branch as target_branch_id,
  target_branch.name AS target_branch_name,
  target_branch.updated_at AS target_branch_updated_at,
  contribution.created_at,
  contribution.updated_at,
  contribution.author_id,
  author.username AS author_username,
  author.display_name AS author_name,
  project.name AS project_name,
  '@'|| project_owner.username || '/' || project.name AS project_shorthand,
  project.owner AS project_owner_id,
  project_owner.username AS project_owner_username
FROM contributions AS contribution
JOIN projects AS project ON contribution.project_id = project.id
JOIN users AS author ON contribution.author_id = author.id
JOIN users AS project_owner ON project.owner = project_owner.id
JOIN branches AS source_branch ON contribution.source_branch = source_branch.id
JOIN branches AS target_branch ON contribution.target_branch = target_branch.id;

Okay, that's a lot to write out at once, but we never need to write that again. Now if we need to answer the same question we did above we do:

SELECT * from debug_contributions 
  WHERE author.username = 'sophie'
    AND project_shorthand = '@unison/cloud'
    ORDER BY source_branch_updated_at DESC;

Which is considerably easier on both my brain and my fingers. I also get all the information I could possibly want in the result!

You can craft one of these debug tables for whatever your needs are for each and every table you work with, and since it's just a view, it's trivial to update or delete, and doesn't take any space in the DB itself.

Obviously querying over project_shorthand = '@unison/cloud' isn't going to be able to use an index, so isn't going to be the most performant query; but these are one off queries, so it's not a concern (to me at least). If you care about that sort of thing you can leave out the computed columns so you won't have to worry about that.

Anyways, that's it, that's the whole trick. Go make some debugging views and save your future self some time.

Hopefully you learned something 🤞! Did you know I'm currently writing a book? It's all about Lenses and Optics! It takes you all the way from beginner to optics-wizard and it's currently in early access! Consider supporting it, and more posts like this one by pledging on my Patreon page! It takes quite a bit of work to put these things together, if I managed to teach your something or even just entertain you for a minute or two maybe send a few bucks my way for a coffee? Cheers! 🍻

This post will introduce a simple caching strategy, with a small twist, which depending on your app may help you not only improve performance, but might also drastically reduce the memory residency of your program.

I had originally written this post in 2022, but looks like I got busy and failed to release it, so just pretend you're reading this in 2022, okay? It was a simpler time.

In case you're wondering, we continued to optimize storage since and modern UCM uses even less memory than back in 2022 😎.

Spoiler warning, with about 80 lines of code, I was able to reduce both the memory residency and start-up times by a whopping ~95%! From 90s -> 4s startup time, and from 2.73GB -> 148MB. All of these gains were realized by tweaking our app to enforce sharing between identical objects in memory.

Case Study

I help build the Unison Language. One unique thing about the language is that programmers interact with the language through the Unison Codebase Manager (a.k.a. ucm), which is an interactive shell. Some users have started to amass larger codebases, and lately we've been noticing that the memory usage of ucm was growing to unacceptable levels.

Loading one specific codebase, which I'll use for testing throughout this article, required 2.73GB and took about 90 seconds to load from SQLite. This is far larger and slower than we'd like.

There are 2 important facets of how Unison stores code that will be important to know as we go forward, and will help you understand whether this technique might work for you.

• Unison codebases are append-only, and codebase definitions are referenced by a content-based hash.

A Unison codebase is a tree with many branches, each branch contains many definitions and also has references its history. In Unison, once a definition is added to the codebase it is immutable, this is similar to how commits work in git; commits can be built upon, and branches can change which commit they point to, but once a commit is created it cannot be changed and is uniquely identified by its hash.

• A given Unison codebase is likely to refer to subtrees of code like libraries many times across different Unison branches. E.g. most projects contain a reference to the base library.

A Unison project can pull in the libraries it depends on by simply mounting that dependency into its lib namespace. Doing so is inexpensive because in effect we simply copy the hash which refers to a given snapshot of the library, we don't need to make copies of any of the underlying code. However, when loading the codebase into memory ucm was hydrating each and every library reference into a full in-memory representation of that code. No good!

What is sharing and why do I want it?

Sharing is a very simple concept at its core: rather than having multiple copies of the same identical object in memory, we should just have one. It's dead simple if you say it like that, but there are many ways we can end up with duplicates of values in memory. For example, if I load the same codebase from SQLite several times then SQLite won't know that the object I'm loading already exists in memory and will make a whole new copy.

In a language where data is mutable by default you'll want to think long and hard about whether sharing is sensible or even possible for your use-case, but luckily for me, everything in Haskell is immutable by default so there's absolutely no reason to make copies of identical values.

There's an additional benefit to sharing beyond just saving memory: equality checks may be optimized! Some Haskell types like ByteStrings include an optimization in their Eq instance which short circuits the whole check if the two values are pointer-equal. Typically testing equality on string-like values is actually most expensive when the two strings are actually equal since the check must examine every single byte to see if any of them differ. By interning our values using a cache we can reduce these checks become a single pointer equality check rather than an expensive byte-by-byte check.

Implementation

One issue with caches like this is that they can grow to eventually consume unbounded amounts of memory, we certainly don't want every value we've ever cached to stay there forever. Haskell is a garbage collected language, so naturally the ideal situation would be for a value to live in the cache up until it is garbage collected, but how can we know that?

GHC implements weak pointers! This nifty feature allows us to do two helpful things:

1. We can attach a finalizer to the values we return from the cache, such that values will automatically evict themselves from the cache when they're no longer reachable.
2. Weak references don't prevent the value they're pointing to from being garbage collected. This means that if a value is only referenced by a weak pointer in a cache then it will still be garbage collected.

As a result, there's really no downside to this form of caching except a very small amount of compute and memory used to maintain the cache itself. Your mileage may vary, but as the numbers show, in our case this cost was very much worth it when compared to the gains.

Here's an implementation of a simple Interning Cache:

module InternCache
  ( InternCache,
    newInternCache,
    lookupCached,
    insertCached,
    intern,
    hoist,
  )
where

import Control.Monad.IO.Class (MonadIO (..))
import Data.HashMap.Strict (HashMap)
import Data.HashMap.Strict qualified as HashMap
import Data.Hashable (Hashable)
import System.Mem.Weak
import UnliftIO.STM

-- | Parameterized by the monad in which it operates, the key type, 
-- and the value type.
data InternCache m k v = InternCache
  { lookupCached :: k -> m (Maybe v),
    insertCached :: k -> v -> m ()
  }

-- | Creates an 'InternCache' which uses weak references to only 
-- keep values in the cache for as long as they're reachable by 
-- something else in the app.
--
-- This means you don't need to worry about a value not being 
-- GC'd because it's in the cache.
newInternCache :: 
  forall m k v. (MonadIO m, Hashable k) 
  => m (InternCache m k v)
newInternCache = do
  var <- newTVarIO mempty
  pure $
    InternCache
      { lookupCached = lookupCachedImpl var,
        insertCached = insertCachedImpl var
      }
  where
    lookupCachedImpl :: TVar (HashMap k (Weak v)) -> k -> m (Maybe v)
    lookupCachedImpl var ch = liftIO $ do
      cache <- readTVarIO var
      case HashMap.lookup ch cache of
        Nothing -> pure Nothing
        Just weakRef -> do
          deRefWeak weakRef

    insertCachedImpl :: TVar (HashMap k (Weak v)) -> k -> v -> m ()
    insertCachedImpl var k v = liftIO $ do
      wk <- mkWeakPtr v (Just $ removeDeadVal var k)
      atomically $ modifyTVar' var (HashMap.insert k wk)

    -- Use this as a finalizer to remove the key from the map 
    -- when its value gets GC'd
    removeDeadVal :: TVar (HashMap k (Weak v)) -> k -> IO ()
    removeDeadVal var k = liftIO do
      atomically $ modifyTVar' var (HashMap.delete k)

-- | Changing the monad in which the cache operates with a natural transformation.
hoist :: (forall x. m x -> n x) -> InternCache m k v -> InternCache n k v
hoist f (InternCache lookup' insert') =
  InternCache
    { lookupCached = f . lookup',
      insertCached = \k v -> f $ insert' k v
    }

Now you can create a cache for any values you like! You can maintain a cache within the scope of a given chunk of code, or you can make a global cache for your entire app using unsafePerformIO like this:

-- An in memory cache for interning hashes.
-- This allows us to avoid creating multiple in-memory instances of the same hash bytes;
-- but also has the benefit that equality checks for equal hashes are O(1) instead of O(n), since
-- they'll be pointer-equal.
hashCache :: (MonadIO m) => InternCache m Hash Hash
hashCache = unsafePerformIO $ hoist liftIO <$> IC.newInternCache @IO @Hash @Hash 
{-# NOINLINE hashCache #-}

And here's an example of what it looks like to use the cache in practice:

expectHash :: HashId -> Transaction Hash
expectHash h =
  -- See if we've got the value in the cache
  lookupCached hashCache h >>= \case
    Just hash -> pure hash
    Nothing -> do
      hash <-
        queryOneCol
          [sql|
              SELECT base32
              FROM hash
              WHERE id = :h
            |]
      -- Since we didn't have it in the cache, add it now
      insertCached hashCache h hash
      pure hash

For things like Hashes, the memory savings are more modest, but in the cases of entire subtrees of code the difference for us was substantial. Not only did we save memory, but we saved a ton of time re-hydrating subtrees of code from SQLite that we already had.

We can even get the benefits of a cache like this when we don't have a separate key for the value, as long as the value itself has a Hashable or Ord instance (if you swap the InternCache to use a regular Map). We can use it as its own key, this doesn't help us avoid the computational cost of creating the value, but it still gives us the memory savings:

-- | When a value is its own key, this ensures that the given value 
-- is in the cache and always returns the single canonical in-memory 
-- instance of that value, garbage collecting any others.
intern :: (Hashable k, Monad m) => InternCache m k k -> k -> m k
intern cache k = do
  mVal <- lookupCached cache k
  case mVal of
    Just v -> pure v
    Nothing -> do
      insertCached cache k k
      pure k

Conclusion

An approach like this doesn't work for every app, it's much easier to use when working with immutable values like this, but if there's a situation in your app where it makes sense I recommend giving it a try! I'll reiterate that for us, we dropped our codebase load times from 90s down to 4s, and our resting memory usage from 2.73GB down to 148MB.

Hopefully you learned something 🤞! Did you know I'm currently writing a book? It's all about Lenses and Optics! It takes you all the way from beginner to optics-wizard and it's currently in early access! Consider supporting it, and more posts like this one by pledging on my Patreon page! It takes quite a bit of work to put these things together, if I managed to teach your something or even just entertain you for a minute or two maybe send a few bucks my way for a coffee? Cheers! 🍻

This article is about a code-transformation technique I used to get 100x-300x performance improvements on a particularly slow bit of code which was loading Unison code from Postgres in Unison Share. I haven't seen it documented anywhere else, so wanted to share the trick!

It's a perennial annoyance when I'm programming that often the most readable way to write some code is also directly at odds with being performant. A lot of data has a tree structure, and so working with this data is usually most simply expressed as a series of nested function calls. Nested function calls are a reasonable approach when executing CPU-bound tasks, but in webapps we're often querying or fetching data along the way. In a nested function structure we'll naturally end up interleaving a lot of one-off data requests. In most cases these data requests will block further execution until a round-trip to the database fetches the data we need to proceed.

In Unison Share, I often need to hydrate an ID into an AST structure which represents a chunk of code, and each reference in that code will often contain some metadata or information of its own. We split off large text blobs and external code references from the AST itself, so sometimes these fetches will proceed in layers, e.g. fetch the AST, then fetch the text literals referenced in the tree, then fetch the metadata for code referenced by the tree, etc.

When hydrating a large batch of code definitions, if each definition takes N database calls, loading M definitions is NxM database round-trips, NxM query plans, and potentially NxM index or table scans! If you make a call for each text ID or external reference individually, then this scales even worse.

The technique in the post details a technique for using traversals to iteratively evolve linear, nested codepaths into similar functions which work on batches of data instead. Critically, It allows keeping all the same codepaths which allow you to keep the same nested code structure, avoiding the need to restructure the whole codebase and allowing you to easily introduce batching progressively without shipping a whole rewrite at once. It also provides a trivial mechanism for deduplicating data requests, and even allows using the exact same codepath for loading 0, 1, or many entities in a typesafe way. First a quick explanation of how I ended up in this situation.

Case study: Unison Share definition loading

I'm in charge of the Unison Share code-hosting and collaboration platform. The codebase for this webapp started its life by collecting bits and pieces of code from the UCM CLI application. UCM uses SQLite, so the first iteration was minimal rewrite which simply replaced SQLite queries with the equivalent Postgres queries, but the codepaths themselves were left largely the same.

SQLite operates in-process and loads everything from memory or disk, so for our intents and purposes in UCM it has essentially no latency. As a result, most code for loading definitions from the user's codebase in UCM was written simply and linearly, loading the data only as it is needed. E.g. we may have a method loadText :: TextId -> Sqlite.Transaction Text, and when we needed to load many text references it was perfectly reasonable to just traverse loadText over a list of IDs.

However, not all databases have the same trade-offs! In the Unison Share webapp we use Postgres, which means the database has a network call and round-trip latency for each and every query. We now pay a fixed round-trip latency cost on every query that simply wasn't a factor before. Something simple like traverse loadText textIds is now performing hundreds of sequential database calls and individual text index lookups! Postgres doesn't know anything about which query we'll run next, so it can't optimize this at all (aside from warming up caches) That's clearly not good.

To optimize for Postgres we'd much prefer to make one large database call which takes an array of a batch of TextIds and returns all the Text results in a single query, this allows Postgres to save a lot of work by finding all text values in a single scan, and means we only incur a single round-trip delay rather than one per text.

Here's a massively simplified sketch of what the original naive linear code looked like:

loadTerm :: TermReference -> Transaction (AST TermInfo Text)
loadTerm ref = do
  ast <- loadAST ref
  bitraverse loadTermInfo loadText ast

loadTermInfo :: TermReference -> Transaction TermInfo
loadTermInfo ref =
  queryOneRow [sql| SELECT name, type FROM terms WHERE ref = #{ref} |]

loadText :: TextId -> Transaction Text
loadText textId =
  queryOneColumn [sql| SELECT text FROM texts WHERE id = #{textId} |]

We really want to load all the Texts in a single query, but the TextIds aren't just sitting in a nice list, they're nested within the AST structure.

Here's some pseudocode for fetching a these as a batch:

batchLoadASTTexts :: AST TermReference TextId -> Transaction (AST TermInfo Text)
batchLoadASTTexts ast = do
  let textIds = Foldable.toList ast
  texts <- fetchTexts textIds
  for ast \textId ->
    case Map.lookup textId texts of
      Nothing -> throwError $ MissingText textId
      Just text -> pure text
  where
    fetchTexts :: [TextId] -> Transaction (Map TextId Text)
    fetchTexts textIds = do
      resolvedTexts <- queryListColumns [sql|
        SELECT id, text FROM texts WHERE id = ANY(#{toArray textIds})
      |]
      pure $ Map.fromList resolvedTexts

This solves the biggest problem, most importantly it reduces N queries down to a single batch query which is already a huge improvement! However, it is a bit of boilerplate, and we'd need to write a custom version of this for each container we want to batch load texts from.

Clever folks will realize that we actually don't care about the AST structure at all, we only need a container which is Traversable, so we can generalize over that:

batchLoadTexts :: Traversable t => t TextId -> Transaction (t Text)
batchLoadTexts textIds = do
  resolvedTexts <- fetchTexts textIds
  pure $ fmap (\textId -> case Map.lookup textId resolvedTexts of
    Nothing -> throwError $ MissingText textId
    Just text -> text) textIds
  where
    fetchTexts :: [TextId] -> Transaction (Map TextId Text)
    fetchTexts textIds = do
      resolvedTexts <- queryListColumns [sql|
        SELECT id, text FROM texts WHERE id = ANY(#{toArray textIds})
      |]
      pure $ Map.fromList resolvedTexts

This is much better, now we can use this on any form of Traversable, meaning we can now batch load from ASTs, lists, vectors, Maps, and can even just use Identity to re-use our query logic for a single ID like this:

loadText :: TextId -> Transaction Text
loadText textId = do
  Identity text <- batchLoadTexts (Identity textId)
  pure text

This approach does still require that the IDs you want to batch load are the focus of some Traversable instance. What if instead your structure contains a half-dozen different ID types, or is arranged such that it's not in the Traversable slot of your type parameters? Bitraversable can handle up to two parameters, but after that you're back to writing bespoke functions for your container types.

For instance, how would we use this technique to batch load our TermInfo from the AST's TermReferences?

-- Assume we've written these batched term and termInfo loaders:
batchLoadTexts :: Traversable t => t TextId -> Transaction (t Text)
batchLoadTermInfos :: Traversable t => t TermReference -> Transaction (t TermInfo)

loadTerm :: TermReference -> Transaction (AST TermInfo Text)
loadTerm termRef = do
  ast <- loadAST termRef
  astWithText <- batchLoadTexts ast
  ??? astWithText -- How do we load the TermInfos in here?

We're getting closer, but Traversable instances just aren't very adaptable, the relevant ID must always be in the final parameter of the type. In this case you could get by using Flip wrapper, but it's not going to be very readable and this technique doesn't scale past two parameters.

We need some way to define and compose bespoke Traversable instances for any given situation.

Custom Traversals

In its essence, the Traversable type class is just a way to easily provide a canonical implementation of traverse for a given type:

traverse :: Applicative f => (a -> f b) -> t a -> f (t b)

As it turns out, we don't need a type class in order to construct and pass functions of this type around, we can define them ourselves.

With this signature it's still requiring that the elements being traversed are the final type parameter of the container t; we need a more general version. We can use this instead:

type Traversal s t a b = Applicative f => (a -> f b) -> s -> f t

It looks very similar, but note that s and t are now concrete types of kind *, they don't take a parameter, which means we can pick any fully parameterized type we like for s and t which focus some other type a and convert or hydrate it into b.

E.g. If we want a traversal to focus the TermReferences in an AST and convert them to TermInfos, we can write:

Traversal (AST TermReference text) (AST TermInfo text) TermReference TermInfo

-- Which expands to the function type:

Applicative f => (TermReference -> f TermInfo) -> AST TermReference text -> f (AST TermInfo text)

If you've ever worked with optics or the lens library before this should be looking mighty familiar, we've just derived lens's Traversal type!

Most optics are essentially just traversals, we can write one-off traversals for any situation we might need, and can trivially compose small independent traversals together to create more complex traversals.

Let's rewrite our batch loaders to take an explicit Traversal argument.

import Control.Lens qualified as Lens
import Data.Functor.Contravariant

-- Take a traversal, then a structure 's', and replace all TextIds with Texts to
-- transform it into a 't'
batchLoadTextsOf :: Lens.Traversal s t TextId Text -> s -> Transaction t
batchLoadTextsOf traversal s = do
  let textIds = toListOf (traversalToFold traversal) s
  resolvedTexts <- fetchTexts textIds
  Lens.forOf traversal s $ \textId -> case Map.lookup textId resolvedTexts of
    Nothing -> throwError $ MissingText textId
    Just text -> pure text
  where
    fetchTexts :: [TextId] -> Transaction (Map TextId Text)
    fetchTexts textIds = do
      resolvedTexts <- queryListColumns [sql|
        SELECT id, text FROM texts WHERE id = ANY(#{toArray textIds})
      |]
      pure $ Map.fromList resolvedTexts

traversalToFold ::
  (Applicative f, Contravariant f) =>
  Lens.Traversal s t a b ->
  Lens.LensLike' f s a
traversalToFold traversal f s = phantom $ traversal (phantom . f) s

The *Of naming convention comes from the lens library. A combinator ending in Of takes an traversal as an argument.

It's a bit unfortunate that we need traversalToFold, it's just a quirk of how Traversals and Folds are implemented in the lens library, but don't worry we'll replace it with something better soon.

Now we can pass any custom traversal we like into batchLoadTexts and it will batch up the IDs and hydrate them in-place.

Let's write the AST traversals we need:

astTexts :: Traversal (AST TermReference TextId) (AST TermReference Text) TextId Text
astTexts = traverse

astTermReferences :: Traversal (AST TermReference TextId) (AST TermInfo Text) TermReference TermInfo
astTermReferences f = bitraverse f pure

Here we can just piggy-back on existing traverse and bitraverse implementations, but if you need to write your own, I included a small guide on writing your own custom Traversals with the traversal method in the lens library, go check that out.

With this, we can now batch load both the texts and term infos from an AST in one pass each.

loadTerm :: TermReference -> Transaction (AST TermInfo Text)
loadTerm termRef = do
  ast <- loadAST termRef
  astWithText <- batchLoadTextsOf astTexts ast
  hydratedAST  <- batchLoadTermInfosOf astTermReferences astWithText
  pure hydratedAST

Scaling up

Okay now we're cooking, we've reduced the number of queries per term from 1 + numTexts + numTermRefs down to a flat 3 queries per term, which is a huge improvement, but there's more to do.

What if we need to load a whole batch of asts at once? Here's a first attempt:

-- Assume these batch loaders are in scope:
batchLoadTermASTs :: Traversal s t TermReference (AST TermReference TextId) -> s -> Transaction t
batchLoadTermInfos :: Traversal s t TermReference TermInfo -> s -> Transaction t
batchLoadTexts :: Traversal s t TextId Text -> s -> Transaction t

batchLoadTerms :: Map TermReference TextId -> Transaction (Map TermReference (AST TermInfo Text))
batchLoadTerms termsMap = do
  termASTsMap <- batchLoadTermASTs traverse termsMap
  for termASTsMap \ast -> do
    astWithTexts <- batchLoadTexts astTexts ast
    hydratedAST <- batchLoadTermInfos astTermReferences astWithTexts
    pure hydratedAST

This naive approach loads the asts in a batch, but then traverses over the resulting ASTs batch loading the terms and texts: This is better than no batching at all, but we're still running queries in a loop. 2 queries for each term in the map is still O(N) queries, we can do better.

Luckily, Traversals are easily composable! We can effectively distribute the for loop into our batch calls by adding composing an additional traverse so each traversal is applied to every element of the outer map. In case you're not familiar with optics, just note that traversals compose from outer to inner from left to right, using .; it looks like this:

batchLoadTerms :: Map TermReference TextId -> Transaction (Map TermReference (AST TermInfo Text))
batchLoadTerms termsMap = do
  termASTsMap <- batchLoadTermASTs traverse termsMap
  astsMapWithTexts <- batchLoadTexts (traverse . astTexts) termASTsMap
  hydratedASTsMap <- batchLoadTermInfos (traverse . astTermReferences) astsMapWithTexts
  pure hydratedASTsMap

If you want, you can even pipeline it like so:

  batchLoadTermASTs traverse termsMap
    >>= batchLoadTexts (traverse . astTexts)
    >>= batchLoadTermInfos (traversed . astTermReferences)

It was a small change, but this performs much better at scale, we went from O(N) queries to O(1) queries, that is, we now run EXACTLY 3 queries, no matter how many terms we're loading, pretty cool. In fact, the latter two queries have no data-dependencies on each other, so you can also pipeline them if your DB supports that, but I'll leave that as an exercise (or come ask me on bluesky).

That's basically the technique, the next section will show a few tweaks which help me to use it at application scale.

Additional tips

Let's revisit the database layer where we actually make the batch query:

import Control.Lens qualified as Lens
import Data.Functor.Contravariant

-- Take a traversal, then a structure 's', and replace all TextIds with Texts to
-- transform it into a 't'
batchLoadTextsOf :: Lens.Traversal s t TextId Text -> s -> Transaction t
batchLoadTextsOf traversal s = do
  let textIds = toListOf (traversalToFold traversal) s
  resolvedTexts <- fetchTexts textIds
  Lens.forOf traversal s $ \textId -> case Map.lookup textId resolvedTexts of
    Nothing -> throwError $ MissingText textId
    Just text -> pure text
  where
    fetchTexts :: [TextId] -> Transaction (Map TextId Text)
    fetchTexts textIds = do
      resolvedTexts <- queryListColumns [sql|
        SELECT id, text FROM texts WHERE id = ANY(#{toArray textIds})
      |]
      pure $ Map.fromList resolvedTexts

traversalToFold ::
  (Applicative f, Contravariant f) =>
  Lens.Traversal s t a b ->
  Lens.LensLike' f s a
traversalToFold traversal f s = phantom $ traversal (phantom . f) s

This pattern is totally fine, but it does involve materializing and sorting a Map of all the results, which also requires an Ord instance on the database key we use. Here's an alternative approach:

import Control.Lens qualified as Lens
import Data.Functor.Contravariant
-- Take a traversal, then a structure 's', and replace all TextIds with Texts to
-- transform it into a 't'
batchLoadTextsOf :: Lens.Traversal s t TextId Text -> s -> Transaction t
batchLoadTextsOf traversal s = do
  s & unsafePartsOf traversal %%~ \textIds -> do
      let orderedIds = zip [0 :: Int32 ..] textIds
      queryListColumns [sql|
        WITH text_ids(ord, id) AS (
          SELECT * unnest(#{toArray orderedIds}) AS ids(ord, id)
        )
        SELECT texts.text 
          FROM texts JOIN text_ids ON texts.id = text_ids.id;
        ORDER BY text_ids.ord ASC
      |]

Using unsafePartsOf allows us to act on the foci of a traversal as though they were in a simple list. The unsafe bit is that it will crash if we don't return a list with the exact same number of elements, so be aware of that, but it's the same crash we'd have gotten in our old version if an ID was missing a value.

This also allows us to avoid the song-and-dance for converting the incoming traversal into a fold.

We need the ord column simply because sql doesn't guarantee any specific result order unless we specify one. This will pair up result rows piecewise with the input IDs, and so it doesn't require any Ord instance.

We can wrap unsafePartsOf with our own combinator to add a few additional features.

Here's a version which will deduplicate IDs in the input list, will skip the action if the input list is empty, and will provide a nice error with a callstack if anything goes sideways.

asListOf :: (HasCallStack, Ord a) => Traversal s t a b -> Traversal s t [a] [b]
asListOf trav f s =
  s
    & unsafePartsOf trav %%~ \case
      -- No point making a database call which will return no results
      [] -> pure []
      inputs -> do
        -- First, deduplicate the inputs as a self indexed map.
        let asMap = Map.fromList (zip inputs inputs)
        asMap
          -- Call the action with the list of deduped inputs
          & unsafePartsOf traversed f
          <&> \resultMap ->
            -- Now map the result for each input in the original list to its result value
            let resultList = mapMaybe (\k -> Map.lookup k resultMap) inputs
                aLength = length inputs
                bLength = length resultList
             in if aLength /= bLength
                  -- Better error message if our query is bad and returns the wrong number of elements.
                  then error $ "asListOf: length mismatch, expected " ++ show aLength ++ " elements, got " ++ show bLength <> " elements"
                  else resultList

Using a tool like this has caveats, it's very easy to cause runtime crashes if your query isn't written to always return the same number of results as it was given inputs, and skipping the action on empty lists could result in some confusion.

Conclusion

I've gotten a ton of use out of this technique in Unison Share, and managed to speed things up by 2 orders of magnitude. I was also able to perform a fully batched rewrite of heavily nested code without needing to re-arrange the code-graph. This was particularly useful because it allowed me to partially large portions of the codebase in smaller pieces by using batched methods with a simple id Traversal, and using simple traverse on methods you haven't rewritten yet.

You may not get such huge gains if your code isn't pessimistically linear in the first place, but this is also a nice, composable way to write batch code in the first place.

Anyways, give it a go and let me know what you think of it!

Hopefully you learned something 🤞! Did you know I'm currently writing a book? It's all about Lenses and Optics! It takes you all the way from beginner to optics-wizard and it's currently in early access! Consider supporting it, and more posts like this one by pledging on my Patreon page! It takes quite a bit of work to put these things together, if I managed to teach your something or even just entertain you for a minute or two maybe send a few bucks my way for a coffee? Cheers! 🍻

I don't know about you, but testing isn't my favourite part of software development.

It's usually the last thing standing between me and shipping a shiny new feature, and writing tests is often an annoying process with a lot of boilerplate and fighting against your system to get your app into a good start starting for the test or mocking out whichever services your app depends on.

Much ink has been spilled about how to organize your code in order to make this easier, but the fact that so many blog posts and frameworks exist for this express purpose suggests to me that we as a community of software developers haven't quite solved this issue yet.

Keep reading to see how I've solved this problem for myself by simply avoiding unit testing altogether.

An alternative testing method

When I first started at Unison Computing I was submitting my first feature when I learned there were precious few unit tests. I found it rather surprising for a codebase for a compiler for a programming language! How do you prevent regressions without unit tests?

The answer is what the Unison team has dubbed transcript tests. These are a variation on the concept of golden-file tests.

A Unison transcript is a markdown file which explains in standard what behaviour it is going to test, then intersperses code-blocks which outline the steps involved in testing that feature using a mix of Unison code and UCM commands (UCM is Unison's CLI tool). After that comes the magic trick; UCM itself can understand and run these transcript files directly and record the results of each block.

When running a transcript file with the ucm transcript command UCM produces a deterministic output file containing the result of processing each code block. Unless the behaviour of UCM has changed since the last time it was run the resulting file will always be the same.

Each block in the markdown file is either a command, which is sent to the UCM shell tool, or it represents an update to a file on the (virtual) file-system, in which case it will be typechecked against the state of the codebase.

Here's a quick example of a transcript for testing UCM's view command so you can get a feel for it.

# Testing the `view` command

First, let's write a simple definition to view:

``` unison
isZero = cases
  0 -> true
  _ -> false
```

Now we add the definition to the codebase, and view it.

``` ucm
scratch/main> update
scratch/main> view isZero
```

We run this transcript file with ucm transcript my-transcript.md which produces the my-transcript.output.md file.

Notice how compiler output is added inline, ignore the hashed names, It's because I'm skipping the step which adds names for Unison's builtins.

# Testing the `view` command

First, let's write a simple definition to view:

``` unison
isZero = cases
  0 -> true
  _ -> false
```

``` ucm :added-by-ucm
  Loading changes detected in scratch.u.

  I found and typechecked these definitions in scratch.u. If you
  do an `add` or `update`, here's how your codebase would
  change:

    ⍟ These new definitions are ok to `add`:
    
      isZero : ##Nat -> ##Boolean
```

Now we add the definition to the codebase, and view it.

``` ucm
scratch/main> update

  Done.

scratch/main> view isZero

  isZero : ##Nat -> ##Boolean
  isZero = cases
    0 -> true
    _ -> false
```

Feel free to browse through the collection of transcripts we test in CI to keep UCM working as expected.

Testing in CI

Running transcript tests in CI is pretty trivial; we discover all markdown files within our transcript directory and run them all. After the outputs have been written we can use git diff --exit-code which will then fail with a non-zero code if anything of the outputs have changed from what was committed. Conveniently, git will also report exactly what changed, and what the old output was.

This failure method allows the developer to know exactly which file has unexpected behaviour so they can easily re-run that file or recreate the state in their own codebase if they desire.

Transcript tests in other domains

I liked the transcript tests in UCM so much that when I was tasked with building out the Unison Share webapp I decided to use transcript-style testing for that too. Fast forward a few years and Unison Share is now a fully-featured package repository and code collaboration platform running in production without a single unit test.

If you're interested in how I've adapted transcript tests to work well for a webapp, I'll leave a few notes at the end of the post.

Benefits of transcript tests

Here's a shortlist of benefits I've found working with transcript tests over alternatives like unit tests.

You write a transcript using the same syntax as you'd interact with UCM itself.

This allows all your users to codify any buggy behaviour they've encountered into a deterministic transcript. Knowing exactly how to reproduce the behaviour your users are seeing is a huge boon, and having a single standardized format for accepting bug reports helps reduce a lot of the mental work that usually goes into reproducing bug reports from a variety of sources. This also means that the bug report itself can go directly into the test suite if we so desire.

All tests are written against the tool's external interface.

The tests use the same interface that the users of your software will employ, which means that internal refactors won't ever break tests unless there's a change in behaviour that's externally observable.

This has been a huge benefit for me personally. I'd often find myself hesitant to re-work code because I knew that at the end I'd be rewriting thousands of lines of tests. If you always have to rewrite your tests at the same time you've rewritten your code, how do you have any confidence that the tests still work as intended?

Updating tests is trivial

In the common case where transcripts are mismatched because some help message was altered, or perhaps the behaviour has changed but the change is intended, you don't need to rewrite any complex assertions, or mock out any new dependencies. You can simply look at the new output, and if it's reasonable you commit the changed transcript output files.

It can't be understated how convenient this is when making sweeping changes; e.g. making changes to Unison's pretty printer. We don't need to manually update test-cases, we just run the transcripts locally and commit the output if it all looks good!

Transcript changes appear in PR reviews

Since all transcript outputs are committed, any change in behaviour will show up in the PR diff in an easy-to-read form. This allows reviewers to trivially see the old and new behaviour for each relevant feature.

Transcript tests are documentation

Each transcript shows how a feature is intended to be used by end-users.

Transcripts as a collaboration tool

When I'm implementing new features in Unison Share I need to communicate the shape of a JSON API with our Frontend designer Simon. Typically I'll just write a transcript test which exercises all possible variants of the new feature, then I can just point at the transcript output as the interface for those APIs.

It's beneficial for both of us since I don't need to keep an example up-to-date for him, and he knows that the output is actually accurate since it's generated from an execution of the service itself.

Transcript testing for Webapps

I've adapted transcript testing a bit for the Unison Share webapp. I run the standard Share executable locally with its dependencies mocked out via docker-compose. I've got a SQL file which resets the database with a known set of test fixtures, then use a zsh script to reset my application state in between running each transcript.

Each transcript file is just a zsh script that interacts with the running server using a few bash functions which wrap curl commands, but save the output to json files, which serve as the transcript output.

I've also got helpers for capturing specific fields from an API call into local variables which I can then interpolate into future queries, this is handy if you need to, for example, create a project then switch it from private to public, then fetch that project via API.

Here's a small snippet from one of my transcripts for testing Unison Share's project APIs:

#!/usr/bin/env zsh

# Fail the transcript if any command fails
set -e

# Load utility functions and variables for user credentials
source "../../transcript_helpers.sh"

# Run a UCM transcript to upload some code to load in projects.
transcript_ucm transcript prelude.md

# I should be able to see the fixture project as an unauthenticated user.
fetch "$unauthenticated_user" GET project-get-simple '/users/test/projects/publictestproject'

# I should be able to create a new project as an authenticated user.
fetch "$transcripts_user" POST project-create '/users/transcripts/projects/containers' '{
    "summary": "This is my project",
    "visibility": "private",
    "tags": []
}'

fetch "$transcripts_user" GET project-list '/users/transcripts/projects'

You can see the output files generated by the full transcript in this directory.

Requirements of a good transcript testing tool

After working with two different transcript testing tools across two different apps I've got a few criteria for what makes a good transcript testing tool, if you're thinking of adding transcript tests to your app consider the following:

Transcripts should be deterministic

This is critical. Transcripts are only useful if they produce the same result on every run, on every operating system, at every time of day.

You may need to make a few changes in your app to adapt or remove randomness, at least when in the context of a transcript test.

In Share there were a lot of timestamps, random IDs, and JWTs (which contain a timestamp). The actual values of these weren't important for the tests themselves, so I solved the issue by piping the curl output through a sed script before writing to disk. The script matches timestamps, UUIDs, and JWTs and replaces them with placeholders like <TIMESTAMP>, <UUID>, and <JWT> accordingly.

A special mode in your app for transcript testing which avoids randomness can be useful, but use custom modes sparingly lest your app's behaviour differ too much during transcripts and you can't test the real thing.

I also make sure that the data returned by APIs is always sorted by something other than randomized IDs, it's a small price to pay, and reduces randomness and heisenbugs in the app as a helpful byproduct.

Transcripts should be isolated

Each individual transcript should be run in its own pristine environment. Databases should be reset to known state, if the file-system is used, it should be cleared or even better, a virtual file-system should be used.

Transcripts should be self-contained

Everything that pertains to a given test-case's state or configuration should be evident from within the transcript file itself. I've found that changes in behaviour from the file's location or name can just end up being confusing.

Difficulties working with Transcripts

Transcripts often require custom tooling

In UCM's case the transcript tooling has evolved slowly over many years, it has it's own parser, and you can even test UCM's API server by using special code blocks for that.

Share has a variety of zsh utility scripts which provide helpers for fetching endpoints using curl, and filtering output to capture data for future calls. It also has a few tools for making database calls and assertions.

Don't shy away from investing a bit of time into making transcript testing sustainable and pleasant, it will pay dividends down the road.

Intensive Setup

As opposed to unit tests which are generally pretty lightweight; transcript tests are full integration tests, and require setting up data, and sometimes executing entire flows so that we can get the system into a good state for testing each feature.

You can mitigate the setup time by testing multiple features with each transcript.

I haven't personally found transcript tests to take too much time in CI, largely because I think transcript testing tends to produce fewer tests, but of higher value than unit testing. I've seen many unit test suites bogged down by particular unit tests which generate hundreds of test cases that aren't actually providing real value. Also, any setup/teardown is going to be more costly on thousands of unit-tests as compared to dozens or hundreds of transcript tests.

Service Mocking

Since transcript tests run against the system-under-test's external interface, you won't have traditional mocking/stubbing frameworks available to you. Instead, you'll mock out the system's dependencies by specifying custom services using environment variables, or wiring things up in docker-compose.

Most systems have a setup for local development anyways, so integrating transcript tests against it has the added benefit that they'll ensure your local development setup is tested in CI, is consistent for all members of your team, and continues to work as expected.

In Summary

Hopefully this post has helped you to consider your relationship with unit tests and perhaps think about whether other testing techniques may work better for your app.

Transcript tests surely aren't ideal for all possible apps or teams, but my last few years at Unison have proven to me that tests can be more helpful, efficient, and readable than I'd previously thought possible.

Let me know how it works out for you!

Hopefully you learned something 🤞! Did you know I'm currently writing a book? It's all about Lenses and Optics! It takes you all the way from beginner to optics-wizard and it's currently in early access! Consider supporting it, and more posts like this one by pledging on my Patreon page! It takes quite a bit of work to put these things together, if I managed to teach your something or even just entertain you for a minute or two maybe send a few bucks my way for a coffee? Cheers! 🍻

New languages are coming out all the time, some experimental, some industrial, others are purpose built for a specific domain. No single language has the people-power or scope to try every cool new feature, so a critical step in designing a new language is to observe how experimental features have borne themselves out in practice.

As the saying goes, good [language designers] copy, great [language designers] steal.

If you've heard anything about the Unison Language it's not a surprise to you that it innovates in many areas. Unison very much tries to reinvent Human-Compiler interactions for the 21st century, and in that pursuit has spawned fully integrated ecosystem between the compiler, codebase-manager, language server, version control and package manager.

While some of these features are still too new to have proven their worth (but we have our fingers crossed); there are aspects that I think new languages should certainly consider as part of their designs.

A Fully Interactive and Incremental Compiler

With the modern era of language servers and programming assistants, developers greatly benefit from instant feedback on their work. With traditional batch compilers it's all too tempting to go for a coffee, or a walk, or a YouTube binge every time you kick off a big build. The context-switching induced by switching tasks while compiling wastes developer time by paging things in and out of their working memory, not to mention: it just feels bad. After the build finishes, the developer is left with a giant wall of text, sentenced to dig through a large list of compiler errors trying to find some root-cause error in the file they're working on.

Unison has a fully interactive compilation experience. The language-server is typechecking your scratch-file on every keystroke providing error feedback right in your editor, and offering helpful information via hover-hints which use your codebase and typechecking info to help you orient yourself. It can even partially typecheck the file to suggest which types or operators you may want to fill into a given slot.

Once you're happy with a chunk of code, you can check it in to the codebase and it won't be compiled again unless you want to change it, or an update is automatically propagated into it from a downstream change.

While most languages won't adopt Unison's scratch-file and codebase model; having an interactive compiler with good support for caching of already-compiled-assets is a huge boon to productivity in any language.

On the topic of the language server, Unison's language server is built directly into the compiler. This ensures we avoid the awkward disagreements between the LSP and compiler that sometimes happen in other languages. It can also help to avoid duplicate work, many languages are running the compiler independently and in their LSP at the same time without sharing any of the work between them, causing redundant work and a waste of precious resources.

Codebase API

It's the compiler's job to understand your code intimately. It knows exactly how every definition is linked together, even if you don't! In many languages it can be frustrating to know that this information exists deep within the compiler, but not having any access to it yourself!

Unison stores all your code as structured data within your codebase and exposes the ability for you to ask it useful questions about your code, exposing that precious understanding to you as a developer.

Unison allows searching by type, finding the dependencies of a definition, or inverting that relationship to finding all definitions which depend on a definition.

Via the UCM CLI you can use utilities like text.find to search only string constants, or find to search only definition names.

Some codebase data is provided via an API which is exposed from the interactive UCM compiler, allowing developers to write tooling to customize their workflow. For example, check out this VS Code plugin someone wrote to view codebase definitions in the sidebar. In other languages you'd typically need to write a scrappy Regex or re-compile the code in a subprocess in order to achieve something similar.

It doesn't have to be an API, it could be a parquet file or a SQLite database or any number of things, the important part is that a language exposes its one-true-source of information about the codebase in some structured format for third-party tools to build upon.

Smart docs

It doesn't matter how great your language's package ecosystem is if nobody can figure out how to use it! Documentation is critical for helping end users understand and use functionality in your language, but it has a fatal flaw: documentation isn't compiled and falls out of date with the code.

In Unison, docs are a data-type within the language itself. This means that docs can be generated dynamically by running Unison code! We've leveraged this ability to enable embedding typechecked runnable code examples into your docs. These examples are compiled alongside the rest of your program, so they're guaranteed to be kept up to date, and the outputs from your example code is run and updated whenever the source definitions change.

You can also write code which generates documentation based on your real application code. For example, you could write code which crawls your web-server's implementation and collects all the routes and parameters the server defines and displays them nicely as documentation.

Unison goes one step further here by providing special support for the documentation format on Unison Share, ensuring any definitions mentioned in docs and code examples are hyper-linked to make for a seamless package-browsing experience.

As an example of how far this can go, check out this awesome project by community contributor Alvaro which generates mermaid graphs in the docs representing the behaviour of simulations. The graphs are generated from the same underlying library code so they won't go out of date.

Get stealing

This subset of topics doesn't touch on Unison's ability system, continuation capturing, or code serialization so I'll probably need at least a part 2!

Hopefully you learned something 🤞! Did you know I'm currently writing a book? It's all about Lenses and Optics! It takes you all the way from beginner to optics-wizard and it's currently in early access! Consider supporting it, and more posts like this one by pledging on my Patreon page! It takes quite a bit of work to put these things together, if I managed to teach your something or even just entertain you for a minute or two maybe send a few bucks my way for a coffee? Cheers! 🍻

Hello! Today we'll be looking into type-based search, what it is, how it helps, and how to build one for the Unison programming language at production scale.

Motivating type-directed search

If you've never used a type-directed code search like Hoogle it's tough to fully understand how useful it can be. Before starting on my journey to learn Haskell I had never even thought to ask for a tool like it, now I reach for it every day.

Many languages offer some form of code search, or at the very least a package search. This allows users to find code which is relevant for the task they're trying to accomplish. Typically you'd use these searches by querying for a natural language phrase describing what you want, e.g. "Markdown Parser".

This works great for finding entire packages, but searching at the package level is often far too rough-grained for the problem at hand. If I just want to very quickly remember the name of the function which lifts a Char into a Text, I already know it's probably in the Text package, but I can save some time digging through the package by asking a search precisely for definitions of this type. Natural languages are quite imprecise, so a more specialized query-by-type language allows us to get better results faster.

If I search using google for "javascript function to group elements of a list using a predicate" I find many different functions which do some form of grouping, but none of them quite match the shape I had in mind, and I need to read through blogs, stack-overflow answers, and package documentation to determine whether the provided functions actually do what I'd like them to.

In Haskell I can instead express that question using a type! If I enter the type [a] -> (a -> Bool) -> ([a], [a]) into Hoogle I get a list of functions which match that type exactly, there are few other operations with a matching signature, but I can quickly open those definitions on Hackage and determine that partition is exactly what I was looking for.

Hopefully this helps to convince you on the utility of a type-directed search, though it does raise a question: if type-directed search is so useful, why isn't it more ubiquitous?

Here are a few possible reasons this could be:

• Some languages lack a sufficiently sophisticated type-system with which to express useful queries
• Some languages don't have a centralized package repository
• Indexing every function ever written in your language can be computationally expensive
• It's not immediately obvious how to implement such a system

Read on and I'll do what my best to help with the latter limitation.

Establishing our goals

Before we start building anything, we should nail-down what our search problem actually is.

Here are a few goals we had for the Unison type-directed search:

• It should be able to find functions based on a partial type-signature
• The names given to type variables shouldn't matter.
• The ordering of arguments to a function shouldn't matter.
• It should be fast
• It should should scale
• It should be good...

The last criterion is a bit subjective of course, but you know it when you see it.

The method

It's easy to imagine search methods which match some of the required characteristics. E.g. we can imagine iterating through every definition and running the typechecker to see if the query unifies with the definition's signature, but this would be far too slow, and wouldn't allow partial matches or mismatched argument orders.

Alternatively we could perform a plain-text search over rendered type signatures, but this would be very imprecise and would break our requirement that type variable names are unimportant.

Investigating prior art, Neil Mitchell's excellent Hoogle uses a linear scan over a set of pre-built function-fingerprints for whittling down potential matches. The level of speed accomplished with this method is quite impressive!

In our case, Unison Share, the code-hosting platform and package manager for Unison is backed by a Postgres database where all the code is stored. I investigated a few different Postgres index variants and landed on a GIN (Generalized inverted index).

If you're unfamiliar with GIN indexes the gist of it is that it allows us to quickly find rows which are associated with any given combination of search tokens. They're typically useful when implementing full-text searches, for instance we may choose to index a text document like the following:

postgres=# select to_tsvector('And what is the use of a book without pictures or conversations?');
                      to_tsvector
-------------------------------------------------------
 'book':8 'convers':12 'pictur':10 'use':5 'without':9
(1 row)

The generated lexemes represent a fingerprint of the text file which can be used to quickly and efficiently determine a subset of stored documents which can then be filtered using other more precise methods. So for instance we could search for book & pictur to very efficiently find all documents which contain at least one word that tokenizes as book AND any word that tokenizes as pictur.

I won't go too in-depth here on how GIN indexes work as you can consult the excellent Postgres documentation if you'd like a deeper dive into that area.

Although our problem isn't exactly full-text-search, we can leverage GIN into something similar to search type signatures by a set of attributes.

The attributes we want to search for can be distilled from our requirements; we need to know which types are mentioned in the signature, and we need some way to normalize type variables and argument position.

Let's come up with a way to tokenize type signatures into the attributes we care about.

Computing Tokens for type signature search

Mentions of concrete types

If the user metnions a concrete type in their query, we'll need to find all type signatures which mention it.

Consider the following signature: Text.take : Nat -> Text -> Text

We can boil down the info here into the following data:

• A type called Nat is mentioned once, and it does NOT appear in the return type of the function.
• A type called Text is mentioned twice, and it does appear in the return type of the function.

There really aren't any rules on how to represent lexemes in a GIN index, it's really just a set of string tokens. Earlier we saw how Postgres used an English language tokenizer to distill down the essence of a block of text into a set of tokens; we can just as easily devise our own token format for the information we care about.

Here's the format I went with for our search tokens: <token-kind>,<number-of-occurrences>,<name|hash|variable-id>

So for the mentions of Nat in Text.take's signature we can build the token: mn,1,Nat.. It starts with the token's kind (mn for Mention by Name), which prevents conflicts between tokens even though they'll all be stored in the same tsvector column. Next I include the number of times it's mentioned in the signature followed by it's fully qualified name with the path reversed.

In this case Nat is a single segment, but if the type were named data.JSON.Array it would be encoded as Array.JSON.data.,

Why? Postgres allows us to do prefix matches over tokens in GIN indexes. This allows us to search for matches for any valid suffix of the query's path, e.g. mn,1,Array.*, mn,1,Array.JSON.* or mn,1,Array.JSON.data.* would all match a single mention of this type.

Users don't always know all of the arguments of a function they're looking for, so we'd love for partial type matches to still return results. This also helps us to start searching for and displaying potentially relevant results while the user is still typing out their query.

For instance Nat -> Text should still find Text.take, so to facilitate that, when we have more than a single mention we make a separate token for each of the 1..n mentions. E.g. in Text.take : Nat -> Text -> Text we'd store both mn,1,Text AND mn,2,Text in our set of tokens.

We can't perform arithmetic in our GIN lookup, so this method is a workaround which allows us to find any type where the number of mentions is greater than or equal to the number of mentions in the query.

Type mentions by hash

This is Unison after all, so if there's a specific type you care about but you don't care what the particular package has named that type, or if there's even a specific version of a type you care about, you can search for it by hash: E.g. #abcdef -> #ghijk. This will tokenize into mh,1,#abcdef and mh,1,#ghijk. Similar to name mentions this allows us to search using only a prefix of the actual hash.

Handling return types

Although we don't care about the order of arguments to a given function, the return-type is a very high value piece of information. We can add additional tokens to track every type which is mentioned in the return type of a function by simply adding an additional token with an r in the 'mentions' place, e.g. mn,r,Text

We'll use this later to improve the scoring of returned results, and may in the future allow performing more advanced searches like "Show me all functions which produce a value of this type", a.k.a. functions which return that type but don't accept it as an argument, or perhaps "Show me all handlers of this ability", which corresponds to all functions which accept that ability as an argument but don't return it, e.g. 'mn,1,Stream' & (! 'mn,r,Stream').

A note on higher-kinded types and abilities like Map Text Nat and a -> {Exception} b, we simply treat each of these as its own concrete type mention. The system could be expanded to include more token types for each of these, but one has to be wary of an explosion in the number of generated tokens and in initial testing the search seems to work quite well despite no special treatment.

Mentions of type variables

Concrete types are covered, but what about type variables? Consider the type signature: const: b -> a -> b.

This type contains a and b which are type variables. The names of type variables are not important on their own, you can rename any type variable to anything you like as long as you consider its scope and rename all the mentions of the same variable within its scope.

To normalize the names of type variables I assign each variable a numerical ID instead. In this example we may choose to assign b the number 1 and a the number 2. However, we have to be careful because we also want to be indifferent with regard to argument order. A search for a -> b -> b should still find const! if we assigned a to 1 and b to 2 according to the order of their appearance we wouldn't have a match.

To fix this issue we can simply sort the type variables according to their number of occurrences, so in this example a has fewer occurrences than b, so it gets the lower variable ID.

This means that both a -> b -> b and b -> a -> b will tokenize to the same set of tokens: v,1,1 for a, and v,1,2, v,2,2, and v,r,2 for b.

Parsing the search query

We could require that all queries are properly formed type-signatures, but that's quite restrictive and we'd much rather allow the user to be a bit sloppy in their search.

To that end I wrote a custom version of our type-parser that is extremely lax in what it accepts, it will attempt to determine the arity and return type of the query, but will also happily accept just a list of type names. Searching for Nat Text Text and Nat -> Text -> Text are both valid queries, but the latter will return better results since we have information about both the arity of the desired function and the return type. Once we've parsed the query we can convert it into the same set of tokens we generated from the type signatures in the codebase.

Performing the search

After we've indexed all the code in our system (in Unison this takes only a few minutes) we can start searching!

For Unison's search I've opted to require that each occurrence in the query MUST be present in each match, however for better partial type-signature support I do include results which are missing specified return types, but will rank them lower than results with matching return types in the results.

Other criteria used to score matches include: * Types with an arity closer to the user's query are ranked higher * How complex the type signature is, types with more tokens are ranked lower. * We give a slight boost to some core projects, e.g. Unison's standard library base will show up higher in search results if they match. * You can include a text search along with your type search to further filter results, e.g. map (a -> b) -> [a] -> [b] will prefer finding definitions with map somewhere in the name. * Queries can include a specific user or project to search within to further filter results, e.g. @unison/cloud Remote

Summary

I hope that helps shed some light on how it all works, and perhaps will help others in implementing their own type-directed-search down the road!

Now all that's left is to go try out a search or two :)

If you're interested in digging deeper, Unison Share, and by-proxy the entire type-directed search implementation, is all Open-Source, so go check it out! It's changing and improving all the time, but this module would be a good place to start digging.

Let us know in the Unison Discord if you've got any suggested improvements or run into any bugs. Cheers!

Hopefully you learned something 🤞! Did you know I'm currently writing a book? It's all about Lenses and Optics! It takes you all the way from beginner to optics-wizard and it's currently in early access! Consider supporting it, and more posts like this one by pledging on my Patreon page! It takes quite a bit of work to put these things together, if I managed to teach your something or even just entertain you for a minute or two maybe send a few bucks my way for a coffee? Cheers! 🍻

Hey folks! Today we'll be talking about GADTs, that is, "Generalized Abstract Data Types". As the name implies, they're just like Haskell's normal data types, but the generalized bit adds a few new features! They aren't actually too tough to use once you understand a few principles.

A lot of the writing out there regarding GADTs is pretty high-level research and academia, in contrast, today I'm going to show off a relatively practical and simple use-case. In this post we'll take a look at a very real example where we can leveraged GADTs in a real-world Haskell library to build a simple and expressive end-user interface.

We'll be designing a library for CSV manipulation. I used all of the following techniques to design the interface for my lens-csv library. Let's get started!

Here's a teensy CSV that we'll work with throughout the rest of the post. Any time you see input used in examples, assume it's this CSV.

Name,Age,Home
Luke,19,Tatooine
Leia,19,Alderaan
Han,32,Corellia

In its essence, a CSV is really just a list of rows and each row is just a list of columns. That's pretty much it! Any other meaning, even something as benign as "this column contains numbers" isn't tracked in the CSV itself.

This means we can model the data in a CSV using a simple type like [[String]], so far, so simple! There's a bit of a catch here though. Although it's clear to us humans that Name,Age,Home is the header row for this CSV, there's no marker in the CSV itself to indicate that! It's up to the user of the library to specify whether to treat the first row of a CSV as a header or not, and herein lies our challenge!

Depending on whether the CSV has a header row or not, the user of our library will want to reference the CSV columns by either a column name or column number. In a dynamic language (like Python) this is easily handled. We would provide separate methods for indexing columns by either header name or column number, and it would be the programmer's job to keep track of when to use which. In a strongly-typed language like Haskell however, we prefer to prevent such mistakes at compile time. Effectively, we want to give the programmer jigsaw pieces that only fit together in a way that works!

For the sake of pedagogy our miniature CSV library will perform the following tasks:

• Decode a CSV string into a structured type
• Get all the values in a given row

An Initial Approach

First things first we'll need a decode function to parse the CSV into a more structured type. In a production environment you'd likely use performant types like ByteString and Vector, but for our toy parser we'll stick to the types provided by the Prelude.

Since this is a post about GADTs and not CSVs encodings we won't worry about comma-escaping or quoting here, We'll do the naive thing and split our rows into cells on every comma. The Prelude, unfortunately, provides lines and words, but doesn't provide a more generic splitting function, so I'll whip one up to suit our needs.

Here's a function which splits a string on commas in such a way that each "cell" is separated in the resulting list.

splitOn :: Eq a => a -> [a] -> [[a]]
splitOn splitter = foldr go [[]]
  where
    go char xs
      -- If the current character is our "split" character create a new partition
      | splitter == char = []:xs
      -- Otherwise we can add the next char to the current cell
      | otherwise = case xs of
          (cell:rest) -> (char:cell):rest
          [] -> [[char]]

We can try it out to ensure it works as expected:

>>> splitOn ',' "a,b,c"
["a","b","c"]

-- Remember that CSV cells might be empty and we need it to handle that properly:
>>> splitOn ',' ",,"
["","",""]

Now we'll write a type to represent our CSV structure. We'll define two constructors: one for a CSV with headers, one for a CSV without headers.

data CSV =
      -- A CSV with headers includes a list of headers
      NamedCsv [String] [[String]]
      -- A CSV without headers contains only the CSV rows
    | NumberedCsv [[String]]
    deriving (Show, Eq)

Great, now we can write our first attempt of a decoding function. The implementation isn't really important here, so just focus on the type!

decode :: Bool -- ^ Whether to parse a header row or not
       -> String --  ^ The csv file
       -> Maybe CSV --  ^ We'll return "Nothing" if anything fails

And here's our implementation just in case you're following along at home:

-- Parse a header row
decode True input =
    case splitOn ',' <$> lines input of
        (headers:rows) -> Just (NamedCsv headers rows)
        [] -> Nothing
-- No header row
decode False input =
  let rows = splitOn ',' <$> lines input
   in Just (NumberedCsv rows)

Simple enough; we create a CSV with the correct constructor based on whether we expect headers or not.

So what if we want to get all of the names from our CSV? Let's write a function to get all the values of a specific column. Here's where things get a bit more interesting:

getColumnByNumber :: CSV -> Int    -> Maybe [String]
getColumnByName   :: CSV -> String -> Maybe [String]

Since each type of CSV takes a different index type we need a different function for each of the index types. Let's implement them!

-- A safe indexing function to get elements by index.
-- This is strangely missing from the Prelude... 🤔
safeIndex :: Int -> [a] -> Maybe a
safeIndex i = lookup i . zip [0..]

-- Get all values of a column by the column index
getColumnByNumber :: Int -> CSV -> Maybe [String]
getColumnByNumber columnIndex (NumberedCsv rows) =
    -- Fail if a column is missing from any row
    traverse (safeIndex columnIndex) rows
getColumnByNumber columnIndex (NamedCsv _ rows) =
    traverse (safeIndex columnIndex) rows

-- Get all values of a column by the column name
getColumnByName :: String -> CSV -> Maybe [String]
getColumnByName  _ (NumberedCsv _) = Nothing
getColumnByName columnName (NamedCsv headers rows) = do
    -- Get the column index from the headers
    columnIndex <- elemIndex columnName headers
    -- Lookup the column from each row, failing if the column is missing from any row
    traverse (safeIndex columnIndex) rows

This works of course, but it feels like we're programming in a dynamic language! If you try to get a column by name from a numbered CSV we know it will ALWAYS fail, so why do we even allow the programmer to express that? Certainly it should fail to typecheck instead.

>>> decode True input >>= getColumnByName "Name"
Just ["Luke","Leia","Han"]

-- If we index a numbered CSV by name we'll get 'Nothing' no matter what.
>>> decode False input >>= getColumnByName "Name"
Nothing

The problem here becomes even more pronounced when we write a function like getHeaders. Which type signature should it have?

This one:

getHeaders :: CSV -> [String]

Or this one?

getHeaders :: CSV -> Maybe [String]

We could pick the first signature and always return the empty list when someone mistakenly tries to get the headers of a numbered CSV, but that seems a bit disingenuous; It's common to check the number of columns in a CSV by counting the headers, and that approach would imply that every numbered CSV has zero columns! If we go with the latter signature it properly handles the failure case of calling getHeaders on a numbered CSV, but we know that getting the headers from a NamedCSV should never fail, so in that case we're adding a bit of unnecessary overhead, all callers will have to unwrap Maybe in that case no matter what 😬.

In order to fix this issue we'll need to go back to the drawing board and see if we can keep track of whether our CSV has headers inside its type.

Differentiating CSVs using types

I promise we'll get to using GADTs soon, but let's look at the "simple" approach that I suspect most folks would try next and see where it ends up so we can motivate the need for GADTs.

The goal is to prevent the user from calling "header" specific methods on a CSV that doesn't have headers. The simplest thing to do is provide two separate decode methods which return completely different concrete result types:

decodeWithoutHeaders :: String -> Maybe [[String]]
decodeWithHeaders    :: String -> Maybe ([String], [[String]])

Next we could implement:

getColumnByNumber :: Int -> [[String]]             -> Maybe [String]
getColumnByName   :: Int -> ([String], [[String]]) -> Maybe [String]

This solves the problem at hand, if we decode a CSV without headers we'll have a [[String]] value, and can't pass that into getColumnByName. However there's an issue with this approach: We can no longer use getColumnByNumber to get a column by number on a CSV which has headers.

We could, of course we could could snd it into [[String]] first, but converting between types everywhere is annoying and also means we can't write code which is polymorphic over both kinds of CSV. Ideally we would have a single set of functions which was smart about which type of CSV so it could do the right thing while also ensuring type-safety.

Some readers are likely thinking "Hrmmm, a group of functions polymorphic over a type? Sounds like a typeclass!" and you'd be right! As it turns out, this is roughly the approach that the popular cassava library takes to its library design.

cassava is more record-centric than the library we're designing, so it provides separate typeclasses for named and unnamed record types; ToNamedRecord, FromNamedRecord, and their numbered variants ToRecord and FromRecord. In our case we'll be defining different typeclass instances for the CSV itself.

Here's the rough idea:

{-# LANGUAGE TypeFamilies #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE InstanceSigs #-}

class IsCSV c where
  -- A type family to specify the "indexing" type of the CSV
  type Index c :: Type
  -- Try parsing a CSV of the appropriate type
  decode :: String -> Maybe c

  getColumnByIndex  :: Index c -> c -> Maybe [String]
  getColumnByNumber :: Int     -> c -> Maybe [String]
  getRow :: Int -> Row c

Let's talk about the Index type family. Numbered CSVs are indexed by an Int, while Named CSVs are indexed by a String. We can use the Index associated type family to specify a different type of Index for each typeclass instance.

The headerless CSV is pretty easy to implement:

instance IsCSV [[String]] where
  type Index [[String]] = Int
  -- You can re-purpose the earlier decoder here.
  decode = ...

  -- The Index type is Int, so we index by Int here:
  getColumnByIndex :: Int -> [[String]] -> Maybe [String]
  getColumnByIndex n rows = traverse (safeIndex n) rows

  -- Since the index is an Int we can re-use the other implementation
  getColumnByNumber :: Int -> [[String]] -> Maybe [String]
  getColumnByNumber = getColumnByIndex

Now an instance for a CSV with headers:

instance IsCSV ([String], [[String]]) where
  -- We can index a column by the header name
  type Index ([String], [[String]]) = String
  decode = ...

  -- The 'index' for this type of CSV is a String
  getColumnByIndex :: String -> ([String], [[String]]) -> Maybe [String]
  getColumnByIndex columnName (headers, rows) = do
    columnIndex <- elemIndex columnName headers
    traverse (safeIndex columnIndex) rows
  -- We can still index a Headered CSV by column number
  getColumnByNumber :: Int -> ([String], [[String]]) -> Maybe [String]
  getColumnByNumber n = getColumnByNumber n . snd

This works out pretty well, here's how it looks to use it:

>>> decode input >>= getColumnByIndex ("Name" :: String)
<interactive>:99:36: error:
    • Couldn't match type ‘Index c0’ with ‘String’
      Expected type: Index c0
        Actual type: String
      The type variable ‘c0’ is ambiguous
    • In the first argument of ‘getColumnByIndex’, namely
        ‘"Name"’
      In the second argument of ‘(>>=)’, namely
        ‘getColumnByIndex "Name"’
      In the expression:
        decode input >>= getColumnByIndex "Name"

Uh oh... one issue with type classes is that GHC might not know which which instance to use in certain situations!

We can help out GHC with a type hint, but it's a bit annoying and the error message isn't always so clear!

>>> decode @([String], [[String]]) input >>= getColumnByIndex "Name"
Just ["Luke","Leia","Han"]

-- Or we can define a type alias to clean it up a smidge
>>> type Named = ([String], [[String]])
>>> decode @Named input >>= getColumnByIndex "Name"
Just ["Luke","Leia","Han"]

This works out okay but it's unintuitive to users to need a type annotation. Let's take a look at how GADTs can allow us to encourage better error messages while also making it easier to read, and reduce the amount of boilerplate required.

The GADT approach

Before we use them for CSVs let's get a quick primer on GADTs, if you're well-acquainted already feel free to skip to the next section.

GADTs, a.k.a. Generalized Abstract Data Types, bring a few upgrades over regular Haskell data types. Just in case you haven't seen one before, let's compare the regular Maybe definition to its GADT version.

Here's how Maybe is written using standard data syntax:

data Maybe a =
    Nothing
  | Just a

When we turn on GADTs we can write the exact same type like this instead:

data Maybe a where
  Nothing :: Maybe a
  Just :: a -> Maybe a

This slightly different syntax, which looks a bit foreign at first, is really just spelling out the type of constructors as though they were functions!

Compare the definition with the type of each constructor:

>>> :t Nothing
Nothing :: Maybe a
>>> :t Just
Just :: a -> Maybe a

Each argument to the function represents a "slot" in the constructor.

But of course there's more than just the definition syntax! Why use GADTs? They bring a few upgrades over regular data definitions. GADTs are most often used for their ability to include constraints over polymorphic types in their constructor definitions. This means you can write a type like this:

{-# LANGUAGE GADTs #-}

data HasEq a where
  HasEq :: Eq a => a -> HasEq a

Where the Eq a constraint gets "baked in" to the constructor such that we can then write a function like this:

checkEq :: HasEq a -> HasEq a -> Bool
checkEq (HasEq one) (HasEq two) = one == two

We don't need to include an Eq a constraint in the type because GHC knows that it's impossible to construct HasEq without one, and it carries that constraint with the value in the constructor!

In this post we'll be using a technique which follows (perhaps unintuitively) from this; take a look at this type:

data IntOrString a where
  AnInt :: Int -> IntOrString Int
  AString :: String -> IntOrString String

Notice how each constructor fills in a value for the polymorphic a type? E.g. IntOrString Int where a is now Int? GHC can use this information when it's matching constructors to types. It lets us write a silly function like this:

toInt :: IntOrString Int -> Int
toInt (AnInt n) = n

Again, this doesn't seem too interesting, but there's something unique here. It looks like I've got an incomplete implementation for toInt; it lacks a case for the AString constructor! However, GHC is smart enough to realize that any values produced using the AString constructor MUST have the type IntOrString String, and so it knows that I don't need to handle that pattern here, in fact if I do provide a pattern match on it, GHC will display an "inaccessible code" warning!

The really nifty thing is that we can choose whether to be polymorphic over the argument or not in each function definition and GHC will know which patterns can appear in each case. This means we can just as easily write this function:

toString :: IntOrString a -> String
toString (AnInt n) = show n
toString (AString s) = s

Since a might be Int OR String we need to provide an implementation for both constructors here, but note that EVEN in the polymorphic case we still know the type of the value stored in each constructor, we know that AnInt holds an Int and AString holds a String.

If you're a bit confused, or just generally unconvinced, try writing IntOrString, toInt and toString in a type-safe manner using a regular data constructor, it's a good exercise (it won't work 😉). Make sure you have -Wall turned on as well. .

GADTs and CSVs

After that diversion, let's dive into writing a new CSV type!

{-# LANGUAGE GADTs #-}
{-# LANGUAGE StandaloneDeriving #-}

data CSV index where
  NamedCsv    :: [String] -> [[String]] -> CSV String
  NumberedCsv ::             [[String]] -> CSV Int

-- A side-effect of using GADTs is that we need to use standalone deriving 
-- for our instances.
deriving instance Show (CSV i)
deriving instance Eq   (CSV i)

This type has two constructors, one for a CSV with headers and one without. We're specifying a polymorphic index type variable and saying that CSVs with headers are specifically indexed by String and CSVs without headers are indexed by Int. Notice that it's okay for us to specify a specific type for the index parameter even though it's a phantom-type (i.e. we don't actually store the index type inside our structure anywhere).

Let's implement our CSV functions again and see how they look.

We still need the end-user to specify whether to parse headers or not, but we can use another GADT to reflect their choice in the type, and propagate that to the resulting CSV. Here's what a CSV selector type looks like where each constructor carries some type information with it (i.e. whether the resulting CSV is either String or Int indexed).

data CSVType i where
  Named :: CSVType String
  Numbered :: CSVType Int

deriving instance Show (CSVType i)
deriving instance Eq (CSVType i)

Now we can write decode like this:

decode :: CSVType i -> String -> Maybe (CSV i)
decode Named s = case splitOn ',' <$> lines s of
    (h:xs) -> Just $ NamedCsv h xs
    _ -> Nothing
decode Numbered s = Just . NumberedCsv . fmap (splitOn ',') . lines $ s

By accepting CSVType as an argument it acts as a proxy for the type information we need. We can provide then provide a separate implementation for each csv-type easily, and the index type provided on the CSVType option is propagated to the result, thus determining the type of the output CSV too!

Now for getColumnByIndex and getColumnByNumber; in the typeclass version we needed to provide an implementation for each class instance, using GADTs we can collapse everything down to a single implementation for function.

Here's getColumnByIndex:

getColumnByIndex :: i -> CSV i -> Maybe [String]
getColumnByIndex  columnName (NamedCsv headers rows) = do
    columnIndex <- elemIndex columnName headers
    traverse (safeIndex columnIndex) rows
getColumnByIndex n (NumberedCsv rows) = traverse (safeIndex n) rows

The type signature says, if you give me the index type which matches the index to the CSV you provide, I can get you that column if it exists. It's smarter than it looks!

Even though the GADT constructor comes after the first argument, by pattern matching on it we can determine the type of i, and we then know that the first argument must match that i type. So when we match on NamedCsv the first argument is a String, and when we match on NumberedCsv it's guaranteed to be an Int

In the original "simple" CSV implementation you could try indexing into a numbered CSV with a String header and it would always return a Nothing, now it's actually a type error; we've prevented a whole failure mode!

-- Decode our input into a CSV with numbered columns
>>> let Just result = decode Numbered input
>>> result
NumberedCsv [
  ["Name","Age","Home"],
  ["Luke","19","Tatooine"],
  ["Leia","19","Alderaan"],
  ["Han","32","Corellia"]]
-- Here's what happens if we try to write the wrong index type!
>>> getColumnByIndex "Name" result
    • Couldn't match type ‘Int’ with ‘String’
      Expected type: CSV String
        Actual type: CSV Int

It works fine if provide an index which matches the way we decoded:

-- By number using `Numbered`
>>> decode Numbered input >>= getColumnByIndex 0
Just ["Name","Luke","Leia","Han"]
-- ...Or by header name using `Named`
>>> decode Named input >>= getColumnByIndex "Name"
Just ["Luke","Leia","Han"]

When indexing by number we can ignore the index type of the CSV entirely, since we know we can index either a Named or Numbered CSV by column number regardless.

getColumnByNumber :: Int -> CSV i -> Maybe [String]
getColumnByNumber n (NamedCsv _ rows) = traverse (safeIndex n) rows
getColumnByNumber n (NumberedCsv rows) = traverse (safeIndex n) rows

In an earlier attempt we ran into problems writing getHeaders, since we knew intuitively that it should always be safe to return the headers from a "Named" csv, but we needed to introduce a Maybe into the type since we couldn't be sure of the type of the CSV argument!

Now that the CSV has the index as part of the type we can solve that handily by restricting the possible inputs the correct CSV type:

getHeaders :: CSV String -> [String]
getHeaders (NamedCsv headers _) = headers

We don't need to match on NumberedCsv, since it has type CSV Int, and that omission allows us to remove the need for a Maybe from the signature. Pretty slick!

This is the brilliance of GADTs in this approach, we can be general when we want to be general, or specific when we want to be specific.

The interfaces provided by each approach look relatively similar at the end of the day. The typeclass signatures have a fully polymorphic variable with a type constraint AND a type family, whereas the GADT signatures are simpler, including only a polymorphic index type, and the consumers of the library won't need to know anything about GADTs in order to use it.

The typeclass approach:

decode :: IsCSV c => String -> Maybe c
getColumnByIndex :: IsCSV c => Index c -> c -> Maybe [String]
getColumnByNumber :: IsCSV c => Int -> c -> Maybe [String]
getHeaders :: IsCSV c => c -> Maybe [String]

The GADT approach:

decode :: CSVType i -> String -> Maybe (CSV i)
getColumnByIndex :: i -> CSV i -> Maybe [String]
getColumnByNumber :: Int -> CSV i -> Maybe [String]
getHeaders :: CSV String -> [String]

Though similar, I find the GADT version easier to understand as a consumer, everything you need to know is available to you, and you can look up the CSV type to learn more about how to build one, or which types are available.

The GADT types also result in simpler type errors when something goes wrong.

Here's one common problem with the typeclass approach, decode has a polymorphic result and getColumnByIndex has a polymorphic argument, GHC can't figure out what the intermediate type should be if we string them together:

>>> decode input >>= getColumnByIndex "Name"
    • Couldn't match type ‘Index c0’ with ‘String’
      Expected type: Index c0
        Actual type: String
      The type variable ‘c0’ is ambiguous
    • In the first argument of ‘getColumnByIndex’, namely
        ‘("Hi" :: String)’
      In the second argument of ‘(>>=)’, namely
        ‘getColumnByIndex ("Hi" :: String)’
      In the expression:
        decode input >>= getColumnByIndex ("Hi" :: String)

We can fix this with an explicit type application, but that requires us to know the underlying type that implements the instance.

>>> type Named = ([String], [[String]])
>>> decode @Named input >>= getColumnByIndex "Name"
Just ["Luke","Leia","Han"]

If we mismatch the index type here, even when providing an explicit type annotation, we get a slightly confusing error since it still mentions a type family:

>>> decode @Named input >>= getColumnByIndex (1 :: Int)
    • Couldn't match type ‘Int’ with ‘String’
      Expected type: Index Named
        Actual type: Int
    • In the first argument of ‘getColumnByIndex’, namely ‘(1 :: Int)’
      In the second argument of ‘(>>=)’, namely
        ‘getColumnByIndex (1 :: Int)’
      In the expression:
        decode @Named input >>= getColumnByIndex (1 :: Int)

Compare these to the errors generated by the GADT approach; first we'll chain decode with getColumnByIndex:

>>> decode Named input >>= getColumnByIndex "Name"
Just ["Luke","Leia","Han"]

There's no ambiguity here! We only have a single CSV type to choose, and the "index" type variable is fully determined by the Named argument. Very nice!

What if we try to index by number instead?

>>> decode Named input >>= getColumnByIndex (1 :: Int)
error:
    • Couldn't match type ‘String’ with ‘Int’
      Expected type: CSV String -> Maybe [String]
        Actual type: CSV Int -> Maybe [String]
    • In the second argument of ‘(>>=)’, namely
        ‘getColumnByIndex (1 :: Int)’
      In the expression:
        decode Named input >>= getColumnByIndex (1 :: Int)
      In an equation for ‘it’:
          it = decode Named input >>= getColumnByIndex (1 :: Int)

It clearly outlines the expected and actual types: