Execution Truth vs Execution Authority
Most coordination systems and primitives focus on one thing authority
Who is allowed to perform work right now?
Redis SETNX issues authority based on a timed expiry lock, Set N key for X time, redlock does the same but with quorum so multiple nodes have to agree on the same conclusion or majority.
The question of execution authority is easy to answer relatively, assume you have a monolith work can only be assigned to a worker based on who holds the lock right now and whether it's expired.
In microservices, a majority of nodes have to agree on whether this action can be performed, keyword being majority because if one or more nodes fail to answer in time or are unreachable there is never perfect consensus across time, there are always tradeoffs.
Infrastructure like ZooKeeper and Etcd are used for consensus algorithm earlier used inside platforms like Kafka, they're in a way primitives for infrastructure, so what is consensus ? and why do applications need consensus?
consensus is a process where multiple nodes agree on a shared state, it's used by applications to keep data in sync even when one or more nodes fail, ensure fault tolerance and allow for coordination over the network by sharing a replicated log across multiple services to read from and finally elect leaders without a central authority to ensure failure doesn't exist at a single point.
Algorithms such as Raft, Paxos, infrastructure such as Etcd, ZooKeeper which use such algorithms and similarly redlock and setnx or any form of a lease, all answer one question who is authoritative at this current point of time, it's a question of state and it can always be answered because it's continuously changing and it's permission for actions to take place, it can be inferred from various factors but does the same hold true for the aftermath of an action after it has been completed ?
This aftermath of an action is execution truth, it's the result from the action that has taken place and it's a much harder question to generalise and answer because crashes happen after api calls have been made, workers are slow or there is a network partition, all of this creates a state where the only thing you know for certain is uncertainty of the current state, it's the space between execution of work and informing the application of it's completion.
In a normal scenario, worker claims a lock, performs the work and finally the details of it's completion reach the db, but when failure happens you cannot determine whether the action took place, the answer is idempotency but idempotency is application specific and cannot be generalised in a majority of cases it's a check-and-insert in the same transaction or check-and-perform the action but this creates race conditions between expiry of lock and period when db is checked for completion.
How do you ideally know execution truth ? the answer is determinism or basically the answer cooked up by Temporal, all the work for given inputs always returns the same output, sound familiar ? because that is idempotency, but Temporal adds a layer to this since it expects determinism, it can reconstruct state in a given workflow from scratch because the answer is always the same but the tradeoff as documented is idempotency as a requirement and giving up native nondeterministic calls (like time.now(), random, direct I/O) and effectively using their runtime but since crashes are still possible even Temporal can execute activities within the workflow twice.
Authority is a coordination problem while Truth is an observability problem, but is there any way to solve the issue of execution truth ? No, most of the best practices follow the philosophy of Temporal which answers only one half.
But the best possible thing an application can do is surface the uncertainty and deal with it, that's the most honest thing an application can do, it is to expose the current state and reconcile in the same direction as completion, execution truth asks
Did the side effect actually happen?
which is the second half of the question asked by authority, which cannot ever be perfectly answered at any point in time, but with durable storage/execution, idempotency and surfacing current state you can build an application which is honest enough to get an approximate answer, enough to be truthful across time.
I ran into this exact issue when building a coordination primitive that I made to solve guarantees across a payments service, the implementation details are in
pip install sentinel-coordination
