Jack Vanlightly

@vanlightly

Likes breaking ideas and systems, writing, picking systems apart @confluentinc Ex @Splunk, @VMware jack-vanlightly.com Credit: ESO/B. Tafreshi

Barcelona, Spain

jack-vanlightly.com

Replies Reposts Media

Jack Vanlightly
@vanlightly
1h
OpenData Buffer now has a TLA+ spec: github.com/Vanlightly/s3-… I had to work to make the cursor management work, as that is outside of the Buffer protocol. Next I'll make a version of Buffer which could act as a single-writer WAL and add snapshots/state machine state to match the
s3-wal-collection/opendata/Buffer.tla at main · Vanlightly/s3-wal-collection
From github.com
Jack Vanlightly
@vanlightly
3h
I decided to specify OpenData Buffer today. It's not quite a log, but it's in the ballpark (S3 only source of truth and only required coordination point, except for consumer cursor). Later I'll hack on a version of Buffer that supports a single-active-writer WAL.
1
Jack Vanlightly
@vanlightly
8h
For my WALs-on-S3 collection, I'm going to stick to strictly S3-based designs. That is, S3 is the only source of truth. Designs which use some servers for simple soft-state coordination will be included. Designs which land data on SSDs durably are excluded, as are ones that
Jack Vanlightly
@vanlightly
Aug 31
My current plan is to verify each WAL-on-S3 design based on a state-machine-replication (SMR) approach. The OSWALD P model does this (SM data is a counter per writer) and the log is a sequence of writer-increment ops. The SM data in my OSWALD TLA+ specification is simply the
Jack Vanlightly
@vanlightly
Aug 31
It's probably worth asking what qualifies as a WAL-on-S3 design. Is it only designs where S3 is the only durable state and only coordination layer? Or is a server-based coordination point acceptable (such as a fast sequencer)? Is landing data on SSDs (as a durable source of truth
2