A Rust framework for correct and performant distributed systems

Distributed systems deserve a native framework

Nearly every application today is a distributed system: services call other services, replicas coordinate state, and data flows across regions. Distribution is how modern software scales, survives failures, and stays close to its users.

Hydro is a Rust framework that treats distribution as a first-class concern. Instead of assembling a system from parts and relying on manual review to catch mistakes at their boundaries, you express, check, test, and deploy the whole distributed system as one program.

Today's frameworks make networks implicit

Most frameworks split a distributed system into single-machine programs that communicate through opaque RPC calls. The network—the part that makes your system distributed—is hidden inside client stubs and await points, invisible to the compiler and your tools.

Reordering, duplication, and partial failure all live in the gap between these files. Because the language cannot see the network, it cannot help you reason about what happens across machines.

node1/src/main.rs

let mut client = EchoClient::connect("http://node2:5000").await?;

// retries? reordering? failures? not visible here.
let reply = client.echo(Msg { text: "hello".into() }).await?;

node2/src/service.rs

impl Echo for EchoService {
    async fn echo(&self, req: Request<Msg>)
        -> Result<Response<Msg>, Status> {
        // where did this request come from? in what order?
        let text = req.into_inner().text;
        Ok(Response::new(Msg { text: text.to_uppercase() }))
    }
}

Hydro is global

Hydro is the first production framework with location-oriented programming: a single function can encapsulate logic spanning several machines. Distributed locations are captured in types, and sending data across the network is an explicit, type-checked operation.

These abstractions are zero-cost: Hydro compiles to the same networked binaries you would write by hand, and you retain full control over the network protocol, compute placement, and serialization format.

src/echo.rs

pub fn echo<'a>(
    client: &Process<'a, Client>,
    server: &Process<'a, Server>,
    requests: Stream<String, Process<'a, Client>>,
) -> Stream<String, Process<'a, Client>> {
    requests
        .send(server, TCP.fail_stop().bincode())
        // ⇒ Stream<String, Process<Server>>
        .map(q!(|s| s.to_uppercase()))
        // ⇒ Stream<String, Process<Server>>
        .send(client, TCP.fail_stop().bincode())
        // ⇒ Stream<String, Process<Client>>
}

Hydro catches distributed bugs at compile time

Just like Rust ensures memory safety through the borrow checker, Hydro ensures distributed safety through stream types that track network behaviors end-to-end. Locations shape these types: when a Cluster of machines sends to a single Process, messages from different members arrive interleaved, so the resulting stream is unordered.

If your logic relies on a message ordering that the network does not guarantee, Hydro rejects the program at compile time. Errors are surfaced through the Rust type system, visible to your editor, language server, and agents. To resolve them, you can prove properties like commutativity and idempotence, or explicitly handle the non-determinism.

pub fn concat_words<'a>(
    server: &Process<'a, Server>,
    words: Stream<String, Cluster<'a, Client>>,
) -> Singleton<String, Process<'a, Server>> {
    words
        .send(server, TCP.fail_stop().bincode())
        // ⇒ Stream<String, Process<Server>, Unbounded, NoOrder>
        .fold(q!(String::new), q!(|acc, x| *acc += &x))
error: `fold` requires an ordered stream, but this stream is `NoOrder`   = note: messages from different cluster members may be interleaved in any order
   = help: prove the closure is commutative with `commutative = manual_proof!(...)`
}

Hydro lets you write distributed tests

Hydro offers built-in deterministic simulation testing, which lets you simulate distributed programs on your laptop. Tests run against varied distributed schedules, including message interleavings, batch boundaries, and state snapshots, to catch concurrency bugs and race conditions.

Because Hydro's type system enforces determinism, the simulator only needs to explore nondet! decision points, so even large protocols can be checked exhaustively: your assertions become guarantees about every possible execution.

let (event_port, events) = clients.sim_input();

let log = events
    .send(&server, TCP.fail_stop().bincode())
    .entries_partially_ordered(nondet!(/** arrival order */))
    .sim_output();

flow.sim().with_cluster_size(&clients, 3).exhaustive(async || {
    event_port.send(0, "a1");
    event_port.send(0, "a2");
    event_port.send(1, "b1");

    let entries: Vec<_> = log.collect().await;
    assert!(pos("a1") < pos("a2")); // per-member order holds
});

Research Backed. Production Ready.

Hydro has its roots in foundational distributed systems research at UC Berkeley, such as the CALM theorem. It is now co-led by a team at Berkeley and AWS, with contributions from the open-source community.

Hydro continues to lead the way with cutting-edge capabilities, such as automatically optimizing distributed protocols, while supporting production use with cloud integrations and observability tooling.

Get Started with Hydro