Rust implementation of the peer-to-peer layer of SQD Network, the decentralized data lake behind SQD. Nodes communicate over libp2p, exchanging queries, query results, heartbeats, and logs. For the network design, see the network architecture wiki and the SQD Network docs.
SQD Network stores blockchain data as chunks distributed across worker nodes. A scheduler assigns chunks to workers; portals route client queries to the workers that hold the relevant data and collect the results. SQD Portal, the data API over 200+ chains, is built on top of this network. This repository provides the transport protocol, message formats, and a few supporting node binaries used by those actors.
The transport layer exposes APIs for the predefined actors on the network:
- Worker: holds data chunks and answers queries routed to it.
- Scheduler: assigns chunks to workers based on collected heartbeats and distributes the assignments.
- Portal: entry point for clients. It sends queries to workers and collects responses. (Previously called Gateway.)
- Pings collector: collects worker heartbeats and stores them.
- Logs collector: persists query logs received from workers for analysis.
This is a Cargo workspace (crates/*).
| Crate | Type | Description |
|---|---|---|
sqd-network-transport |
library | libp2p transport and per-actor behaviours. Actors are gated behind Cargo features (worker, portal, observer, pings-collector, logs-collector, portal-logs-collector, sql-client). |
sqd-messages |
library | Protobuf (prost) message schemas for inter-node communication. Source schema: crates/messages/proto/messages.proto. |
sqd-assignments |
library | FlatBuffers chunk-assignment format with builder and reader features. Schema: crates/assignments/schema/assignment.fbs. |
sqd-contract-client |
library | Reads onchain registries, allocations, and node sets from the network's smart contracts over an RPC node (ethers). |
sqd-bootnode |
binary | Minimal node that helps new nodes discover peers and bootstrap into the network. |
sqd-keygen |
binary | Generates a libp2p keypair for a node and prints its peer ID. |
sqd-observer |
binary | Joins the network, observes gossip and peers, and exposes Prometheus metrics over HTTP. |
Requires the Rust toolchain pinned in rust-toolchain (1.89.0) and
protoc (protobuf-compiler) for building sqd-messages.
cargo build --releaseBuild a specific binary:
cargo build --release --bin sqd-bootnode
cargo build --release --bin sqd-keygen
cargo build --release --bin sqd-observerThe transport library compiles only the actor APIs you enable. For example, to build with the worker actor:
cargo build -p sqd-network-transport --features workerThe Dockerfile builds the three binaries and provides image targets for
bootnode, keygen, and observer. The published images are
subsquid/bootnode, subsquid/keygen, and subsquid/observer.
Generate a keypair and write it to a key file in the current directory, or
print the peer ID if the file already exists:
docker run -u $(id -u):$(id -g) -v .:/host subsquid/keygen:latest /host/keyBoth nodes share the transport CLI flags (also settable via environment variables):
| Flag | Env | Description |
|---|---|---|
--key |
KEY_PATH |
Path to the libp2p key file. |
--p2p-listen-addrs |
P2P_LISTEN_ADDRS |
Multiaddrs the node listens on. |
--p2p-public-addrs |
P2P_PUBLIC_ADDRS |
Public multiaddrs the node advertises. |
--boot-nodes |
BOOT_NODES |
Boot nodes as <peer_id> <address> entries. |
--rpc-url |
RPC_URL |
RPC node URL for reading network contracts. |
The observer additionally takes --port (HTTP port for metrics, default 8000)
and --network (mainnet or tethys).
- SQD Network: https://docs.sqd.dev/en/network
- SQD: https://sqd.dev
AGPL-3.0-or-later. See LICENSE.md.
