Federated JEPA world models for sovereign robotics data.

Lensemble is a research implementation of federated, action-conditioned JEPA world models. It is built for the hard case: each participant keeps raw trajectory data local, trains against the same protocol, and only releases update artifacts for aggregation.

The central problem is the JEPA latent gauge. In self-supervised representation learning, two participants can learn useful but rotated latent frames. Naive FedAvg then averages incompatible coordinates. Lensemble implements the frame-anchor, aggregation, privacy, provenance, and evaluation machinery needed to study that failure mode directly.

• A typed Python package and lensemble CLI.
• End-to-end JEPA model components, SIGReg/anchor objectives, and federated round machinery.
• Secure aggregation and differential-privacy plumbing for the release path.
• Hash-bound checkpoints, manifests, dataset provenance, and evidence bundles.
• CPU-oriented tests for contracts, determinism, residency, aggregation, DP, and dynamic-env evaluation.
• A browser federated demo for QR joins, WebSocket orchestration, bounded tiny browser updates, aggregation, inference, and evidence export.

git clone https://github.com/AbdelStark/Lensemble.git
cd Lensemble

uv venv .venv --python 3.11
uv pip install "torch>=2.4,<3" --index-url https://download.pytorch.org/whl/cpu
uv pip install -e ".[dev,docs]"

Check the CLI:

uv run lensemble --help

Run the browser federated demo:

uv run lensemble demo federated --port 8765

Open the printed URL, usually:

http://127.0.0.1:8765/web/federated-demo/

For tunnel or LAN rehearsal, bind the coordinator and provide the external base URL used in QR codes and WSS URLs:

uv run lensemble demo federated \
  --host 0.0.0.0 \
  --public-base-url https://YOUR-TUNNEL.trycloudflare.com/web/federated-demo \
  --public-demo \
  --deployment-target cloudflare-tunnel

participant data stays local
  -> local JEPA training
  -> update artifact, hash, metadata
  -> secure aggregation and DP accounting
  -> frame-anchored global update
  -> checkpoint, manifest, evidence bundle
  -> evaluation against explicit baselines

The design is specified in SPEC.md, with normative sections in docs/spec and decision records in docs/rfcs.

Start here:

• RFC-0002: latent gauge and frame-anchored aggregation
• RFC-0005: evaluation protocol
• RFC-0017: dynamic-env metric gate
• Browser federated demo roadmap
• Tapestry-like LeWM TwoRooms pivot contract
• Dynamic-env evidence roadmap

The project treats results as artifact-bound. The important public evidence surfaces are checked in:

• Phase 3 evidence bundle
• Phase 3 model card
• Tapestry-like LeWM demo card
• LeWM TwoRooms system-composed probe
• LeWM TwoRooms probe seed sweep
• Dynamic-env roadmap and acceptance matrix
• Browser federated demo docs

The short read: Lensemble has credible systems and gauge-control evidence. It does not yet have a claim-grade result that federated training materially beats local-only on the binding dynamic-env metric.

The LeWM TwoRooms browser demo has a credible result for a narrow claim: federated adaptation of a bounded adapter on a frozen checkpoint. The headline probe number is produced by the shipped system, not offline math. Real adapter deltas pass the server validation and aggregation path, and the probe scores the server-produced revision. The held-out gain (+12.3% MSE on the committed seed) is checked to be bias-correction, not latent collapse, and holds across 5 seeds (worst +5.4%). It is not full-model federated training, runs through a single local coordinator, and does not wire secure aggregation or DP in that path.

SO-100 is not a downstream-useful world model: held-out latent magnitude collapse (~7.5e-6; thoughts/collapse_fix_probe.py), the central ceiling probe (thoughts/central_ceiling_probe.py), skill_vs_identity is gameable, and effective_rank is scale-invariant.

Useful local gates:

uv run ruff check .
uv run pyright
uv run pytest tests/unit tests/property tests/integration tests/ml tests/e2e tests/regression
uv run python scripts/check_docs_links.py docs SPEC.md README.md
uv run python -m mkdocs build --strict
git diff --check

The blocking CI gates run on CPU and do not download private datasets, checkpoints, or probes. See CONTRIBUTING.md for the full merge rule.

lensemble/              Python package and CLI
tests/                  unit, property, integration, ML, e2e, regression
docs/spec/              normative architecture and public API docs
docs/rfcs/              design records and research contracts
docs/roadmap/           implementation and evidence state
docs/evidence/          checked-in reports, bundles, and model cards
web/federated-demo/     browser federation demo
deploy/hfjobs/          Hugging Face Jobs launchers
scripts/                release, evidence, docs, and benchmark producers

Lensemble is not a production federation stack, not a browser-training product, not a cryptographic proof system, and not evidence of closed-loop physical robot success. Raw participant trajectories are not released and should not cross trust boundaries.

Code is Apache-2.0. Documentation is CC-BY-4.0. Released data artifacts use CDLA-Permissive-2.0.