Device measurement: trust model needs revisit

File: morphling/entrypoint/run_device.py:98
Current FIXME text: FIXME(#45): server-side trust model for device-reported FLOPS/bandwidth/latency.

The device currently self-reports its FLOPS, bandwidth, and latency to the server. The contract should be:

Latency and bandwidth measured by the server, not the device.
Server sends random GEMM tasks to measure FLOPS; result verification needed.

This is a known design gap surfaced during the open-source readiness audit. Not in scope for the EdgeSys '26 release pass; tracked here so the marker has a stable reference.

See also commit chore(release): purge paper figures, plot scripts, and morphling.evaluation for the broader audit context.

chore(code): link 6 high-visibility TODOs to tracking issues

Issue #45 — Device measurement trust model

Problem (verified against current code)

The device runner self-reports flops, memory, ul_bw, dl_bw, ul_lat, dl_lat via CLI flags in morphling/entrypoint/run_device.py:80-108. These values are stuffed into MORPHLING_* environment variables, read back by the proxy client in csrc/backend/proxy_cli.cpp:559-565 as a DeviceProfileData payload, and forwarded to the server which registers them verbatim in DEVICE_TRACKER.RegisterDevice (csrc/backend/proxy_svr.cpp:188-192).

The FIXME at run_device.py:98 flags this: a malicious or misconfigured device can lie about its capabilities, and the server takes the numbers on faith.

Scope decision

This is a design-gap issue, not a release blocker. We will land the measurement infrastructure, but the trust-model policy (replace vs. cross-check vs. clamp) is left as a follow-up because it requires paper-level decisions (does Morphling's stride-aware model assume honest input?).

This plan implements the measurement infrastructure only.

Out of scope (explicit non-goals)

Cryptographic device attestation.
Replacing the device-reported FLOPS with the server-measured value in DEVICE_TRACKER (we record both; reconciliation policy is a separate decision).
Multi-round adaptive probing.
Network jitter / packet-loss modeling.

Design

Three independent measurement paths, each gated by a server-side config flag (default off, so existing behavior is unchanged):

M1. Server-measured network latency (`ul_lat`, `dl_lat`)

On DeviceProfileData registration, server sends an N-byte ping (N=64) and records RTT.
Repeat K=5 times after min_interval_ms=10ms, take median, split into ul_lat = dl_lat = median/2 (we cannot distinguish directions without device cooperation; record that limitation in the result).
New struct field DeviceProfileData.measured_lat_ns (uint64, 0 = not measured).

M2. Server-measured bandwidth (`ul_bw`, `dl_bw`)

After M1 completes, server sends a B-byte block (B=4 MiB) over the existing connection, measures wall time, subtracts measured RTT, divides B by that → measured_dl_bw_bps (server→device, i.e. device downlink).
Device echoes the block back; server measures again → measured_ul_bw_bps.
Echo handler in proxy_cli.cpp is a new opcode; reuse the same socket; no new transport.
New struct fields measured_ul_bw_bps, measured_dl_bw_bps (uint64, 0 = not measured).

M3. Server-measured FLOPS via challenge GEMM

After M2, server picks (m, n, k) from a small fixed set (e.g. {(1024,1024,1024), (2048,2048,2048)}) plus a uniformly random (m, n, k) from [256..2048]^3 with seed=device_uuid, ships a MatMulRequestMessage via the existing path, and times the round-trip minus measured 2 × dl_lat.
Verification: server seeds A, B with seed=device_uuid so it can recompute the expected C and check element-wise tolerance (atol=1e-3, rtol=1e-3 for float32). Mismatch ⇒ measured_flops_bps = 0 and a warning is logged.
FLOPS = 2 * m * n * k / elapsed_seconds. Reported in same units as profile.flops.
New struct fields measured_flops, measured_flops_verified (bool).

M4. Wire format

Correction after first review pass: DeviceProfileData IS protobuf-backed. Per csrc/backend/server_base.h:214-219, Serialize/Deserialize only accept SerializationFormat::PROTOBUF; the underlying message lives in proto/global_api.proto:39-47:

message DeviceProfileData {
    required uint64 uuid = 1; required uint64 flops = 2; required uint64 memory = 3;
    required uint64 ul_bw = 4; required uint64 dl_bw = 5;
    required uint64 ul_lat = 6; required uint64 dl_lat = 7;
}

This is a proto/*.proto change, which CLAUDE.md §5 flags as requiring explicit user approval before edit. Options:

Option A (chosen, contingent on user approval): append new optional fields (measured_flops = 8, measured_flops_verified = 9, measured_lat_ns = 10, measured_ul_bw_bps = 11, measured_dl_bw_bps = 12). All optional = backward-compatible: older clients ignore unknown fields; newer servers see has_measured_flops() == false for old payloads. No tag number reuse. Update SerializeProto / DeserializeProto in csrc/backend/matrix_partition.cpp:719-791 to gate the new fields with has_* checks. Before starting implementation, ask the user explicitly: "OK to extend proto/global_api.proto per #45 plan §M4 Option A?"

Option B (fallback if proto change is rejected): store the server-measured numbers exclusively server-side. Specifically: extend DeviceTracker with a parallel MeasuredProfile map keyed by device_id, populated by DeviceMeasurementService after registration. Logs include both; no wire-format change. Schedulers that want to use measured values consult the map. Downside: device cannot learn its own measured numbers without an out-of-band channel.

Option C (also fallback): introduce a new top-level proto message DeviceMeasurementReport and a new MessageType for server-to-device delivery. Higher cost; not justified by current consumers; skip unless A and B are both rejected.

M5. Config knobs

New section in config/proxy/server.yaml (introduce file if absent; otherwise extend existing):

device_measurement:
  enable_latency_probe: false   # M1
  enable_bandwidth_probe: false # M2 (requires M1)
  enable_flops_probe: false     # M3 (requires M1)
  latency_probe_rounds: 5
  bandwidth_probe_bytes: 4194304
  flops_probe_sizes: [[1024,1024,1024], [2048,2048,2048]]
  flops_verification_tolerance: 0.001

Loaded in ProxySvrHandle ctor; passed to the new DeviceMeasurementService collaborator.

Atomic implementation steps

Each step lists: edit, tool to verify, command, expected output.

Step 0 — User-approval gate (BLOCKING)

Edit: none.
Tool: human.
Command: ask user "OK to extend proto/global_api.proto per §M4 Option A?"
Expected output: explicit "yes" → proceed with Option A. "no" → switch to Option B (parallel map only) for steps 1, 4, 6.

Step 1 — Proto + serializer extension (Option A only)

Edit: proto/global_api.proto:39-47 — append 5 new optional fields at tags 8-12.
Edit: csrc/backend/server_base.h:204-226 — add 5 new fields on DeviceProfileData struct.
Edit: csrc/backend/matrix_partition.cpp:691-791 — extend SerializeProto, DeserializeProto, DebugString with has_* guards for the new optional fields.
Verify (LSP): lsp_diagnostics on csrc/backend/server_base.h, csrc/backend/matrix_partition.cpp → no new errors.
Verify (build): docker build -t device-emulator:latest . → exits 0.
Verify (round-trip test): add tests/cpp/unit/backend/test_device_profile_roundtrip.cpp (new) using gtest + the existing DeviceProfileData::Serialize/Deserialize. Wire it in tests/cpp/CMakeLists.txt following the test_dispatch_gate pattern at L231-247 (executable + add_test).
Verify (command): docker run --rm device-emulator:latest /app/tests/cpp/build/test_device_profile_roundtrip → "PASSED" line for LegacyFieldsOnly and LegacyPlusMeasured cases.

Step 2 — `DeviceMeasurementService` (server-side coordinator)

Edit: new files csrc/backend/device_measurement.h + csrc/backend/device_measurement.cpp.
Public API: MeasureLatency(conn, profile), MeasureBandwidth(conn, profile), MeasureFlops(conn, profile). Each no-op if its config flag is off.
Uses the existing libevent connection from ProxySvrHandle — no new sockets.
Add the new source to the BACKEND_SRC glob in CMakeLists.txt:148 (already a wildcard csrc/backend/*.cpp, so the file picks up automatically; verify with the build).
Verify (LSP): lsp_diagnostics on the two new files → clean (LSP core/logger.h not-found errors are pre-existing and unchanged).
Verify (build): docker build -t device-emulator:latest . → exits 0. The new symbols link into _Msg (server-side) via the BACKEND_SRC glob.
Verify (symbol presence): docker run --rm device-emulator:latest bash -lc 'nm /app/morphling/_Msg.cpython-*.so | grep -E "DeviceMeasurementService" | head' → at least 3 lines, one per public method. (Verified Dockerfile:63 WORKDIR /app and _setup_helpers.py:93,210 confirming extensions are dropped at /app/morphling/_Msg.cpython-*.so.) Defer behavior validation to step 4.

Step 3 — Device-side probe handlers

Edit: csrc/backend/proxy_cli.cpp — register 3 new opcode handlers OP_PROBE_LAT, OP_PROBE_BW, OP_PROBE_GEMM (constants added in csrc/backend/proto_opcodes.h or wherever opcodes live; verify during impl).
Verify (LSP): lsp_diagnostics clean on the touched files.
Verify (build): docker build → exits 0.
Verify (integration): add tests/cpp/integration/test_device_measurement_loopback.cpp (new), wire into tests/cpp/CMakeLists.txt following test_barrier_integration pattern at L249-264. Test runs server + client in the same process, drives a full M1+M2+M3 sequence, asserts measured values are within +/-30% of synthetic ground truth.
Verify (command): docker run --rm --gpus all device-emulator:latest /app/tests/cpp/build/test_device_measurement_loopback → PASSED.

Step 4 — Server-side wiring

Edit: csrc/backend/proxy_svr.cpp:188-192 — after DEVICE_TRACKER.RegisterDevice(...), if device_measurement.enable_* flags are on, call DeviceMeasurementService in sequence M1→M2→M3 (each gated by its own flag). For Option A, write measured fields back into the protobuf-backed struct via DEVICE_TRACKER.UpdateMeasuredProfile(device_id, measured) (new method). For Option B, store in a parallel std::unordered_map<int64_t, MeasuredProfile> in DeviceTracker.
Verify (LSP): clean.
Verify (logs): rebuild + run scripts/run_devices.py (note: .py, not .sh). Server log MUST contain a line of the form Device <uuid> reported flops=X, measured=Y for each registered device. Capture log excerpt for PR.

Step 5 — Config knob loading

Pre-step probe (one bash command, no edit): docker run --rm device-emulator:latest bash -lc 'grep -rn "yaml\|YAML\|env_or" morphling/common/ csrc/backend/ csrc/core/ 2>/dev/null | head -20'. Read the output to decide between:
- (a) extending an existing YAML loader in morphling/common/config.py, or
- (b) the env_or(...) lambda already used in proxy_cli.cpp:553-557 for env-var defaults.
Edit: in the chosen loader, add 3 new env vars or config keys: MORPHLING_MEASURE_LAT, MORPHLING_MEASURE_BW, MORPHLING_MEASURE_FLOPS, each defaulting to false/0. Wire them into DeviceMeasurementService flags.
Verify (LSP): lsp_diagnostics clean on every file touched in this step.
Verify (parse test): add tests/python/unit/common/test_measurement_flags.py (new) with one test reading the flags via the same loader path and asserting defaults are off.
Verify (command): docker run --rm --gpus all device-emulator:latest python3 -m pytest tests/python/unit/common/test_measurement_flags.py -v → 1 passed.

Step 6 — Source-level marker

Edit: replace # FIXME(#45) at run_device.py:98 with a docstring-style block describing the trust model: device-reported values are advisory; server-side measurement (when enabled) takes precedence. No code logic change.
Verify (grep): git grep "FIXME(#45)" returns empty.

Step 7 — Docs

Edit: extend docs/deployment.md with a "Server-measured device profile" subsection: enumerate the 3 enable flags, what each probes, the accuracy expectations, and the reconciliation policy decision deferred to the follow-up issue.
Verify (codespell, the project's actual doc-lint gate): the project image's Dockerfile:67 only installs requirements.txt (not requirements-lint.txt), so pre-commit is NOT in the image. Run the gate on the host instead: pre-commit run codespell --files docs/deployment.md (executed by the developer or CI in the lint job per the existing .pre-commit-config.yaml; commit e6cfc89 "run pre-commit auto-fixes and exclude generated docs from codespell" confirms this is the gate). Expected output: exits 0.
Verify (web mirror): grep -l "deployment" docs/web/*.html 2>/dev/null — if any file matches, mirror the new subsection; otherwise note in the PR description that there is no HTML mirror to update.

Step 8 — Final E2E

Command: docker build -t device-emulator:latest . && docker run --rm --gpus all device-emulator:latest python3 -m pytest tests -v.
Expected output: same pre-existing failure set as the audit baseline; no new failures attributable to this PR.
Command: docker run --rm --gpus all device-emulator:latest python3 scripts/run_devices.py --measurement on --duration 60s (flag added during step 5; verify name during impl).
Expected output: server stdout contains Device <uuid>: reported flops=<X>, measured=<Y>, verified=<bool> line for each of the N devices launched.

Final verification wave (after all steps)

Build: docker build -t device-emulator:latest . → exits 0.
C++ unit + integration: new gtests from steps 1 and 3 (test_device_profile_roundtrip, test_device_measurement_loopback) are listed in tests/cpp/CMakeLists.txt and pass when run via /app/tests/run_cpp_tests.sh inside the container.
Python suite: docker run --rm --gpus all device-emulator:latest python3 -m pytest tests -v shows no new failures vs. baseline.
Behavioral check (manual, recorded in PR): docker run --rm --gpus all device-emulator:latest python3 scripts/run_devices.py --measurement on (or equivalent flag from step 5); capture server logs showing reported vs. measured for at least 2 devices. Cite log excerpts in PR description.
Regression: same script with measurement off produces a DeviceProfileData::DebugString() line for each device whose content is identical to pre-PR for the 7 legacy fields.

Risk & rollback

Risk: bandwidth/FLOPS probes add startup latency. Mitigation: all flags default off; probes are sequenced (not parallel) to avoid skewing each other.
Risk: malicious device returns wrong echo bytes for M2. Mitigation: server fills the probe buffer with a (device_uuid, seed)-keyed pattern and verifies the echo; mismatch ⇒ probe result discarded, warning logged.
Rollback: revert the patch; defaults are off so reverting cannot regress production.

Follow-up issues (not in this plan)

Reconciliation policy: should flops reported to schedulers be min(reported, measured), measured, or weighted? Open a separate issue once measurement data exists.
Per-link latency asymmetry: requires device cooperation; not addressed.
Long-running re-measurement: periodic re-probe to catch thermal throttling / link degradation.

Progress update

Steps 1 and 2 of the approved plan landed:

Step 1 — wire format: commit 0e4f087. Added 5 optional fields to proto.DeviceProfileData (tags 8-12: measured_flops, measured_flops_verified, measured_lat_ns, measured_ul_bw_bps, measured_dl_bw_bps). Extended the C++ struct + SerializeProto/DeserializeProto/DebugString. Locked in by test_device_profile_roundtrip (3 cases including LegacyFieldsOnly for backward compat).
Step 2 — service skeleton: commit 05e5f80. Added DeviceMeasurementService with MORPHLING_MEASURE_{LAT,BW,FLOPS} env-var gates (all default off). MeasureLatency/MeasureBandwidth/MeasureFlops are no-op returns until the probe wire protocol lands. Verified by test_device_measurement_service (3 cases).

Steps 3-8 (probe opcodes, server wiring, integration test, device-side execution, docs, E2E demo) tracked in #55 to keep this issue focused. Will leave #45 open until #55 lands so the FIXME-to-resolution chain stays linkable.

Closed alongside #55. The trust model gap originally documented here is now addressed by the server-measured device profile pipeline:

Proto: new measured_* fields on DeviceProfileData (commit 0e4f087)
Service skeleton + env-var ctor: commit 05e5f80
Probe wire protocol + device handlers: commits 7fd4599, 6860d74
State machine + session storage + server wiring: commits 73e0ac1, 211e6c2
INI config + Python parse test: commit 1e9f32c
FIXME(Device measurement: trust model needs revisit #45) → NOTE block: commit c84eb80
Deployment doc: commit a655f98
--measurement E2E shim + verified E2E run: commit 7077777

Full chain and E2E result tracked on #55. The seven device-reported legacy fields are preserved untouched; the five new measured_* fields are populated by the server when the operator enables probes. Reconciliation policy (whether schedulers should consume reported, measured, or a blend) is deliberately out of scope and will be filed as a follow-up once enough measured data has accumulated to inform that decision.

feat(proto): add server-measured fields to DeviceProfileData (#45 ste…

feat(backend): add DeviceMeasurementService skeleton (#45 step 2)

feat(proto): add probe wire messages for device measurement (#55 step…

docs(device): replace FIXME(#45) with trust-model NOTE block (#55 ste…

feat(backend): add ProbeMessageCodec + device-side probe handlers (#55 …

chore(code): link 6 high-visibility TODOs to tracking issues

feat(proto): add server-measured fields to DeviceProfileData (#45 ste…

feat(backend): add DeviceMeasurementService skeleton (#45 step 2)

feat(proto): add probe wire messages for device measurement (#55 step…

docs(device): replace FIXME(#45) with trust-model NOTE block (#55 ste…

feat(backend): add ProbeMessageCodec + device-side probe handlers (#55 …

Navigation Menu

Device measurement: trust model needs revisit #45

Issue #45 — Device measurement trust model

Problem (verified against current code)

Scope decision

Out of scope (explicit non-goals)

Design

M1. Server-measured network latency (`ul_lat`, `dl_lat`)

M2. Server-measured bandwidth (`ul_bw`, `dl_bw`)

M3. Server-measured FLOPS via challenge GEMM

M4. Wire format

M5. Config knobs

Atomic implementation steps

Step 0 — User-approval gate (BLOCKING)

Step 1 — Proto + serializer extension (Option A only)

Step 2 — `DeviceMeasurementService` (server-side coordinator)

Step 3 — Device-side probe handlers

Step 4 — Server-side wiring

Step 5 — Config knob loading

Step 6 — Source-level marker

Step 7 — Docs

Step 8 — Final E2E

Final verification wave (after all steps)

Risk & rollback

Follow-up issues (not in this plan)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Device measurement: trust model needs revisit #45

Description

Activity

drunkcoding commented on May 30, 2026

Issue #45 — Device measurement trust model

Problem (verified against current code)

Scope decision

Out of scope (explicit non-goals)

Design

M1. Server-measured network latency (ul_lat, dl_lat)

M2. Server-measured bandwidth (ul_bw, dl_bw)

M3. Server-measured FLOPS via challenge GEMM

M4. Wire format

M5. Config knobs

Atomic implementation steps

Step 0 — User-approval gate (BLOCKING)

Step 1 — Proto + serializer extension (Option A only)

Step 2 — DeviceMeasurementService (server-side coordinator)

Step 3 — Device-side probe handlers

Step 4 — Server-side wiring

Step 5 — Config knob loading

Step 6 — Source-level marker

Step 7 — Docs

Step 8 — Final E2E

Final verification wave (after all steps)

Risk & rollback

Follow-up issues (not in this plan)

drunkcoding commented on May 30, 2026

drunkcoding commented on May 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

M1. Server-measured network latency (`ul_lat`, `dl_lat`)

M2. Server-measured bandwidth (`ul_bw`, `dl_bw`)

Step 2 — `DeviceMeasurementService` (server-side coordinator)