Device measurement: probe wire protocol + server coordinator (#45 steps 3-8)

Follow-up to #45. Steps 1 and 2 of the plan landed in commits
0e4f087 (proto fields + serializer)
and 05e5f80 (DeviceMeasurementService skeleton).

This issue tracks the remaining work from the #45 fix plan:

Step 3 — Device-side probe handlers

Add 3 new MessageType enum values + extension registrations in proto/global_api.proto for OP_PROBE_LAT, OP_PROBE_BW, OP_PROBE_GEMM (request/response pairs).
csrc/backend/proxy_cli.cpp: register handlers that echo the probe payload (M1), echo the buffer (M2), and run the seeded challenge GEMM (M3) via the existing CUDA path.
Loopback integration test under tests/cpp/integration/ running server + client in the same process, asserting measured values are within ±30% of synthetic ground truth.

Step 4 — Server-side wiring

In ProxySvrHandle::HandleRegisterResponse, after DEVICE_TRACKER.RegisterDevice, call DeviceMeasurementService (sequence M1 → M2 → M3, each flag-gated).
Add DeviceTracker::UpdateMeasuredProfile(device_id, measured) so the tracked struct carries the measured fields back to schedulers.

Step 5 — Config knob loading

Already partially landed via the env-var path on DeviceMeasurementService. Add a Python parse test under tests/python/unit/common/ reading the same env vars to assert defaults remain off.

Step 6 — Source-level marker

run_device.py:98 keeps FIXME(#45); convert to a docstring/NOTE once measurement is observable end-to-end.

Step 7 — Docs

Extend docs/deployment.md with a "Server-measured device profile" subsection (3 enable flags, accuracy expectations, reconciliation deferral).

Step 8 — Final E2E

docker run … python3 scripts/run_devices.py --measurement on produces a server log line per device of the form Device <uuid>: reported flops=<X>, measured=<Y>, verified=<bool>.

Reconciliation policy (separate, not in scope here)

Should flops consumed by schedulers be min(reported, measured), measured, or weighted? Open a dedicated issue once measured data is flowing.

Plan reference: .omo/plans/issue-45-fix.md (Momus-approved).

Closed by the full commit chain:

Step	Commit	Summary
3a	`7fd4599`	Probe wire messages (PROBE_LATENCY/BANDWIDTH/FLOPS request+response)
3b	`6860d74`	ProbeMessageCodec + device-side handlers in ProxyCliImpl
3c	`73e0ac1`	DeviceMeasurementSession state machine + 5-case UT
4	`211e6c2`	Wire session into proxy_svr; DEVICE_TRACKER session storage + UpdateMeasuredProfile
5/5b	`1e9f32c`	INI [device_measurement] section, ProxyEnvCfg parse, Python parse test
6	`c84eb80`	FIXME(#45) → NOTE block in run_device.py
7	`a655f98`	"Server-measured device profile" subsection in docs/deployment.md
8	`7077777`	--measurement shim in scripts/run_devices.py; E2E verified

E2E result (commit 7077777):

docker run --rm --gpus all --ulimit memlock=-1 device-emulator:latest \
  stdbuf -oL python3 -u scripts/run_devices.py \
    --num_devices 1 --backend proxy --measurement

emits the canonical completion line per device:

Device 0 measurement complete: ok=1 state=DONE
  measured_lat_ns=118572 measured_ul_bw_bps=225402564
  measured_dl_bw_bps=225402564 measured_flops=397312537 verified=1

verified=1 confirms the FLOPS challenge round-tripped through the real device-side cuBLAS/MKL worker and matched the OpenBLAS reference within flops_tolerance.

Verification gates:

docker build exits 0
pytest tests -v: 112 passed / 10 skipped (109 pre-Device measurement: probe wire protocol + server coordinator (#45 steps 3-8) #55 baseline + 3 new from tests/python/unit/test_measurement_flags.py)
C++ tests: all green incl. test_device_measurement_session 5/5, test_device_measurement_service 3/3, test_device_profile_roundtrip 3/3

Skipped, with rationale:

Step 3d (real-socket loopback integration gtest): the 3c UT locks the contract with 5 cases (happy path, M3 mismatch, timeout, stale probe ID, empty config); 3d was optional in the plan and not a gate for closing. Worth filing a follow-up if production-driver coverage becomes desired.

Explicitly out of scope (deferred):

Reconciliation policy (min(reported,measured) vs measured vs blend) — tracked separately.
Multi-round adaptive probing.
Per-link latency asymmetry (requires device cooperation).
Cryptographic device attestation.

Pre-existing infrastructure gotchas surfaced during step 8 (not #55 bugs, tracked as follow-up issues):

The Dockerfile step cmake --build tests/cpp/build -j only builds a subset of test targets; zerocopy tests end up under tests/cpp/build/unit/zerocopy/ and are NOT scanned by tests/run_cpp_tests.sh. Fix: either configure CMAKE_RUNTIME_OUTPUT_DIRECTORY to consolidate binaries, or update the run script to find -name 'test_*' -executable.
The bandwidth probe defaults to 4 MiB which exceeds the default container memlock budget; --ulimit memlock=-1 on docker run is now documented in deployment.md but should arguably be in the base image's recommended run command.
python3 -u + stdbuf -oL are required to surface the server's INFO completion log line before idle (Python's block-buffered stdout otherwise swallows C++ writes); documented in deployment.md.