An edge-native AI wearable system that connects Meta Ray-Ban Smart Glasses to a smartphone relay, which tunnels over a Tailscale mesh VPN to an NVIDIA DGX Spark (GB10 Grace Blackwell) running all AI inference locally. The system provides real-time spoken alerts about food safety, street conditions, transit disruptions, pedestrian hazards, filming locations, architecture, and neighborhood context by combining continuous voice transcription, intermittent camera vision, and live NYC Open Data.

No paid cloud compute. No third-party AI APIs. All inference runs on-device.

Meta Ray-Ban Glasses (visual or audio)
        |
        v
iOS Application (NemoAssistant)
  - Captures PCM audio from glasses mic via AVAudioSession (BluetoothHFP)
  - Streams audio frames over WebSocket to middleware
  - Sends GPS location updates every location change
  - Receives text responses and speaks them via AVSpeechSynthesizer
  - Receives proactive alert notifications every 5 minutes
        |
        | WebSocket (Acer Veriton, ws://[tailscale-ip]:8080)
        | Tailscale mesh VPN (WireGuard)
        v
NVIDIA DGX Spark — Ubuntu 22.04, Docker Compose, metaglass-net bridge
  |
  |-- middleware (Node.js 20)
  |     WebSocket server, ASR client, NemoClaw HTTP client,
  |     proactive alert engine, cooldown cache
  |
  |-- riva-asr (NeMo Parakeet TDT 0.6B v3)
  |     Speech-to-text, 16kHz PCM input, HTTP /transcribe
  |
  |-- nemoclaw (Python 3.11, FastAPI)
  |     Intent classification, tool dispatch, response synthesis
  |     Calls user-profile for personalization scoring
  |
  |-- nemotron-nim (Nemotron-3-Nano-30B-A3B via llama-cpp-python)
  |     LLM inference, OpenAI-compatible HTTP API on port 8000
  |     Full GPU offload on GB10, n_ctx=4096
  |
  |-- user-profile (Python 3.11, FastAPI)
  |     Behavioral personalization engine
  |     Tracks place visits, transit patterns, alert engagement
  |     Computes priority scores and suppression decisions
  |
  |-- postgres-postgis (PostgreSQL 15 + PostGIS 3.4)
  |     Spatial restaurant data, collision hotspots, landmarks
  |     Film permit data, Wikidata productions, subway entrances
  |     User behavioral data (place_visits, transit_patterns, etc.)
  |
  |-- data-pipeline (Python 3.11)
        Nightly ingestion of NYC Open Data and Wikidata
        APScheduler cron at 02:00 UTC

• NVIDIA DGX Spark — Acer Veriton GN100 with NVIDIA GB10 Grace Blackwell SoC
• GPU — NVIDIA GB10, 128GB unified memory, CUDA 13.0, Driver 580.142
• Storage — 3.67TB NVMe
• Network — Tailscale mesh VPN (WireGuard) for secure phone-to-server tunnel

Implements the WebSocket protocol so the iOS app connects without modification. Handles:

• Binary and JSON WebSocket message demultiplexing
• PCM audio accumulation and forwarding to Parakeet ASR via HTTP
• NemoClaw HTTP client for agent queries
• Proactive alert engine: fires every 5 minutes per connected client, fetches situation report, formats as bullet points, pushes via WebSocket
• Admin endpoint POST /admin/alert for manual alert triggering
• Per-client state tracking (GPS coordinates, processing state)
• Cooldown cache for manual alert deduplication (30-minute TTL)

FastAPI service implementing the full query pipeline:

• Intent classification via Nemotron LLM with 12 distinct intents
• Tool dispatch to 10 specialized query functions
• Off-topic detection: general knowledge questions answered directly by LLM without touching any dataset
• Non-local city detection for food queries
• Two-step synthesis: Nemotron for analysis, Ollama/Nemotron for final spoken output
• User-profile integration: records location, place visits, transit patterns; checks priority scores before surfacing alerts
• Positive reinforcement signal posting after every successful response

Supported intents and their tools:

Behavioral personalization engine running on port 8081:

• Tracks place visits, transit patterns, neighborhood familiarity (H3 resolution 9), alert engagement, and interest weights
• Computes priority scores using engagement ratio, interest weights, visit penalties, and familiarity bonuses
• Suppresses alerts below a configurable threshold (default 0.2)
• Proactive transit surfacing: flags routes with 5+ visits at the same hour/day
• Nightly cleanup job at 03:00 UTC removes data older than configurable retention window (default 90 days)
• All data remains on-device; no external transmission

APScheduler-based nightly ingestion at 02:00 UTC:

• Restaurant inspections: fetches CSV from Socrata, drops no-violation rows, upserts with PostGIS geometry
• Vision Zero collisions: fetches JSON, filters to last 3 years, upserts with geometry
• Wikidata productions: SPARQL query for NYC filming locations by production type, paginated in batches of 500, filters generic city-centroid coordinates
• All ingesters use psycopg2 transactions with rollback on failure

The proactive alert system pushes location-based notifications to the iOS app without user interaction:

• Timer starts 15 seconds after WebSocket connection (allows GPS to arrive)
• Fires every 5 minutes while connected, regardless of content repetition
• Fetches full situation report: MTA alerts, 311 complaints, collision hotspots, edge case alerts
• Formats results as bullet points
• Sends via WebSocket as serverContent.proactiveAlert.bullets
• iOS app renders bullets as a local UNUserNotificationCenter notification

Manual trigger available via POST /admin/alert on the middleware HTTP server.

The user-profile service implements a local reinforcement learning loop:

• Every agent request records a GPS location visit (H3 cell at resolution 9)
• Restaurant queries record place visits and check suppression scores
• Transit queries record route patterns and check proactive flags
• Successful responses post positive engagement signals and increment interest weights
• Interest weights (food, transit, safety) adjust alert priority scores over time
• Neighborhood familiarity reduces discovery alert verbosity in frequently visited areas

Requirements: Docker, Docker Compose, Tailscale, NVIDIA GPU with CUDA 13.0+

cp .env.example .env
# Fill in DATABASE_URL, MTA_API_KEY, SOCRATA_APP_TOKEN, TAILSCALE_GN100_IP

docker compose up -d

Services start in dependency order. The user-profile service runs Alembic migrations on startup. The data pipeline runs its first ingestion at 02:00 UTC.

Health checks:

curl http://localhost:8081/health   # user-profile
curl http://localhost:8080/health   # middleware
curl http://localhost:8090/health   # nemoclaw

Manual alert trigger:

curl -X POST http://[tailscale-ip]:8080/admin/alert

pseudo-meta-glass/
├── docker-compose.yml
├── .env.example
├── services/
│   ├── middleware/          Node.js WebSocket server and alert engine
│   ├── nemoclaw/            Python agent with tool dispatch
│   ├── user-profile/        Python personalization service
│   ├── riva-asr/            NeMo Parakeet ASR HTTP server
│   └── riva-tts/            NeMo FastPitch TTS HTTP server (unused, replaced by iOS AVSpeechSynthesizer)
└── data/
    └── human-impact/        Data pipeline and ingesters