EasyMonitor Probe Node

A lightweight Go binary that executes monitoring checks on behalf of an EasyMonitor server. Probes pull work from the server's Redis Streams, run HTTP/ICMP/TCP checks, and publish results back.

Requires an EasyMonitor server v0.2.0 or newer for TCP checks. Probes skip check types they do not recognize, so running an older probe against a newer server degrades gracefully (those monitors are simply not checked by that probe).

Deploy one or more probes across different regions/networks. The server applies cross-probe quorum so alerts only fire when a majority of probes agree.

Pre-built images are published for every release:

• Docker Hub: easymonitor/probe-node:latest
• GitHub Container Registry: ghcr.io/easymonitordev/probe-node:latest

Both are multi-arch (linux/amd64, linux/arm64) and built from the same source.

• An EasyMonitor server already running (main repo)
• A probe JWT token generated on that server
• The server's Redis URL and password (from its .env)
• A network path from this host to the server's Redis — typically via Tailscale or Cloudflare Tunnel. Never expose Redis over plaintext on the public internet.

Generate a token on the EasyMonitor server:

docker compose exec php php artisan probe:generate-token \
  --node-id=us-east-1 \
  --tags=us-east-1,production \
  --expires=365

Copy the printed token.

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

Join the same tailnet your EasyMonitor server is on. Verify:

ping -c 2 <server-tailscale-ip>

docker run -d \
  --name easymonitor-probe \
  --restart unless-stopped \
  --network host \
  -e NODE_ID="us-east-1" \
  -e REDIS_URL="redis://<server-tailscale-ip>:6379/0" \
  -e REDIS_PASSWORD="<redis password from server .env>" \
  -e JWT_TOKEN="<probe token from step above>" \
  easymonitor/probe-node:latest

--network host is required so the container shares the host's tailscale0 interface — without it, the container's own network namespace can't reach the tailnet. Port 8080 is automatically exposed on the host, no -p needed.

curl -s http://localhost:8080/health
# {"status":"healthy","node_id":"us-east-1"}

On the EasyMonitor server, confirm the probe registered:

docker compose exec redis redis-cli -a "$REDIS_PASSWORD" --no-auth-warning \
  XINFO GROUPS checks

You should see a consumer group named probe-us-east-1.

• Server side: cloudflared tunnel create + ingress config exposing tcp://localhost:6379 under a hostname
• Probe side: cloudflared access tcp --hostname=... --url=127.0.0.1:6379 + Cloudflare service token
• Run probe with --network host and REDIS_URL=redis://127.0.0.1:6379/0

Full instructions are in the main repo's PROBE_NODE_SETUP.md.

Any private network path works. Point REDIS_URL at the appropriate private address.

ssh -L 6379:127.0.0.1:6379 -N user@server &

docker run -d \
  --name easymonitor-probe \
  --restart unless-stopped \
  --network host \
  -e NODE_ID="eu-west-1" \
  -e REDIS_URL="redis://127.0.0.1:6379/0" \
  -e REDIS_PASSWORD="..." \
  -e JWT_TOKEN="..." \
  easymonitor/probe-node:latest

• GET /health — 200 if the probe is consuming, 503 otherwise
• GET /ready — same semantics as /health
• GET /version — build version and timestamp

                ┌──────────────────────────┐
                │  EasyMonitor server      │
                │  (Laravel + Redis)       │
                └──────────┬───────────────┘
                           │
        private tunnel     │  (Tailscale / Cloudflare / VPN)
                           │
       ┌───────────────────┼───────────────────┐
       ▼                   ▼                   ▼
┌──────────┐         ┌──────────┐         ┌──────────┐
│ probe    │         │ probe    │         │ probe    │
│ us-east  │         │ eu-west  │         │ ap-south │
└──────────┘         └──────────┘         └──────────┘

Each probe uses a unique Redis Streams consumer group (probe-<NODE_ID>) so every check is delivered to every probe. The server groups per-probe results by a round_id and decides monitor status by majority vote.

Requires Go 1.24+.

git clone https://github.com/easymonitordev/probe-node.git
cd probe-node
make build
./bin/probe-node

make build-linux         # linux/amd64
make build-linux-arm     # linux/arm64

make test                # -race -coverprofile=coverage.out
make test-coverage       # generate coverage.html

make docker-build                 # single-arch
make docker-build-multiarch       # amd64 + arm64 via buildx

Create /etc/systemd/system/easymonitor-probe.service:

[Unit]
Description=EasyMonitor Probe
After=network.target

[Service]
Type=simple
User=probe
EnvironmentFile=/etc/easymonitor/probe.env
ExecStart=/usr/local/bin/probe-node
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

/etc/easymonitor/probe.env:

NODE_ID=us-east-1
REDIS_URL=redis://100.x.y.z:6379/0
REDIS_PASSWORD=...
JWT_TOKEN=...

sudo systemctl enable --now easymonitor-probe

A minimal manifest:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: easymonitor-probe
spec:
  serviceName: probe
  replicas: 1
  selector:
    matchLabels:
      app: easymonitor-probe
  template:
    metadata:
      labels:
        app: easymonitor-probe
    spec:
      containers:
        - name: probe
          image: easymonitor/probe-node:latest
          env:
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: easymonitor-probe
                  key: redis_url
            - name: REDIS_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: easymonitor-probe
                  key: redis_password
            - name: JWT_TOKEN
              valueFrom:
                secretKeyRef:
                  name: easymonitor-probe
                  key: jwt_token
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080

The probe is stateless: all configuration lives in environment variables, and all runtime state (the consumer group, delivery positions) lives in the server's Redis. There are no local keys or data files to preserve, and no re-registration step.

• Keep the same JWT_TOKEN. The token is tied to the server's JWT_SECRET and its own expiry date, not to the probe version. It remains valid across upgrades.
• Keep the same NODE_ID. The probe re-attaches to its existing consumer group (probe-<NODE_ID>) on the server automatically. Changing NODE_ID creates a new consumer group and leaves the old one orphaned in Redis.
• Upgrade the server first. New check types (e.g. TCP and SSL expiry, added in v0.2.0) require a matching or newer server. A newer probe against an older server is harmless — the server simply never enqueues check types it doesn't know about.
• Downtime during the swap is expected and safe. On startup the probe resets its read position to the stream tail, so checks that were enqueued while it was down are skipped rather than replayed. With multiple probes, quorum covers the gap.

upgrade.sh automates the whole swap: it reads the running container's configuration (env vars, network mode, restart policy, port mappings), pulls the new image, replaces the container, and verifies /health and /version.

curl -fsSLO https://raw.githubusercontent.com/easymonitordev/probe-node/main/upgrade.sh
chmod +x upgrade.sh
./upgrade.sh                                # upgrade to :latest
./upgrade.sh easymonitor/probe-node:0.2.0   # or pin a specific version

If your container isn't named easymonitor-probe, set PROBE_CONTAINER=<name>. If the new probe fails its health check, the script prints a rollback command that restarts the previous image.

If you don't remember the original docker run flags, capture them before removing the container:

docker inspect easymonitor-probe --format '{{json .Config.Env}}'

Then:

docker pull easymonitor/probe-node:latest   # or pin a version, e.g. :0.2.0
docker stop easymonitor-probe
docker rm easymonitor-probe

Then start a fresh container from the newly pulled image with the same docker run command you used at install time — for the Tailscale setup:

docker run -d \
  --name easymonitor-probe \
  --restart unless-stopped \
  --network host \
  -e NODE_ID="us-east-1" \
  -e REDIS_URL="redis://<server-tailscale-ip>:6379/0" \
  -e REDIS_PASSWORD="<same password as before>" \
  -e JWT_TOKEN="<same token as before>" \
  easymonitor/probe-node:latest

:latest resolves to whatever you last pulled, so the docker pull above is what actually swaps the version.

Build or download the new binary, replace it, and restart:

sudo systemctl stop easymonitor-probe
sudo cp bin/probe-node /usr/local/bin/probe-node
sudo systemctl start easymonitor-probe

/etc/easymonitor/probe.env needs no changes.

curl -s http://localhost:8080/version
curl -s http://localhost:8080/health

/version should report the new build; /health should return {"status":"healthy",...} once the probe is consuming.

Tokens should be rotated periodically (every 90–365 days recommended). On the server:

docker compose exec php php artisan probe:generate-token \
  --node-id=us-east-1 \
  --expires=365

Update the token on the probe and restart the container.

"AUTH failed" in logs — REDIS_PASSWORD doesn't match the server. Check it's copied exactly from the server's .env.

"failed to validate token" — token expired or was generated with a different JWT_SECRET. Regenerate on the server.

Probe starts but no checks run — confirm Horizon is running on the server and that there are active monitors. Also verify the probe shows up: docker compose exec redis redis-cli -a "$REDIS_PASSWORD" XINFO GROUPS checks.

Can't reach Redis — verify the tunnel: tailscale status / systemctl status cloudflared. Test connectivity: redis-cli -h <host> -a "$REDIS_PASSWORD" ping should return PONG.

See CONTRIBUTING.md for the development setup and PR guidelines.

MIT — see LICENSE.

Part of the EasyMonitor project.