NFLTR›Blog

Relay metrics on a laptop, through nfltr

The nfltr.xyz relay exports Prometheus metrics on a port that listens only on its VM's loopback. We wanted graphs and alerts over weeks without opening that port to the internet, without a hosted monitoring service, and without copying the data anywhere but the operator's laptop. The answer was nfltr itself: an owner-only agent on the server, nfltr tcp-connect on the laptop, and Prometheus and Grafana in Docker next to it. This post shows the setup with placeholders, so it works for any server with a Prometheus endpoint, and the one mistake we nearly made with labels.

The shape

  • On the server, an nfltr up agent with one TCP service that forwards to the metrics port on loopback. It connects out to the relay; nothing listens on a public address.
  • On the laptop, three containers: a small sidecar that runs nfltr tcp-connect to that service and serves the metrics to Prometheus, Prometheus itself, and Grafana. Their host ports bind to 127.0.0.1 only.

Only keys of the account that owns the agent reach it. A TCP service gets no share URL, and the agent dials only the targets in its file. The scraped data lives in a Docker volume on the laptop.

On the server

A config file for nfltr up, with your metrics port in place of <port>:

agent: <server>
services:
  metrics: { tcp: <port>, forward: "127.0.0.1:<port>" }
nfltr up -c nfltr.yaml

On nfltr.xyz this is one service of the agent we use for operator access to the VM, described in nfltr on nfltr, part 2. Run it as a service with its key in a file only that service can read, not on a command line.

One detail to know: the relay resolves the agent name in a tcp-connect request inside the calling key's own namespace. If the laptop and the server use the same key, or the server's key was minted on the dashboard under the laptop key's name, the laptop reaches the agent by name. If they use two unrelated keys of the same account, it does not yet.

Check it by hand

nfltr tcp-connect <agent>/metrics --listen :9182
curl -s localhost:9182/metrics | grep -v '^#' | head

If that prints samples, the tunnel works. Everything below automates it.

On the laptop: three containers

A trimmed version of our compose file. The sidecar image is Debian slim with curl and socat, and nfltr installed by the public install.sh, which verifies its signature. The API key is a 0600 file mounted as a compose secret, so it is never in the image, the compose file or a command line.

name: metrics
services:
  sidecar:
    build: .
    environment:
      METRICS_AGENT: <agent>
    secrets: [{source: nfltr_env, target: nfltr.env}]   # NFLTR_API_KEY=...
    cpus: 0.25
    mem_limit: 128m
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:<version>@sha256:<digest>
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.retention.time=30d
      - --storage.tsdb.retention.size=2GB
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports: ["127.0.0.1:9090:9090"]
    cpus: 0.5
    mem_limit: 512m
    restart: unless-stopped

  grafana:
    image: grafana/grafana:<version>@sha256:<digest>
    environment:
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin_password
      GF_AUTH_ANONYMOUS_ENABLED: "false"
      GF_USERS_ALLOW_SIGN_UP: "false"
    secrets: [grafana_admin_password]
    volumes: [grafana-data:/var/lib/grafana]
    ports: ["127.0.0.1:3000:3000"]
    cpus: 0.5
    mem_limit: 640m
    restart: unless-stopped

secrets:
  nfltr_env: {file: ~/.config/metrics/nfltr.env}
  grafana_admin_password: {file: ~/.config/metrics/grafana-admin-password}

volumes:
  prometheus-data:
  grafana-data:

The sidecar's entrypoint loads the key from the secret file into its environment, then runs two things and exits when either stops, so Docker restarts the pair:

set -a; . /run/secrets/nfltr.env; set +a
nfltr tcp-connect "$METRICS_AGENT/metrics" --listen :9182 &
socat TCP-LISTEN:9180,fork,reuseaddr EXEC:/usr/local/bin/serve-metrics &
wait -n

serve-metrics answers one scrape: it fetches /metrics through the tunnel, passes it through a short awk filter (next section), and answers 503 when the server is unreachable, so Prometheus marks the target down instead of storing nothing quietly.

# prometheus.yml
global:
  scrape_interval: 30s
  scrape_timeout: 25s
scrape_configs:
  - job_name: <job>
    static_configs:
      - targets: [sidecar:9180]
    metric_relabel_configs:
      - action: labeldrop
        regex: agent_id|owner|account|user|email|ip|remote_addr|peer|key_id|tenant

Sum the labels, then drop them

nfltr.xyz serves other accounts, and the relay labels some series by agent with agent_id, which holds other accounts' agent names. Those names should not be stored on our laptop. The obvious fix is a labeldrop in prometheus.yml. On its own it is wrong.

Series that differ only in agent_id become identical once the label is dropped. Prometheus then sees duplicate series in one scrape and keeps one sample, not their sum. The connected-agents gauge has one series per agent, so with a labeldrop alone it would read 1 however many agents were connected, and every per-agent counter would undercount the same way.

So the sidecar sums first. Its awk filter removes the identifying labels from each sample and adds up the samples they told apart; counters, gauges and histogram buckets, sums and counts all add. The labeldrop in prometheus.yml stays, with the same list, as a second line of defence, and a test checks the two lists are equal. The test's own fixture shows the difference:

# in
c{agent_id="other.acct",pod_id="p"} 1
c{agent_id="mine.vm",pod_id="p"} 1
h_bucket{agent_id="a",route="/{$}",le="+Inf"} 2
h_bucket{route="/{$}",agent_id="b",le="+Inf"} 3

# out
c{pod_id="p"} 2
h_bucket{route="/{$}",le="+Inf"} 5

Our list covers agent_id, which is the only identifying label the relay exports today, plus account, user, email, address, key and tenant names that it might add later. When it adds one, it goes into both lists, and a status check fails if Prometheus stores any listed label name.

Summing has a cost. When an agent disconnects, its share of a summed counter goes with it, and rate() reads the drop as a counter reset. Where the server exports an unlabelled total, such as HTTP requests or process network bytes, the dashboard uses that instead.

Dashboard and alerts

Grafana starts with a provisioned Prometheus data source and one dashboard: scrape health, uptime, connected agents, goroutines and memory; HTTP request rate, errors and p95 latency by route; gRPC streams by method and code; throughput; TCP, SSH, SNI, terminal and agent-to-agent sessions; garbage collection, CPU and open files. Anonymous access, sign-up and update checks are off.

Prometheus evaluates alert rules with no notifier: the target down for five minutes, a restart, goroutines, heap or memory growing over six hours, more than 5 % HTTP 5xx, and gRPC errors above 20 % of finished streams. They show at localhost:9090/alerts. The thresholds are starting points.

The same tunnel also carries Go's profiler when the server enables it:

go tool pprof http://localhost:9182/debug/pprof/heap

A CPU profile costs the server CPU while it runs, so it is only for when you need it.

What went wrong, and the limits

  • Grafana's first memory limit was too tight. It idled at 332 MiB of a 384 MiB limit, so the limit is now 640 MiB. Prometheus stays at 512 MiB and the sidecar at 128 MiB.
  • The metrics share the relay's fate. The agent connects through the relay it reports on. If that relay is down, the target goes down, an alert fires, and you look at the server another way; for nfltr.xyz that is the cloud provider's SSH. For a server that is not the relay itself, this does not apply.
  • No version metric. The relay exports no build-info metric, so the dashboard shows Go's version; the relay's own version is at https://nfltr.xyz/api/v1/version.
  • The label list is maintained by hand. A new identifying label on the server is stored until someone adds it to both lists; the check catches only names already on the list.
  • It runs while the laptop does. A sleeping laptop leaves gaps in the graphs.

For the tunnel commands themselves, SSH to a machine behind NAT shows nfltr tcp-connect with other TCP services, such as a database.

← All posts