nfltr on nfltr, part 2

October 10, 2026

In nfltr.xyz now runs on nfltr the public sites moved out of the relay and onto an ordinary nfltr agent. Since then four more parts of running nfltr.xyz have moved onto nfltr itself: a status page built from the hourly canary, operator access to the production VM, a Linux machine for agents and the canary, and the release process, which a hub now runs with every production step approved by the owner's fingerprint. This post covers each one, the bugs the first hub-run release found, and the limits.

status.nfltr.xyz, built from the canary

A canary runs every hour on the operator's Mac. It starts a fresh hub on the hosted relay, spawns an agent on each canary node (one of them pushes a branch), runs a monitor, and checks that every completion arrives exactly once, that the pushed commit is on the origin and that nothing is left behind. The agents are a scripted stand-in for Claude, so a run costs no model calls. Each run appends one JSON line to a log that stays on that Mac.

After each run, the canary writes a small typed summary beside the log: the newest run and its result, the newest relay version, the success rate over 24 hours and 7 days, the time from spawn to first progress and to completion (p50 and p95), and up to five recent failures as reason codes. A failure reason the page does not know becomes other. A run's details, its hub, nodes, machine names and agent ids never go into the summary, and the site build refuses a summary with unknown fields or reasons. The build renders the page to static HTML, and the same site agent that serves the other sites serves it at status.nfltr.xyz.

A static page can go stale without anyone noticing, so it says when it was made and marks itself stale when its newest run is more than two hours old. It does that twice: when it is built, and with a six-line script in your browser, so a page that stopped being republished turns stale on its own instead of showing an old pass.

We rejected having the relay store or serve canary results (the relay keeps no view of runs), serving the page from the Mac (a laptop sleeps, and the page would vanish exactly when it matters), and a hosted status service (the data would leave our machines). The limit is plain on the page itself: one machine runs the canary, so a gap means that machine was offline, not that the relay was down.

Operator access to the production VM

Until now the only way onto the relay's VM was the cloud provider's identity-aware SSH tunnel (IAP). Now a second agent runs on the VM, next to the relay: an nfltr up agent with two TCP services and nothing else.

agent: access
services:
  ssh: { tcp: 22, forward: "127.0.0.1:22" }        # the VM's own sshd
  metrics: { tcp: 8082, forward: "127.0.0.1:8082" } # the relay's metrics listener

The shell goes through the VM's own OpenSSH server, with the same login and key as before. The agent is only a pipe; sshd still authenticates every session. From the laptop, ssh uses nfltr ssh-proxy as its proxy command, with the host key pinned, so the relay sees only SSH ciphertext. The metrics service is how the relay's /metrics reaches Prometheus and Grafana on the operator's laptop, with no port open to the internet.

Only keys of the owner account reach the agent. It has no HTTP service, so it has no share URL, and the agent dials only the two loopback ports in its file. Its systemd unit runs as a dynamic user with a read-only filesystem, no capabilities, the relay's secrets and the site's files hidden, memory and CPU caps, and an OOM score that makes the kernel kill it before the relay. Its key comes from the cloud's secret store into a root-only file read by systemd before it drops privileges; it is never on a command line or in a log.

One relay behaviour shaped the setup. The relay resolves the agent name in a tcp-connect or ssh-proxy request inside the calling key's own namespace. So a laptop key cannot name an agent that runs under another key of the same account, although the ownership rule would allow it. The VM's key is therefore minted on the dashboard under the laptop key's name. A follow-up will let any key of the owning account reach the account's agents.

The cloud provider's SSH stays as break-glass. The access agent goes down with the relay by construction, so every deploy, rollback and recovery script still uses IAP, and none of them uses the access agent. A local end-to-end test runs the shipped config against a relay with real accounts: the owner gets a shell and metrics, a key sshd does not know is refused, another account's key and an unknown key get 404 and 401, and sshd sees only the owner's connections. The production verify step checks the same from both sides.

We rejected the embedded nfltr shell server with sudo (it needs setuid inside the sandbox of a process the relay can reach, and a second SSH implementation), a WireGuard tunnel for one port, and a share link, which is a link for other people.

A Linux node on the Mac

Everything above runs from a Mac, and nfltr ships for Linux too. So the Mac's Docker VM now runs two long-lived Debian 12 containers, each an nfltr node joined to nfltr.xyz on the operator's account:

• A node for real Claude Code agents, available to any hub of the account: two agents at a time, every tool, the Go toolchain the module names, git, make and sudo, and the network privileges kernel WireGuard and TUN tests need. The operator's repository is mounted read only; an agent clones from the node's mirror, builds and tests, and returns its result or a patch, because it cannot push there. A hub reaches it with constraints.machine.
• A canary node with the scripted stand-in for Claude. While its container runs, every hourly canary run crosses macOS and Linux through the hosted relay. When the container is down, the run goes on with the Mac nodes and records a typed skip reason (docker_unavailable or linux_node_down); a skip is not a failure and never reaches the status page.

The image installs the released nfltr with the public install.sh, which verifies its signature; it is never built from source. The API key and the Claude token are written to 0600 files and mounted as secrets, never into the image, the compose file or a command line. A guard test checks that offline and, while the containers run, against the image layers, docker inspect, the container logs and every process's arguments. Idle, each node uses under 1 % CPU and about 10 MB.

Releases run by a hub

A release is now a hub goal, "release vX.Y.Z". The hub, the operator's own Claude Code session with the hub tools added, runs it on a build node on the operator's Mac. nfltr provides the pieces; the plan lives in a short procedure the hub is given (2.9 KiB of operator-owned text), and every decision in a run, whether to go on, retry, ask the owner or roll back, is the hub's. Nothing in Go encodes the plan.

• One step per agent turn. A step script runs one step and prints one RELEASE_STEP JSON line: the step, the version, ok, the exit code and its facts. The hub judges a step only by that line. The steps are preflight, gates, publish, relay, site, verify, canary, status and summary, plus rollback and site rollback.
• Checks, not decisions. The relay step refuses to run until the version's downloads are published, because the relay serves install.sh, which installs the newest published version. Publishing a version again re-uploads the same bytes or refuses different ones; deploying the live version again keeps the rollback target; a rollback acts once; a second production step for the same version waits while one runs.
• The node's tool policy. Production steps run only through the step script and only with the owner's grant for that exact step. Direct deploy commands, the cloud CLI, shells, interpreters, file changes, git writes and credential reads are denied in every turn, approved or not.

A grant is an Ed25519 signature over the node, the exact step prompt and an expiry, spent on its first use. The key stays in the operator's macOS login keychain, and a small Swift helper signs with it only after Touch ID; there is no password fallback. The dialog shows the step, the node, the expiry and the prompt's SHA-256, the same digest the hub's approval request shows. The owner can sign each production step when the hub asks, or sign them all before the run.

This works without a paid Apple developer entitlement, with a compromise. Having the keychain itself demand Touch ID needs an entitlement only a paid developer profile can grant, and the Secure Enclave has no Ed25519 keys. So the item lives in the file-based login keychain with an access list that trusts only the helper's code hash, and the helper checks Touch ID itself under the hardened runtime. Checked by hand on the operator's Mac: 8 checks passed, 0 failed. Another program asking for the key got a keychain dialog and was denied, a re-signed copy of the helper was refused, and a cancelled Touch ID produced no grant. The limits are real: a process of the same user can delete the key, though not read it, and the build node runs as the operator's user, so its tool policy is a policy, not a sandbox. A dedicated build user with its own deploy credentials is the stronger setup, at the cost of the canary and update tooling that run as the operator.

The first hub-run release found three bugs

v1.0.694 was the first release the hub ran. The owner signed publish, relay and site with Touch ID. Publish took 373 s, relay 235 s with its verify passed, site 41 s and verify 40 s, and v1.0.694 went live. Then the run stopped at the canary step, as the procedure says it should. On the way it found three bugs, each now fixed with a guard:

• The build node was joined without --allow-all-tools. A real Claude Code agent was refused every shell call before the node's policy was even consulted. Our end-to-end test missed it because the scripted stand-in ignores permission modes. The guide's join line is fixed and checked by a test.
• Mac-side scripts needed bash 4. The node's PATH had only macOS's bash 3.2, so the gates step failed with mapfile: command not found. Every Mac-side script now runs under bash 3.2, and a check enforces it.
• The node's environment leaked into nested nodes. The canary step starts canary nodes from inside the build node's agent. Those nodes read a variable the build node had set for its own agents, the approval key, as their own configuration, refused to start, and the step reported the previous canary record. The fix is general: a node's configuration never comes from the variables a node exports to its agents, and those variables now have their own names. An audit found the same class of leak in four more places.

A fourth finding was harmless: the summary step prints RELEASE_SUMMARY where the procedure asks for a RELEASE_STEP line, and the agent returned it verbatim and said so.

The second ran every step ok

v1.0.695 ran on the same hub session and build node. The owner signed all five production grants before the run (five Touch IDs) and deleted the grant file afterwards; rollback was not needed.

Limits

• The status page is one machine's view. It is as fresh as the last canary run on the operator's Mac.
• Access through nfltr needs the relay. When the relay is down, so is the access agent; the cloud provider's SSH is the way in then.
• The Linux node lives on the operator's Mac. It is there only while the Mac and its Docker VM run; otherwise the canary records a skip.
• Releases still start at the Mac and need the owner's finger for each production step. Grants signed from a paired phone, so the approval itself is the grant, are a follow-up.