Three programs and a relay with no orchestration inside

October 9, 2026

Until this week the relay, the server every nfltr connection goes through, had orchestration built into it. It ran its own task orchestrator, stored task status and events, queued dashboard commands for hubs and held artifact bytes. The nfltr CLI, in turn, contained the whole relay server, the hub, the MCP servers and the tunnel client in one binary.

Both are now split. The relay holds no task state and links no orchestration code; it is built as its own project with its own client SDK and protocol module. The CLI is three programs: nfltr for tunnels and sharing, nfltr-orch for orchestration and nfltr-relay for running a relay. Every command people type is unchanged. This post covers what replaced the orchestration code in the relay, the protocol change that came with it, how the three programs fit together, what got smaller, the bugs the split exposed, and what is left.

Four releases

What orchestration code was inside the relay

The relay is supposed to be pure transport: machines connect out to it, and it accepts, routes and copies connections. It runs no model and makes no decisions. That rule held for decisions, but not for state. An inventory at the start of the work found 22 orchestration, hub and MCP packages linked into the relay binary, and 36 relay files named orchestration_* with 8,838 lines between them, the largest group in the relay package. Of the relay's 43 database migrations, 22 were orchestration schema. The relay ran an orchestrator of its own on an internal loopback agent, and kept:

• task status, events, cancellations, steering and approvals, served over a task REST API;
• a digest of every hub's tasks for the dashboard and orch list;
• a command queue that carried dashboard steer, answer and abort actions to hubs;
• the hub hooks queue and the paired-device messages of hub remote access, both under orchestration names;
• artifact bytes, under task names, and a capacity event stream that sniffed task-terminal frames to tell hubs a slot had freed.

None of this was needed in the relay. The hub already ran its own orchestrator in its own process, and the hub's exactly-once delivery of agent results to its model was already local to the hub. The relay's copy was a second place where task state lived.

Generic primitives in place of orchestration state

Each piece of orchestration state was replaced by something generic that knows nothing about tasks, or removed because the hub already had it:

With the state in the hub, nfltr orch task reads and controls go to the hub that owns the task, as an agent-to-agent call through the relay. The dashboard no longer reads task content through the relay at all: runs are shown by the hub, on the hub's own machine, and the dashboard keeps what presence tells it. Hub artifacts stay in the hub's own content-addressed store; when the CLI fetches one, the hub publishes that one artifact to the blob store and the CLI downloads it and checks its SHA-256.

That changes one thing people can see. A hub's tasks are read from the hub, not the relay: through the relay while that hub runs, and on the hub's own machine from its state files (nfltr orch hub runs, nfltr orch hub show) when it does not. Before, the relay held a status digest, but the task commands could not read hub tasks at all, so in practice they gain more than they lose.

A ratchet that went to empty

The split ran in steps, each landing green on its own. The first step added a test, TestRelayLinksNoOrch, that lists the packages the relay binary links and fails on any orchestration, hub, MCP or model package. It started with an allowance: the exact packages the relay still needed. When a step removed one, the test failed with "no longer links" until the package was struck from the allowance, and nothing could ever be added to it; a new need had to go behind a relay-owned interface instead.

After the relay orchestrator went, three packages were left on the allowance: the command queue, the hooks queue and hub remote. The mailbox took over the hooks and remote messages, the dashboard's controls started going straight to the hub, and the command queue and its database table were deleted. The allowance was empty.

The test then became TestBinaryLinkTable, one row per binary that is built from this codebase: nfltr links no orchestration package and no hosted-only backend, nfltr-orch links no relay server and no website, and the hosted relay links no orchestration package. The relay's own repository keeps the same check for nfltr-relay, and a module boundary test that fails if the relay imports anything from the nfltr codebase, or if the client SDK imports anything from the relay server.

Relay protocol v1, in one hard cut

Making the relay its own project meant its wire protocol had to become a contract other code depends on, rather than something both ends happened to agree on. The audit before the change found a lot of protocol carried outside the protobuf definitions: metadata keys used for negotiation, magic values in fields, and HTTP endpoints used as protocol.

It also found a real bug. Labels given to an nfltr http, nfltr tcp or nfltr shell agent with --labels had been silently dropped since the first version, because of a mismatch in how labels travelled between the agent and the relay. Agents on the other connection type, which hubs, nodes and workers use, were never affected, so hub placement by label always worked. One package now owns that encoding for every sender and reader.

The protocol change was a hard cut, chosen over a compatibility window: the relay and its clients release together. Every call to the relay's gRPC service now carries a Hello with the protocol version and the client's name, version and build, as binary call metadata, and the relay answers with a Welcome carrying its own version in the response header. Metadata rather than a first message on the stream costs no extra round trip, covers single calls and streams alike, and lets one check refuse a call before authentication or any handler reads it.

A client released before v1.0.686, connecting to the upgraded relay, gets one clear error on every connect:

this nfltr is too old for this relay: it does not speak relay protocol v1; run `nfltr update`

The reverse case is covered too: a new client that reaches a relay older than v1, such as one a team runs on its own network and has not upgraded, fails with a message saying the relay needs updating, and never silently talks the old protocol.

Two parts of the wire were deliberately kept:

• The data plane framing. TCP tunnels use a custom framing, NFDP, with per-session credit windows. It is now written down as a specification. Standard multiplexers such as yamux or smux were considered and rejected: they lack the per-session credit behaviour we measured, and add a dependency.
• Four agent streams. HTTP serving, agent-to-agent messages, TCP and WireGuard stay separate streams. One multiplexed session stream would be cleaner, but would bring back the per-stream writer contention the TCP throughput work removed.

One CLI, three programs

Nobody has to learn the new names. When nfltr gets a command it does not have, it looks up the program that owns it: nfltr orch, nfltr node and nfltr mcp run nfltr-orch, nfltr relay serve runs nfltr-relay serve, and any other installed nfltr-<name> runs in the way git runs its subcommands. It looks next to nfltr first and then on PATH, so a stray older copy on PATH never answers for a newer nfltr. On Unix it execs the program, so the process id, exit code, signals and terminal are the plugin's, with nothing forwarded; on Windows it runs the program as a child with the same input and output and exits with its code. Claude Code configurations that start nfltr mcp --toolset hub keep working without a change once nfltr-orch is installed.

If the program is not installed, nfltr exits with status 127 and says how to install it:

`nfltr orch` needs nfltr-orch, which is not installed next to nfltr or on PATH. Install it: nfltr update --apply --with orch (or: curl -fsSL https://nfltr.xyz/install.sh | sh -s -- --with orch)

nfltr help lists the plugin commands with the program each needs and whether it is installed, and shell completion reaches into nfltr-orch's commands when it is there. The nfltr relay serve alias was planned to last one release; since dispatch costs nothing, it stays.

Sizes

Someone who only shares a local port downloads less than half of what they did. Each binary has its own size budget, and the build fails when one grows past it.

Releases per component

Each program is published on its own, with its own signed checksum file. The install script installs nfltr, and --with orch, --with relay or both add the others, each verified against its own signatures. nfltr update checks nfltr and every nfltr-* program installed next to it; nfltr update --apply --with orch also installs nfltr-orch when it is missing. A bad signature leaves that one program in place and exits with an error; the others still update.

An nfltr installed before the split knows only the old download layout, so the release keeps publishing nfltr there too. Its next nfltr update --apply brings in the new nfltr; tunnels and shares work as before, and the orchestration commands then print the install hint above until nfltr-orch is installed once. Machines that run a node as a service need that one extra install before or right after the update.

Separate modules

The last step moved the relay out of the nfltr codebase. The relay server, its client SDK and its protocol definitions are three modules, versioned and tagged separately. The nfltr codebase now depends on tagged releases of them like any other dependency, and v1.0.688 was the first release built that way. A cross-version check builds and tests nfltr against either the pinned relay or the relay's latest code, so a relay change that breaks a client can be caught before it is pinned.

The same day, six small contract modules that only nfltr itself used were folded back into the nfltr codebase: a separate module with a single consumer is two places to change for no reuse. The agent wire definitions moved into the relay's protocol module, since the relay, the SDK and nfltr all speak them, and the relay's HTTP API definitions moved in with the relay server. Their protobuf package and file names did not change, so the bytes on the wire are identical.

Bugs the split exposed

Moving code across boundaries runs it in new combinations, and several of those combinations failed. Each was fixed where it lived, with a test that fails on the old code:

• The Docker image build. The new protocol definitions lived in a directory the Dockerfile never copied, so every image build failed. A test now fails when the Dockerfile skips a directory the binaries import.
• Process groups outliving their leader. Stopping a child process returned as soon as the group leader exited, leaving members that ignored the signal running and never escalating to a kill. It now waits for the whole group, escalates, and waits again.
• A result acknowledgement lost while a hub was frozen. A hub resumed after being stopped read the result its dead stream had buffered and sent the acknowledgement into that dead stream; the node agent then waited for it until the node stopped. Only the worker ending the call confirms an acknowledgement now, and an unconfirmed one is delivered again over a fresh stream.
• A clean close read as end of call. When the relay closed a connection cleanly, for example while draining for a deploy, pending calls ended with the same end-of-stream signal as a peer finishing normally. Callers could not tell the two apart. A closed connection now ends pending calls with its own retryable error.
• A data-plane credit deadlock. A TCP session waiting for flow-control credit could still hold its last frames in its write buffer. The peer returns credit only for bytes it received, so the session hung unless other traffic flushed the buffer. It now flushes before it waits.
• Keys from nfltr config login without an owner record. They broke the first site served through the relay; the story is in nfltr.xyz now runs on nfltr.
• Workers ignoring the process that started them. Nodes and local relays exit when the process that started them is gone; worker processes did not, and outlived killed test runs for hours. They now exit too.

What it cost and what is left

• Browser tests rebuilt. The browser tests for hub remote access and the hub's live run view drove the relay's internals directly, so they could not move with the relay. They were rebuilt over a harness that runs a whole relay in the test process and is reached only over HTTP and gRPC, signing in and minting a key as a user would. The docs helper's browser test was deleted: the helper is off in production.
• Direct tasks without a hub. The setup proof task of nfltr orch fleet setup now runs in the calling process, so it no longer shows up in nfltr orch task or the dashboard, and it ends with the command.
• The rest of protocol v1. The Hello is the first part. Still to come: typed frames on the four agent streams in place of the remaining magic values, the remaining metadata keys moved into the Hello, a second version of the data plane with refusal reasons and feature bits, and a client identity for the HTTP API, which carries no Hello today.
• The old download layout. It stays for two more minor releases, so installed clients can update. After that, a client from before the split gets a "not found" from nfltr update and needs the install script once.
• The relay's own releases. nfltr-relay is still published with nfltr's releases and carries nfltr's version number. Its own release pipeline and an install script on relay.nfltr.xyz come next, as does a command reference per product.
• Static analysis. A directory exclusion meant for the old relay directory matched every directory with "relay" in its name. With the relay in its own repository the exclusion is gone, and the findings it had hidden are recorded so new ones fail the check; they are waiting for triage.

How it was checked

• At every step: build and vet of every module, the full test suite with the race detector, and the hub end-to-end tests with a scripted stand-in for Claude, which spawns, steers, stops and reattaches agents across a real relay at no model cost.
• A 30-minute fault soak of hubs on nodes after steps 3, 4 and 5 and the module cleanup: the relay killed and restarted, nodes killed and paused, hubs killed and reattached. Every goal passed each time (28 of 28, 25 of 25, 38 of 38, 37 of 37), with the idle relay at about 0.15 % CPU and the nodes at about 0.03 %.
• The relay conformance suites (HTTP/2, gRPC, WebSocket, request smuggling, MCP) after the relay became its own module, with the same results as before the split.
• A dry run of the new release layout: every program built for every platform, signed with a throwaway key, uploaded to a stand-in bucket and installed from it with --with orch,relay.

The relay is now transport and nothing else, in what it decides and in what it stores. To run one on your own network, see Run your own relay; the install page has the --with options.