Distributed Workers
A fleet on nfltr.xyz does not have to live on one machine. A pod in a cluster, an ephemeral CI runner, your laptop, a coworker's laptop, a macOS box and an accelerator host can all be the same fleet. This guide shows the real nfltr worker command for each of those eight cases, the labels each one advertises, and how a run lands on the right machine.
1. Every worker dials out to nfltr.xyz
There is one direction of travel. Workers connect out to the hosted relay; the relay never connects in to you. That is what makes the eight cases below work at all: a pod with no ingress, a CI runner behind NAT and a laptop on hotel WiFi are all just outbound gRPC clients.
$ nfltr worker --mcp-command "nfltr claude-mcp --cwd ~/project --task-mode \${TASK_MODE}"
Two flags carry the connection, and both already default to the hosted service:
--server ADDR— the relay's gRPC address. Defaults togrpc.nfltr.xyz:443, with--tlson by default. You do not run a relay; nfltr.xyz is the relay.--api-key KEY— the key from Settings → API keys. The worker also readsNFLTR_API_KEYfrom the environment, or--key-name NAMEto pull a named key out of the config store you populated withnfltr config add-api-key. Without a key (or a client cert via--tls-cert/--tls-key) the worker refuses to start.
--mcp-command is required: it is the local AI tool the worker launches to actually do the work, and it must expose the tool named by --mcp-tool (default run_task).
How it stays connected
Staying attached is the worker's job, not yours. The relevant knobs, all readable from nfltr worker --help:
| Flag | Default | What it governs |
|---|---|---|
--retries / --retry-wait | 12 attempts, 5s apart | Reconnection after the stream drops |
--heartbeat-interval | 15s | App-level A2A heartbeat. A stall watchdog cancels the connection when nothing has shipped for 2× this interval, so it reconnects instead of hanging |
--keepalive-time / --keepalive-timeout | 60s / 20s | gRPC keepalive pings and ACK deadline — lower them on networks that silently evict connections |
--resume-state-dir / --resume-stale-after | state dir per agent id, 6h | In-flight tasks are persisted, so a restarted worker emits task.resume rather than dropping work. Anything older than the window is discarded instead |
--detach | off | Re-exec into the background with its own session and log files; the parent prints the child PID and exits 0 |
Confirm the worker arrived from any machine with a key:
$ nfltr orch fleet workers --json
Workers are meant to run wherever your code and tools already are — that is the whole point. Authentication, task storage, routing and history stay on nfltr.xyz. There is nothing to host.
2. What a worker advertises
A worker is a bag of labels and a capacity number. Selection reads that bag. Some labels the worker fills in for itself; the rest you supply with --labels key=value,key=value.
Set for you at connect time
| Label | Where the value comes from |
|---|---|
flavor, provider | --flavor when you set it (claude-code, copilot, cursor, codex, openai, generic); otherwise detected from --mcp-tool and --mcp-command |
nfltr.machine_id | The machine's hostname |
nfltr.workspace_workdir | The --cwd / --workdir value parsed out of your --mcp-command |
nfltr.workspace_mutable | true unless you override it |
nfltr.workspace_isolation | per_task when you pass --per-task-worktree. A worker that shares one checkout should say so explicitly: nfltr.workspace_isolation=shared |
nfltr.supports_setup_phase | true when --setup-executor resolves an executor |
nfltr.harness.version, nfltr.harness.models, nfltr.harness.skills, nfltr.harness.features | Discovered from the local AI harness at startup, on a 3-second budget. Version is scalar; the rest are comma-separated |
Supplied by you with --labels
| Label | Meaning |
|---|---|
nfltr.supports_task_mode.<mode>=true | The task modes this worker will accept — implement, review, summarize, inspect, and so on. The worker derives its advertised task-mode list from exactly these labels, so a worker with none advertises none |
nfltr.command.<tool>=true | A toolchain executable this machine has: git, go, make, node, npm, python, python3, expect, screencapture |
nfltr.worker.tool=run_task | The MCP tool the worker executes through |
nfltr.artifact_exchange=true, nfltr.artifact_transport.a2a_chunk=true | This worker can stream artifact bytes over the task channel |
| Anything else | Free-form. gpu=a100, ram=64g, pool=dev, region=us-east are all just strings a phase can require |
nfltr.worker.max_tasks reports concurrency and mirrors --max-tasks (default 3). When you bring workers up with nfltr orch fleet launch PROFILE instead of by hand, the launcher writes the task-mode, tool, artifact-transport and max-tasks labels for you, and fills in nfltr.command.* by doing a PATH lookup for each of those nine executables on the machine it is launching on.
Labels are not frozen at connect time. --labels-watch-file PATH points the worker at a JSON file; it polls every 5s and pushes a worker.labels.update to the relay whenever the file's mtime advances. That is how a worker whose environment changes mid-life re-advertises without reconnecting.
3. How work is routed
Nothing is statically assigned to a machine. A dispatch states a contract, and the runtime selects any connected worker that satisfies it:
On orch start task | Means |
|---|---|
--task-mode MODE | Requires nfltr.supports_task_mode.MODE=true |
--required-command CMD | Hard contract: the worker must advertise nfltr.command.CMD=true. Repeatable, or comma-separated |
--require-label key=val | Hard contract on any label. Repeatable |
--workspace-isolation shared|per_task | Hard contract on nfltr.workspace_isolation |
--required-tool NAME | Hard contract on the advertised MCP tool |
--preferred-flavor, --preferred-cost-tier, --preferred-speed-tier | Soft hints — prefer a match, do not reject a non-match |
--worker AGENT_ID | Pin. The named worker must still satisfy every hard contract, or dispatch fails |
In a spec, the same thing is a phase's worker_label (a comma-separated key=value string), plus required_labels and required_commands. A phase that must run on one specific box just requires its nfltr.machine_id:
$ nfltr orch start task --task-mode implement --required-command go,make \
--require-label nfltr.machine_id=build-linux-01 \
--objective "Build the release binaries" --watch
The worker reports its OS, CPU architecture and CPU core count from the Go runtime. Everything else in its resource block is a flag value, not a measurement: --resource-memory-mb (default 4096), --resource-disk-mb (default 10240) and --resource-load-pct (default 25) are whatever you or your launch script say they are, and --resource-cpu-cores merely defaults to the detected core count. There is no GPU probe, no memory probe and no benchmark. A machine gets accelerator work because someone labelled it gpu=a100 and someone required that label — that is the entire mechanism.
4. Eight machines, one fleet
1 — Kubernetes worker hosts
This section is about placing workers next to cluster workloads — not about deploying the NFLTR control plane. Production control-plane installs use a standalone GCE VM with the embedded web/ tree inside rpc-server (rebuild and reinstall the bundle); Kubernetes/Helm and make deploy-ui are legacy / non-production for the control plane. A worker is a normal long-running process, so a Deployment with one container is enough. Put the CLI on the image's PATH, feed the API key from a Secret as NFLTR_API_KEY, and make the container command:
$ nfltr worker --name build-pod-1 --flavor claude-code --max-tasks 2 --per-task-worktree \
--labels nfltr.supports_task_mode.implement=true,nfltr.worker.tool=run_task,nfltr.command.git=true,nfltr.command.go=true,nfltr.command.make=true,nfltr.artifact_transport.a2a_chunk=true,pool=cluster \
--mcp-command "nfltr claude-mcp --cwd /workspace/repo --task-mode \${TASK_MODE}"
Advertises: flavor=claude-code, nfltr.machine_id=the pod name, nfltr.workspace_workdir=/workspace/repo, nfltr.workspace_isolation=per_task, nfltr.workspace_mutable=true, nfltr.supports_task_mode.implement=true, nfltr.command.git|go|make=true, nfltr.worker.max_tasks=2, pool=cluster.
Two details matter in a cluster. Set --name explicitly so a rescheduled pod keeps a stable agent identity rather than inheriting a new generated one. And because pod hostnames are the nfltr.machine_id, prefer pool=cluster over machine id when you want "any pod" — machine id is the wrong hook for a replica set. No Service, no Ingress: the pod dials out.
2 — CI runner
A CI worker exists only for the job. Start it detached as one step, run the orchestration, and let the runner die — the connection dies with it, and the relay simply loses a candidate.
$ nfltr worker --detach --name "ci-${CI_JOB_ID}" --max-tasks 1 --per-task-worktree \
--labels nfltr.supports_task_mode.implement=true,nfltr.worker.tool=run_task,nfltr.command.git=true,nfltr.command.node=true,nfltr.command.npm=true,lifetime=ephemeral,pool=ci \
--mcp-command "nfltr claude-mcp --cwd \${CI_WORKSPACE} --task-mode \${TASK_MODE}"
Advertises: nfltr.machine_id=the runner's hostname, lifetime=ephemeral, pool=ci, nfltr.workspace_isolation=per_task, nfltr.worker.max_tasks=1, plus the three command labels.
--detach prints the child PID and returns 0, so the step does not block. NFLTR_API_KEY comes from the CI secret store. Keep --max-tasks 1: a runner is sized for one job. Because the worker vanishes when the job ends, never target it by machine id from a later run — target pool=ci and let selection find whichever runner is alive right now. Give tasks aimed at ephemeral workers a bound with --hard-timeout-ms so a runner that disappears mid-task fails fast instead of sitting.
3 — Your own developer laptop
The workstation case: the checkout is already on disk, and you want the work done in it, not in a copy.
$ nfltr worker --labels nfltr.supports_task_mode.implement=true,nfltr.supports_task_mode.inspect=true,nfltr.worker.tool=run_task,nfltr.workspace_isolation=shared,nfltr.command.git=true,nfltr.command.go=true,pool=dev \
--mcp-command "nfltr claude-mcp --cwd ~/project --task-mode \${TASK_MODE}"
Advertises: nfltr.machine_id=your hostname, nfltr.workspace_workdir=~/project as written, nfltr.workspace_isolation=shared, nfltr.workspace_mutable=true, two task modes, pool=dev.
Note what changed from the cluster case: no --per-task-worktree, and nfltr.workspace_isolation=shared stated explicitly. That is the difference between "work in my repo" and "work in a scratch copy" — and it is the label a phase requires when it needs to run against a specific checkout. Because tasks land in your live tree, this is also the worker you want --progress-interval on: it samples git diff HEAD in --progress-workdir and emits the diff inline, so you can watch the patch build before the task is terminal. The key comes from your local config, so no --api-key on the command line.
4 — A coworker's laptop
Another person's machine joins the same fleet by running the same command against their own account's key. Nothing else is shared.
$ nfltr worker --name alice-mbp --per-task-worktree --max-tasks 1 \
--labels nfltr.supports_task_mode.review=true,nfltr.worker.tool=run_task,nfltr.command.git=true,owner=alice,pool=shared \
--mcp-command "nfltr claude-mcp --cwd ~/project --task-mode \${TASK_MODE}"
Advertises: nfltr.machine_id=alice-mbp (or her hostname), owner=alice, pool=shared, nfltr.workspace_isolation=per_task, nfltr.supports_task_mode.review=true.
The honest framing on trust: a worker executes whatever objective it is handed, using its own local credentials, in its own filesystem. Someone else's laptop is someone else's blast radius. Three things follow.
- Isolate by default.
--per-task-worktreegives each task a fresh workspace instead of her live checkout — a fresh git worktree when the workdir is a checkout, a fresh empty directory otherwise. - Advertise narrowly. A worker that only advertises
reviewwill never be selected for animplementphase, because task mode is a hard contract. Capability advertisement is the access policy. - Gate the risky work. Dispatch with
--require-approvaland--approval-reasonso a human confirms in the HITL queue before a task on a shared machine starts, and use--read-onlyfor phases that should not write at all.--allow-dangerousis an explicit opt-in on the worker side; leave it off on machines you do not own.
Label by owner= as well as by pool. It makes nfltr orch fleet workers readable when six people's laptops are connected, and it gives you a hard contract to require or avoid.
5 — A macOS machine
Some work only runs on macOS — anything touching screencapture, or a build that needs Apple toolchains. Say so with commands, not with a comment.
$ nfltr worker --name mac-studio-01 \
--labels nfltr.supports_task_mode.implement=true,nfltr.worker.tool=run_task,nfltr.command.git=true,nfltr.command.screencapture=true,nfltr.command.expect=true,os=darwin,pool=mac \
--mcp-command "nfltr claude-mcp --cwd ~/project --task-mode \${TASK_MODE}"
Advertises: nfltr.machine_id=mac-studio-01, nfltr.command.screencapture=true, nfltr.command.expect=true, os=darwin, pool=mac.
Route to it by the capability rather than the platform:
$ nfltr orch start task --task-mode implement --required-command screencapture,expect \
--objective "Capture the console screenshots for the docs run" --watch
The worker does report darwin and arm64 in its manifest resources from the Go runtime, so that information is genuinely there — but --required-command is the better contract, because "has screencapture" is what the task actually needs. The extra os=darwin label is belt and braces for humans reading the fleet list.
6 — GPU-heavy host
The accelerator case, stated plainly: nfltr never looks for a GPU. There is no probe. The host is a GPU host because you said so, and the work lands there because the dispatch required the label you invented.
$ nfltr worker --name gpu-a100-01 --max-tasks 1 \
--labels nfltr.supports_task_mode.implement=true,nfltr.worker.tool=run_task,nfltr.command.python3=true,nfltr.command.git=true,gpu=a100,gpu_count=4,accelerator=cuda,pool=gpu \
--mcp-command "nfltr claude-mcp --cwd /srv/training --task-mode \${TASK_MODE}"
Advertises: gpu=a100, gpu_count=4, accelerator=cuda, pool=gpu, nfltr.command.python3=true, nfltr.worker.max_tasks=1.
$ nfltr orch start task --task-mode implement --require-label gpu=a100 --required-command python3 \
--objective "Run the fine-tuning sweep and report final loss" --watch
gpu, gpu_count and accelerator are strings you chose; nothing validates them. If the card is pulled out of the machine, the label keeps saying a100 until someone changes it — either by restarting the worker with new --labels, or by rewriting the --labels-watch-file JSON, which pushes the update within about five seconds. Keeping labels honest is an operational duty, not something the platform does for you. --max-tasks 1 is usually right here: one task should own the accelerator.
7 — CPU-heavy host
Parallel builds and wide test matrices. Same mechanism, different labels — with one wrinkle that is genuinely automatic.
$ nfltr worker --name build-linux-01 --max-tasks 6 --per-task-worktree \
--resource-cpu-cores 64 \
--labels nfltr.supports_task_mode.implement=true,nfltr.worker.tool=run_task,nfltr.command.git=true,nfltr.command.go=true,nfltr.command.make=true,cpu=high,cores=64,pool=build \
--mcp-command "nfltr claude-mcp --cwd /srv/build --task-mode \${TASK_MODE}"
Advertises: cpu=high, cores=64, pool=build, nfltr.command.go|make|git=true, nfltr.workspace_isolation=per_task, nfltr.worker.max_tasks=6.
--resource-cpu-cores is the one hardware number the worker will fill in by itself: left unset it defaults to the CPU count the Go runtime reports, and the value is published in the manifest's resource block alongside the OS and architecture. It is still only a hint for a planner reading the roster — selection contracts match on labels. So set cores=64 as a label too if you want to require it, and use --max-tasks to say how many of those slices the box should hold at once.
8 — High-memory host
Large datasets and memory-bound analysis. Note carefully what the memory number is:
$ nfltr worker --name mem-512g-01 --max-tasks 2 \
--resource-memory-mb 524288 \
--labels nfltr.supports_task_mode.implement=true,nfltr.supports_task_mode.summarize=true,nfltr.worker.tool=run_task,nfltr.command.python3=true,ram=512g,memory=high,pool=analysis \
--mcp-command "nfltr claude-mcp --cwd /srv/data --task-mode \${TASK_MODE}"
Advertises: ram=512g, memory=high, pool=analysis, nfltr.command.python3=true, two task modes, nfltr.worker.max_tasks=2.
--resource-memory-mb has a fixed default of 4096 and is never measured — a 512 GB machine that omits the flag will publish 4096 MB and look like the smallest box in the fleet. The same is true of --resource-disk-mb (default 10240) and --resource-load-pct (default 25); the load figure in particular is a constant, not a live reading. Set them deliberately, and treat the ram=512g label as the thing that actually routes work:
$ nfltr orch start task --task-mode implement --require-label memory=high --required-command python3 \
--objective "Load the full event export and report the top ten regressions" --watch
5. Routing walkthrough: one goal, six machines
The point of all of this is that a phase graph does not care where its workers are. Take a plain-English goal:
"Add the new export format, run the full test matrix, capture the console screenshots for the docs, and summarize what changed."
Connected to the fleet at that moment: the pod (pool=cluster), a CI runner (pool=ci), your laptop (pool=dev, nfltr.workspace_isolation=shared), Alice's laptop (owner=alice, review only), the Mac Studio (nfltr.command.screencapture=true) and the 64-core builder (cpu=high). Draft the graph and run it:
$ nfltr orch spec decompose --goal "Add the new export format, run the full test matrix, capture the console screenshots for the docs, and summarize what changed" | nfltr orch start spec - --watch
Each phase carries its own contract, so each lands somewhere different:
| Phase | Contract | Where it lands, and why |
|---|---|---|
| Implement the export format | task_mode=implement, required_commands: [git, go], nfltr.workspace_isolation=shared | Your laptop. It is the only connected worker advertising a shared checkout, so the isolation contract picks it out — the pod and builder advertise per_task and are rejected |
| Run the test matrix | task_mode=implement, required_commands: [go, make], worker_label: cpu=high | The 64-core builder. --max-tasks 6 lets six slices of the matrix run there at once, each in its own worktree |
| Capture the screenshots | task_mode=implement, required_commands: [screencapture, expect] | The Mac Studio. It is the only worker advertising those two commands; no phase had to name it |
| Review the diff | task_mode=review | Alice's laptop or any other review-capable worker. Task mode alone is enough to exclude every implement-only worker |
| Summarize | task_mode=summarize | Whichever summarize-capable worker has a free slot — here, the high-memory host |
Watch the whole thing land:
$ nfltr orch fleet status
$ nfltr orch task status --task TASK_ID --events 20
Three things are worth noticing. Nothing in the goal mentioned a machine — the phases described what they needed, and capability did the rest. The physical location of each worker was irrelevant: a pod in a cluster, a laptop on WiFi and an ephemeral CI runner were interchangeable candidates wherever their labels matched. And when you do need one specific box, that is a one-flag escape hatch rather than a different system:
$ nfltr orch start task --task-mode implement --worker mac-studio-01 \
--objective "Re-capture the help overlay screenshot" --watch
The corollary is the operational rule for a mixed fleet: a machine is only as targetable as its labels are honest. If a phase never lands where you expected, read the roster before you read the graph — nfltr orch fleet workers --json shows exactly what each machine claimed, and the mismatch is almost always there.
Next steps
- Multi-agent orchestration — phases, parallel slices and specs in depth
- nfltr orch: The Complete Tutorial — the full operator console and command reference
- Team collaboration — sharing a fleet across people
- Troubleshooting — when a worker will not connect or a phase will not route