Architecture and Guarantees
One page for what nfltr guarantees, which part guarantees it, and what it does not. Every guarantee is a rule the code enforces and a test protects; the evidence section says how strong each kind of test is. Claims the sources do not support are listed under Not guaranteed and Known gaps.
At a glance
- The relay carries traffic and decides nothing. It derives identity from the credential, never from what a client says. With end-to-end encryption on, it sees who talks to whom, when and how much, not what. Threat model.
- Protocol, concurrency and cross-process state machines are checked with TLA+ before their code. Evidence.
- Your machines are the trust boundary. Whoever holds a machine's key can act as that machine. Trust boundaries.
This page describes main as of 2026-10-26; the evidence and known gaps were re-checked on 2026-11-06. The code is the source of truth. Each claim rests on nfltr's internal specification, which cites the code and the test for every rule. The code and the specification are not public; this page states their rules in plain words. Agent orchestration (nfltr-orch) is not part of nfltr and is not publicly available.
Architecture
+------------------------+ +------------------------+
| MACHINE nfltr | | MACHINE nfltr |
| cat serve, up, http | | cat pipe, cat cp, ... |
+-----------|------------+ +-----------|------------+
| outgoing connection | outgoing connection
v v
+----------------------------------------------------------+
| RELAY nfltr.xyz or your own relay |
| routes frames by agent id, checks machine certificates,|
| lists connected agents (presence), mailboxes |
+----------------------------------------------------------+
data plane between machines: link, multiplexed sessions
with per-session flow control (tunnels, TCP)
- Machine. A computer enrolled with its own certificate, running the
nfltrclient: it serves local services and connects to services of other machines. It connects out, so it needs no open port. - Relay. Transport and directory. It routes frames between authenticated agents of one account. It also stores mailboxes and answers presence queries.
- link. The data-plane library under tunnels and TCP sessions. It multiplexes sessions over one connection with per-session flow control.
Which layer guarantees what
Two layers give guarantees. A guarantee of a lower layer is not a promise of a higher one: link says nothing about the relay's routing.
link: the data-plane library
link multiplexes sessions over one connection that the caller dials and secures. It imports only the Go standard library. The wire format is version 2.
- A slow or stuck reader stalls only its own session. Each side announces a window per session and a sender may have at most that much unacknowledged.
- A dead peer surfaces as an error on every session with it, on both ends, within
Interval + Timeoutof its last frame (4.5 s to 60 s by preset, 15 s by default). io.EOFmeans the peer ended its writes on purpose. Any other cut reads an error.- Bytes of one session arrive in order and intact.
- The wire is held byte for byte by test vectors and a recorded session. A change is a new version negotiated in the hello.
The relay
- Identity comes from the credential, never from client metadata. The derived agent id overwrites any client-supplied one.
- A frame routes only between agents of one account (the source's key namespace or an agent of its owner). A refusal looks like a missing target.
- After an agent's drain is acked, no new call reaches it. Calls to a draining or departed id get
Unavailableand are not delivered. - Mailbox and presence contracts: see Contracts per primitive.
- A relay restart ends no running agent. Registrations are not kept, so every client redials under the same instance id and the relay treats it as a reconnect.
Production guarantees and other evidence
A guarantee here is a rule the code enforces and a check protects. How strong the check is differs. This table keeps the kinds apart.
| Kind of evidence | What it shows | What it does not show |
|---|---|---|
| Unit and contract tests, run on every change. Examples: the mailbox contract tests. | The rule holds in the tested cases. | Behaviour under a real network or a real fleet. |
| TLA+ model checking. The state machines of link and the relay are TLA+ specs, checked with TLC before their code and on every push. Every past bug in an area is a mutant the checker must catch, and steps recorded by the Go tests are checked against the specs. What TLA+ found in nfltr. | Every reachable state of the checked configurations keeps the invariants, and the recorded runs of the code match the spec. | Configurations larger than the ones checked, and code paths no recorded run takes. |
| Chaos lab on Kubernetes (kind). Experiments kill relays and Redis, partition and delay the network, and skew clocks. Shared steady-state probes check the result. | Real relays under network faults. | There is no general unit-level clock-skew test. |
| link fuzzing and soak. Decoders are fuzzed (4.85 hours in total, no crasher in the library). Soaks under faults ran 2.89 hours on macOS and Linux CI with leak and integrity checks. | The wire decoders and sessions survived what was run. | The project's own target for both is 24 hours. It has not been met. |
Missing from this evidence:
- No production history on your own relay. It is a beta. A smoke test covers its feature matrix and restart. It has one process and no failover.
Trust boundaries
Who trusts whom, and what each party can do if it misbehaves.
| Party | Trusted to | Not trusted to | Control |
|---|---|---|---|
| Relay | Authenticate machines, route frames inside an account, hold mailboxes and presence. | Read content, decide, or vouch for a peer's key. | Organisation sessions (nfltr cat with --allow-group) and end-to-end HTTP tunnels are encrypted between the two machines. |
Contracts per primitive
Each contract states the guarantee, the bound and the failure. Where the sources give no rule, the row says so.
Streams (link sessions)
| Question | Contract |
|---|---|
| Backpressure | Each side announces a per-session window (128 KiB to 1 GiB). A sender may have at most the peer's window unacknowledged. The receiver returns credit as bytes leave its buffer. A slow reader stalls only its own session. |
| Ordering | Bytes of one session arrive in order. There is no ordering between sessions. |
| Cancellation | Reset ends a session and the peer reads an error. CloseWrite half-closes: the peer reads io.EOF and can still write. Abort reaches the far end through every hop as a typed code. Deadlines are each side's own; link carries none from hop to hop. |
| Termination | io.EOF is a deliberate end. A reset, a closed connection, a dead peer or a drain deadline reads an error that is not io.EOF. After a go-away, no new sessions open; open ones run to their end. |
| Dead peer | Detected within Interval + Timeout. Idle connections cost a few bytes. |
| Limits | Sessions per connection are bounded by the peer's announced maximum. An open past it ends the connection (ErrFull). A data payload is at most 1 MiB per frame. |
| HTTP through the relay | A client that opts in has at most 4 response chunks and 4 upload chunks in flight per transaction. Any exit before the end sends a cancel frame. Clients that do not opt in get bodies whole. |
| A2A streams | When an agent's connection ends, every pending call ends with ErrConnectionClosed (gRPC Unavailable, retryable), never io.EOF. |
Not specified in the sources: a fairness guarantee between sessions beyond "every session keeps getting its turn", and any end-to-end encryption in link itself. link does no encryption. Confidentiality needs TLS or an encrypted payload inside the session.
Mailbox
A mailbox is a named, ordered queue of opaque envelopes owned by one account. Only the owner's principals append, pull or acknowledge. The relay does not read payloads.
| Question | Contract |
|---|---|
| Delivery | At least once. A pull never acknowledges. An envelope has a relay-assigned increasing seq. |
| Lost acknowledgement | The envelope is returned again with a higher delivery_count. The reader dedups by seq. An ack removes every envelope up to and including through_seq. |
| Consumer gone | Envelopes stay until acknowledged or expired. A reader that is away loses nothing before expiry. A full mailbox refuses the append with MAILBOX_FULL. |
| Expiry | A time-to-live up to 7 days. Unset or 0 means 7 days. After expires_at the relay drops the envelope, and a pull returns only unexpired ones. |
| Duplicates from the sender | The same idempotency_key from the same sender within 24 hours returns the first seq with duplicate = true and stores nothing. At most 1000 keys are remembered per mailbox. |
| Limits | Payload 1 MiB. 500 envelopes and 8 MiB per mailbox. 64 MiB and 256 mailboxes per account. A pull returns at most 100 and waits at most 25 s. |
| Webhooks | A POST is at most 64 KiB and 600 per minute. Refusals are counted (dropped_rate_limited, dropped_mailbox_full, rejected_too_large) and the next envelope carries dropped_before. |
| Your own relay (beta) | Mailboxes and webhooks are kept in the relay's data directory. A restart keeps them, with their unacknowledged envelopes. |
Presence
Presence lists the connected agents an account may reach, with their labels, and a change feed.
| Question | Contract |
|---|---|
| What it is | A snapshot of connected agents plus a cursor taken before the snapshot. Watching from that cursor misses nothing. |
| Staleness | An agent counts as connected until the relay ends its stream. Heartbeats run every 15 s. With heartbeat checks on, the relay ends a session that sent no frame for 45 s. Its reaper evicts agents idle for more than 180 s (it scans every 60 s). A frozen process can stay listed until then. |
| A listed agent may be dead | A call to a frozen but listed agent is answered NotFound before acceptance, so nothing ran. |
| Feed | Watch is a long poll that waits at most 25 s and returns at most 256 events, oldest first. The revision orders events within one relay process. |
| Cursor reset | If the cursor is older than the relay's history, or from another relay process, the response sets reset. Changes may be missing. The client must list again. |
| Several relay replicas | With Redis down, a replica's feed shows only its own agents. |
| Labels | Labels are advertised data. The relay filters on exact values and never interprets keys. Labels set by the operator at connect cannot be changed by UpdateLabels. |
Typed errors
Clients branch on a machine-readable reason, never on message text.
| Class | Meaning | Examples |
|---|---|---|
| Transient | The sources mark the error retryable. The same call may succeed later. | Relay errors AGENT_OFFLINE, RELAY_STARTING and DIRECTORY_UNAVAILABLE (marked retryable), RELAY_DRAINING (carries a retry hint). link: ErrGoAway (open on another connection; Pool and Node do), ErrConnClosed (open a new session). |
| Permanent | The call is refused or wrong, and the sources mark no retry. Change the request or the setup. | UNAUTHENTICATED, LIMIT_EXCEEDED, INVALID_ARGUMENT. A replaced session (FailedPrecondition) is final. |
Rules that protect the classes:
- A client redials on every error except a closed fatal list (identity refused, version refused, invalid configuration, session replaced, own context cancelled). Error text is never consulted.
The full list is generated from code: the ErrorReason enum in the relay protocol.
Threat model
Scope: the hosted relay at nfltr.xyz, your own relay (beta), the nfltr command and the dashboard. The full model, with a table per boundary, is internal; this section summarizes it.
Who can read your traffic, and what the relay sees
What the relay always sees: the account, agent ids, labels, which machine connects to which, when, frame sizes and timing. For tunnels, it sees the traffic of ordinary HTTP tunnels and shares (TLS ends at the relay) and raw TCP bytes. It does not see end-to-end TLS passthrough content.
Organisation sessions (nfltr cat with --allow-group) are TLS between the two machines with their machine certificates: the relay sees the two machines, the service, sizes and timing, not the bytes. End-to-end encryption has the details.
Key management and revocation
| Key | Where it lives | Revocation or rotation |
|---|---|---|
| Machine key and certificate | The key is made on the machine (machine.key, mode 0600, beside the config) and never leaves it; the relay signs a certificate for it that lasts 24 hours and renews itself. | nfltr machines revoke ends the streams the machine opened with Unauthenticated and refuses its renewals; a machine off for longer than its certificate enrols again. |
| Dashboard session | A signed cookie naming a server-side session, 24 hours by default. | Logout or removing the account from the allow-list ends it. A WebSocket opened before logout stays open until it closes (finding F-27). |
| Webhook secret | A bearer secret is hashed at the relay. A GitHub secret is derived from the relay's seal key. | DELETE /api/v1/webhooks/<id> removes the hook. Rotating the seal key means re-creating every GitHub hook. |
| Your own relay's machine CA and TLS certificate | Its data directory. Clients pin the TLS certificate or trust a CA. | Stop the relay, delete its machine-ca/, machines and admin-enrol-code (a new machine CA) or its tls/ directory (a new TLS certificate), start it and repeat the client step on each machine, with a new code after a new CA. Certificate files you provide can be replaced in place: the relay serves the new pair within about 10 seconds, without a restart. A machine enrolled without --account is its own account. |
| Release signing key | Held outside the repository; the public keys are embedded in install.sh and nfltr update. | The installer verifies the signature before the checksum. |
Device compromise
| What is compromised | What an attacker gets | What limits it |
|---|---|---|
| A machine, and its machine key | The key acts as that machine, inside its account and groups. It can pick its own agent id under the machine. There is no per-service credential. | Revoke the machine (nfltr machines revoke). This is audit finding F-20, residual. |
| The relay | It can deny service, drop or delay frames, observe metadata and route frames inside an account. It could serve a tampered install.sh. | Release signature checks in install.sh and nfltr update. End-to-end TLS inside sessions. |
What persists locally
| Where | What | How long |
|---|---|---|
| Relay (hosted), as inventoried on 2026-10-03 | Accounts, key hashes, machine records, mailbox envelopes until acked or expired, weekly activity, page-visit counts, contact messages, service logs, daily database backups. | Contact messages 180 days. Mailbox envelopes at most 7 days. Retention of the others is not confirmed in the sources. |
| Your own relay | A data directory with its machine CA, machines file, TLS certificate and file-backed stores. | Yours. A data directory readable by group or others is refused. |
Not guaranteed
- Availability of one relay. nfltr.xyz runs as a single relay host. A deploy window is down for 30 to 90 seconds; agents reconnect. Your own relay is one process with no failover.
- Delivery by the relay when it is hostile. It can drop or delay frames. It can always deny service.
- Ordering between sessions or mailboxes. Order is per session and per mailbox
seq. - Clock-independent timing in all cases. Leases and bounds use durations. There is no general unit-level clock-skew test.
Known gaps
These are open findings and claims the sources do not support. They are listed so you do not assume them.
- No per-service credential (audit F-20, residual). A machine key can pick its own agent id under its machine.
- Open findings of the security audit: F-33
trust_loopbackbehind a same-host reverse proxy that adds no forwarding header, F-27 relay-served browse WebSockets not rechecked after logout, F-36 minor items. The audit is the source of truth; this list is as of the date above. - Hosted retention and backups. On nfltr.xyz the relay's databases are copied to Cloud Storage daily and the copies deleted after 30 days; there are no disk snapshots. Server logs are kept 30 days. Weekly activity rows are to be kept 13 months, but nothing deletes them yet.
- link's own targets. 24 hours of cumulative fuzzing and soak are targets, not results. The API is pre-1.0.
- Your own relay (beta). There is no dashboard, Postgres or Redis, or automatic certificate; share links need a domain of yours and a wildcard certificate on the machine that shares; it has one process.
- Authorization is per account and group. The relay authenticates the machine, not the person: a machine reaches the agents of its own group and of the groups its group is allowed to reach (
nfltr machines move,nfltr groups allow).
Sources
Each claim above is a rule of nfltr's internal specification. The specification cites the code that enforces each rule and the test that protects it, and a check fails the build when a cited file or symbol no longer exists. The code, the specification, the security audit and the threat model live in private repositories, so this page states their rules instead of linking them. What is published:
- The CLI reference, every command and flag named here.
- End-to-end encryption: what the relay can and cannot read.
- Protocol packages named on this page:
nfltr.mailbox.v1,nfltr.presence.v1and theErrorReasonenum of the relay protocol; link's wire format and itspeerlinkcontract.
Next
- Run Your Own Relay (Beta): step by step.
- Organisations: machine identity and groups.