The relay as several replicas on Kubernetes

October 10, 2026

The relay is the meeting point of nfltr: agents connect out to it, and visitors, tunnels and hubs reach those agents through it. On nfltr.xyz it is one process on one VM, and it stays that way. This post is about the other shape: the relay as several replicas behind one Kubernetes Service, for people who run their own relay and for a scale-out we may need later. It covers what assumed one process, how a visit crosses replicas, a 6 hour fault soak that failed, the defect behind it, the run that passed after the fix, and what is still not good enough. Everything here was tested on kind, a Kubernetes cluster in containers on one laptop; no production traffic ran on it.

What assumed one process

The relay already had a multi-replica mode: Redis holds a record of which replica each agent is connected to, and a replica that gets a visit for an agent held elsewhere forwards it there. We deployed three Redis-backed replicas on three kind nodes and measured each path that crosses replicas. Most worked; these did not:

How a visit crosses replicas

An agent holds one connection to whichever replica the Service gave it. That replica writes the agent's record to Redis and renews it while the connection lives, TCP agents included now. A visit can land on any replica. If the agent is elsewhere, the replica forwards the visit to the agent's replica over a peer entry that requires the shared peer secret; a replica in Kubernetes mode without that secret does not start. The header that marks a request as coming from a peer is set by the forwarding replica and stripped before anything reaches the agent. The agent's replica repeats every check that needs the agent's own stream, such as a protected share or a site host mapping. Mailbox wakeups travel over Redis pub/sub, and a multi-replica relay without SQL refuses mailboxes instead of keeping one per process.

After the fixes, the cross-replica path moves 1222 MB/s for one upload session against 1294 MB/s on one replica, and 1922 against 2742 MB/s with eight sessions. It is still one gRPC stream per session; at that speed it is not what limits a relay whose links to the internet are far slower.

Shutdown is now ordered. Endpoints drop the pod first. Then each agent is let go at a random moment in the first half of the shutdown window and redials another replica, and its record is removed only if it still names this replica. The peer entries stay open until the last agent has left. Readiness no longer follows Redis: when Redis is down a replica keeps serving its own agents and refuses cross-replica routing (fail closed). Each replica also keeps a small expiring marker key in Redis; when the marker is gone, Redis lost its data, and the replica rewrites its agents' records.

The soak

Unit tests cover each fix. To see the whole thing hold up over hours we wrote a soak tool and a script that brings up a fresh kind cluster for it:

• The cluster: three relay replicas on three kind nodes, deployed with the relay's Helm chart, Redis without persistence, TCP tunnels on.
• The load: 52 agents (40 HTTP, 12 TCP) in one process on the host, and one visit per agent per second through the Service's node port: an HTTP request, or a 4 KiB echo through a TCP tunnel. About two thirds of HTTP visits cross replicas.
• The faults, in rotation every 20 minutes: a rolling restart of the deployment; a forced delete of one replica; a delete of the Redis pod, which comes back empty; and a 30 s network partition of one replica. Our first partitions did nothing: kind's default network plugin does not enforce NetworkPolicy. The tool now drops the replica's traffic with iptables on its node.
• The samples: every replica's metrics and Redis's key count every 30 s, each failed visit with its error and the replicas involved, the agents' reconnects, every replica's log for the whole run, and goroutine dumps two minutes in and at the end.
• A check on the check: every visit must be answered by the relay, which says so in a response header. In an early short run another program held the test port for a minute; the tool now refuses a port that is not the relay and fails any run another server answered.

The run is judged against fixed bounds, written down before the long runs:

The first 6 hour run failed

The first long run went overnight, on the relay version that v1.0.695 ships. Between faults it was fine: steady availability 99.9991 % for HTTP and 100 % for TCP, p99 223 ms, over about 1.1 million visits. The faults were another matter:

Two bounds were breached. The first was real: one partition took 62.4 s to recover, and every replica kill sat near 40 s, just under the bound. The second was ours: one replica's goroutines went from 185 to 336 in 80 minutes. That was not a leak. The run's tool was built before a fix to its own sampler, which looked up the connected-agents metric by the wrong name, so every replica read zero agents and the goroutine count was not netted. That replica had simply taken over agents from the others, its last sample fell inside a rolling restart, and the three replicas' total stayed flat for six hours. The goroutine dumps showed no stack growing beyond the agents' own connections. The judge now uses only steady samples and nets goroutines, file descriptors and series by the agents each replica holds; the bounds did not change.

The defect: stale agent records

We reran partitions and kills on fresh clusters with every replica's log followed and each failure recorded with its replicas. The pattern was the same in both faults.

A partitioned or killed replica does not say goodbye. Its agents notice through their gRPC keepalive, 40 s after the traffic stops, and redial; the Service sends them to another replica. That replica refused them with "already connected to another pod", because Redis still named the old replica, until the old record expired. That was the 40 s of every replica kill: not the agents' redial backoff (they redialled in under a second) and not endpoint propagation, just a stale record.

After a partition it was worse. The old replica could not see that its agents had gone, so it kept renewing their records to itself, and other replicas forwarded visits back to it. It served them into dead streams until its own transport keepalive closed them, which with the chart's settings at the time was 60 s plus 20 s. That was the 62 s breach. Records that expired during a partition came back only with each agent's next message, which left 404s on the other replicas for up to 27 s.

The fix makes a record name its session: the session's start time and a hash derived from the agent's credential. The newest session of a credential wins. The replica an agent redials admits it although the record names another replica, and the old replica's next renewal finds the newer record, writes nothing and ends its old session. Another credential, or a record written by an older relay, is still refused. Records lost with Redis are rewritten by every session on a replica at once when Redis takes writes again, not one by one as agents speak. We rejected an ownership check per request (a Redis read for every locally served visit) and a takeover message between replicas (HTTP agents publish no presence events, and the renewal already sees the newer record).

Two smaller problems turned up on the way. The chart sent agent heartbeats every 45 s, exactly the agents' own 45 s dead-relay threshold, so idle agents redialled; the chart now uses the relay's defaults (15 s heartbeats, 30 s keepalive pings closed 10 s later). And the first version of the fix read each replica's own internal relay agent, which runs on every replica under one name, as a newer session of itself; a short run caught that, and the next version exempts it. One short run also breached because the judge charged any failure within five minutes of a fault to that fault; a fault's recovery now ends at the first 60 s without a failed visit, and later failures count against steady availability.

The 6 hour run after the fixes passed

Three short runs passed with the fixes, then a second 6 hour run with the same faults every 20 minutes passed with no breach:

Netted by agents, the three replicas' goroutines averaged 388 over the first half and 383 over the second. File descriptors and memory were flat, Redis kept 69 to 70 keys, and each replica idled at 0.002 cores or less afterwards. The logs show the fix at work: 33 agents admitted on a replica while another still held their old session, and 14 re-announcements after Redis restarts and partitions.

What remains

• A replica that dies silently costs up to about 40 s for the agents it held, because that is how long their keepalive takes to notice. A shorter client keepalive needs a client release and a measurement against every relay version in use first.
• A rolling restart fails about 10 s of in-flight visits per replica: a visit already sent to an agent's stream when its replica lets the agent go fails after about 4 s, roughly 15 visits per replica at this load. The fix is make-before-break, where the relay asks the agent to connect elsewhere first and only then ends the old stream; it needs a client release.
• Visits that reach the old replica of an agent that moved still go into the dead stream until that replica's next renewal (at most 22.5 s for HTTP, 15 s for TCP) or the re-announcement after a partition heals.
• Redis without persistence loses share codes on a restart. Run Redis with persistence, or SQL.
• Not in the soak yet: end-to-end encrypted shares, site hosts and mailboxes (they need SQL and wildcard DNS in the cluster; their cross-replica paths have unit tests), and clock skew.

The Helm chart for running your own

The soaks deploy the relay's Helm chart, which gained what several replicas need. It generates the peer secret and the session signing secret and keeps both across upgrades, or takes an existing secret. With more than one replica it adds a PodDisruptionBudget of one unavailable pod and a preferred anti-affinity that spreads replicas over nodes. A preStop pause lets endpoints drop a pod before it drains, an optional autoscaler can follow connected agents per pod, and an optional client IP session affinity keeps a busy agent's extra connections on one replica. The chart's heartbeat and keepalive defaults are now the relay's, and an offline check fails when any of these go missing from the rendered chart.

A multi-replica relay needs Kubernetes mode, Redis with persistence, SQL for mailboxes, browser sessions and site hosts, an S3 artifact store (the chart refuses to render a shared filesystem one), and TLS terminated outside the pods, since in-pod ACME would order one certificate per replica. Rate limits stay per replica, so N replicas allow N times the configured rate. The chart and the soak are tested on kind only.

nfltr.xyz stays on one VM

The hosted relay at nfltr.xyz is one process on one VM, as described in nfltr.xyz now runs on nfltr, and this work does not change that. The first round of fixes shipped in v1.0.695 and changed little for one process: agents now leave in the first half of the shutdown window, and readiness no longer looks at a Redis it does not have. The session fixes are merged and ship with the next release; on one replica they change nothing observable, because admitting a newer session across replicas needs another replica. For a single relay of your own, nfltr relay serve is still the simpler path: see Run your own relay.