link: why relay replicas stopped talking gRPC
A relay replica and an agent each carry many long-lived byte streams between two endpoints: TCP tunnels, end-to-end encrypted shares, proxied HTTP. For a while those streams rode on gRPC. This post lists the classes of problem that came from that, how a stream multiplexer, which we call link, removes each, and the numbers we measured before and after. It also says where gRPC remains the right tool, and where link is still young.
The problem
Between two relay replicas, and between an agent and its relay, there are hundreds of sessions at once. Each is a byte stream with its own pace: a visitor downloading a large file, a terminal that sends a key now and then, a tunnel that sits idle for an hour. They all share one pair of endpoints, and the failure of one connection should cost its sessions an error, not minutes of silence.
The question that started the work came from the Kubernetes soak: why does one IP-to-IP link between two pods need two different connection codes, one for agents and one for replicas? The answer was that it did not, and that the second one, built on gRPC, carried most of the problems below.
What gRPC between two pods gets wrong, and what link does
Liveness
gRPC keepalive clamps its ping interval to at least 10 s, counts only bytes received, and a write into a dead socket still succeeds. A pod that disappeared left visitors waiting for between 80 s (with the chart's settings) and 2 h (with the defaults). In link both ends ping once they have received nothing for the interval, every ping is answered, and the connection is dead within one interval plus one timeout of the peer's last frame, busy or idle. Open sessions then get an error. The default preset detects a dead peer in 15 s and the fast one in 4.5 s.
Isolation and head-of-line blocking
Many sessions on one gRPC stream share one receive loop, so a stuck reader blocks the rest. In the test we call F1, with a cold start, 0 of 8 other downloads finished in 10 s while one reader was stuck. Giving each session its own gRPC stream fixes that, at +10.1 KiB and +4 goroutines per session. In link, each session has its own credit, nothing waits on the receive loop, and sessions take turns writing; the same test finished 8 of 8.
Per-byte overhead
A tunnel byte over gRPC is a protobuf message inside an HTTP/2 frame, with flow control twice: gRPC's and HTTP/2's. Forwarding between replicas reached 0.53 to 0.61 of same-replica throughput, with 1.8 to 2.4 times the heap. Link reached 0.95 to 0.98 of same-replica throughput, and an idle session costs +2.3 KiB and about no goroutines.
Window tuning
Per-session flow control needs a window sized for the path. A 4 MiB window halved download speed at 40 ms round-trip time; 8 MiB was needed. With one flow-control layer instead of two, there is one number to tune.
Lifecycle and drain
A rollout should let sessions in flight finish while new ones go elsewhere. Link has a goaway frame for it: the draining side says so, the peer opens no more sessions on the connection, the old ones finish, and the connection closes. In the drain test, 8 of 8 old sessions finished and 8 of 8 new ones succeeded elsewhere.
Scaffolding
Around gRPC sat a connection pool, an expiring cache, a circuit breaker, token metadata on a separate port, and keepalive tuning. Link brings the pool and the liveness itself, and the change deleted about 4,000 lines.
What it does not fix
- TCP head-of-line blocking is shared by every TCP transport. Under 1 % packet loss a pool of 4 connections was 3.5 to 5 times faster than one. Only QUIC removes it.
- Failed visits during a rollout and the time to recover from a network partition were not gRPC's doing; they belong to how the relay records and drains agents, and are separate work.
Where each fits
| gRPC | Stream multiplexer (link) | |
|---|---|---|
| Unit of work | Typed request and response, or a message stream | Raw byte streams, many at once |
| Contract | Protobuf schema, generated clients | None; the caller defines the bytes |
| Tooling | Rich: interceptors, load balancers, reflection | Small: sessions, credit, liveness, drain |
| Long-lived tunnels | Possible, with the problems above | The purpose |
| Slow reader | Can hold a shared stream | Holds only its session |
nfltr uses both. gRPC carries control, the mailbox, presence and the APIs: small typed calls where a schema helps. Link carries the agent data connections and the forwarding between replicas: byte streams where isolation and fast failure detection matter.
Keepalive presets
Link's liveness is set by a preset name, not by raw numbers. Idle cost is one small frame each way per interval per connection, and a dead peer is found within interval plus timeout.
| Preset | Interval / timeout | Dead peer found in |
|---|---|---|
| fast | 1.5 s / 3 s | 4.5 s |
| default | 5 s / 10 s | 15 s |
| nat | 10 s / 20 s | 30 s |
| datacenter | 15 s / 30 s | 45 s |
| mobile | 15 s / 45 s | 60 s |
A preset beats two raw knobs because it names the network once. The dialer states it and the listener adopts the dialer's, so the two ends cannot disagree, and the interval is bounded to 1 to 15 s so a typo cannot turn pings into a flood or into silence. On the agent it is the --keepalive flag or NFLTR_KEEPALIVE; in the Helm chart, peerLink.liveness sets it for the replica link.
The trade-off is false alarms. In our loss tests the fast preset declared a dead peer wrongly in 1 of 8 runs at 0.1 CPU with loss; the default never did. A reasonable choice: default for most, fast inside a datacenter where a quick failover matters and loss is rare, nat for agents behind devices that expire idle mappings, datacenter or mobile for links with long pauses or flaky radios. Common combinations are default on the replica link with agents on default, fast, nat or mobile according to where each runs.
link and yamux
yamux is the established Go multiplexer, so we ran the same tests against both before trusting ours.
| link | yamux | |
|---|---|---|
| Throughput, clean network, 8 sessions on one connection | 3 to 15 % more over mTLS, 17 to 47 % more in plaintext | Baseline |
| Throughput under loss | Ahead at 2 ms, a tie at 20 to 50 ms (TCP-bound) | Tie at 20 to 50 ms |
| Heap per session after moving 1 MiB each way | 9 KiB / 7 KiB | 992 KiB / 1.66 MiB |
| CPU per MB | Equal over mTLS, 18 to 38 % less on the sender in plaintext | Baseline |
| One reader stopped for good | 8 of 8 other transfers intact | 8 of 8, a tie |
| False dead peer under loss and CPU limits | 0 of 4, and 0 of 12 fairness runs | 2 of 4, and 3 of 12 fairness runs lost a connection |
| Connection killed mid-transfer | Sessions end with an error | A cut stream reads as a clean EOF |
| Dead peer found, default preset | 15 s busy; up to 20 s idle in the worst case | 13 s |
| Drain with the pool full | Was worse (below), fixed | Old and new sessions all fine |
| Maturity | Days old, 10 fuzz targets | 12 years in production, widely imported: yamux leads |
The decisive difference is the cut connection. A stream cut off by a dead connection can look to a reader like a stream that ended normally. For a file or a database dump that is silent corruption, and link reports an error. yamux is ahead on detection speed as configured and on maturity. The comparison also exposed four defects in link, among them a dead peer that took up to 20 s to be found when idle against its own 15 s bound, and old sessions lost and new ones refused when a drain met a full pool. All are fixed in link v0.4.2.
How link was made
- From evidence. The Kubernetes soak and the F1 test showed the stuck reader and the dead-peer hang, and raised the question of why one IP-to-IP link needed two connection codes.
- Phase 1 extracted the existing protocol byte for byte, checked against golden recorded bytes, so nothing changed on the wire.
- Wire version 2 added credit, goaway, liveness presets and reset. The replica link was built on it and the gRPC path was deleted.
- A fault matrix of kills, stalls and loss found a 30 s write-starvation bug.
- The yamux comparison found four more.
- The v2.0.0 hard cut: client generation 2, wire version 1 dropped, older clients refused with a message to run
nfltr update.
It was built test-first by parallel agent lanes, each with measured targets, on 10 and 11 October 2026.
What is next
The 6-hour multi-replica soak has since passed, with the replica link built on the same contract other code can now program against: three replicas, 66 agents and rolling restarts, pod kills, Redis restarts and network partitions in rotation. Every kind of traffic held 100 % steady availability (p99 217–235 ms), about two thirds of it crossing replicas over link; mailbox delivery stayed exactly once across 21,585 messages; goroutines and Redis keys did not grow. Rolling restarts failed 2–3 visits each, down from 17 before a drain bug found by the first attempt was fixed. One of three partitions took 26 s to recover, above the 10 s target; that is the next thing being measured. Link is not yet a public repository. That is planned once it has had a 24-hour soak, fuzzing, six weeks in production and a security review.
Users on an older client need to update: nfltr update --with orch --apply. Clients older than v2.0.0 are refused.