gRPC over the connect sidecar: drop client keepalive, keep deadlines

October 17, 2026

Once a gRPC channel runs through the connect sidecar, the network between machines is no longer the application's connection. It is a chain of link connections that watch themselves. Most keepalive settings that were there for the network can go. Deadlines stay, and matter more. We measured every claim in this post with grpc-go against a private relay, real sidecars and injected faults; the harness is in the repository and the numbers are from its results.json.

Who watches which hop

A call through a connect: mapping crosses four hops. The application talks to a local socket. The sidecar and the agent each hold a link connection to the relay. The agent dials the server.

The sidecar never parses HTTP/2. gRPC's own PING frames, deadlines and RST_STREAMs are bytes it carries, so everything gRPC does end to end still works, including the server's enforcement of how often a client may ping.

Deadlines and cancellation pass through

On one channel we made 20 unary calls, opened a stream and made 5 more calls. The sidecar's /status counted one session for all of it: one gRPC channel is one HTTP/2 connection, and that is one session to one service.

• A call with a 3,000 ms deadline reached a handler that saw 2,999 ms left. grpc-timeout is a header, and nothing on the way touches it.
• A 1 s deadline on a handler that never returns: the client got DEADLINE_EXCEEDED at 1,002 ms and the handler's context ended at 1,002 ms.
• A cancel after 500 ms: CANCELED at the client, the handler's context ended at 500 ms.

The handler's context ended with "canceled" in both cases: the client's RST_STREAM arrived before the server's own timer from grpc-timeout fired. That is what grpc-go does without a sidecar too.

A dead path ends the call in 15 seconds, with no keepalive

We opened an idle server stream with no client keepalive configured, then dropped every packet between two machines with iptables: no FIN, no RST, the worst kind of failure.

Each link connection sends a small ping once nothing has arrived for 5 s and declares its peer dead 15 s after its last frame. When a hop dies, the sessions on it are reset both ways, so the sidecar resets the application's local connection and gRPC returns UNAVAILABLE. The server's handler sees its context end. Neither side hangs until a TCP retransmission timer gives up minutes later.

Relay restarts and drains

A relay that leaves (a deploy, a restart) drains: new sessions go elsewhere, and open ones keep their bytes until the drain deadline. We restarted the only relay under an open stream, with a 10 s drain window.

• The stream ended UNAVAILABLE 10.1 s after SIGTERM, and the sidecar's /status gave the reason: drain_deadline.
• A retried idempotent call succeeded 11.6 s after SIGTERM, on a new session, once the relay was back (4 attempts, 500 ms apart). In an earlier run it took 15.7 s.
• Right after a relay-to-agent path came back, we also saw the first opens of a mapping fail UNAVAILABLE for about two seconds before one succeeded. The same retry covers it.

With several relay replicas, new sessions open on another replica at once and only the sessions still open at the deadline are cut.

Client keepalive is optional, and can hurt

Keepalive pings are gRPC's own frames, so the server still counts them, sidecar or not. grpc-go's default server allows one ping every 5 minutes on a connection with streams and none without, and answers a client that pings faster with a GOAWAY (ENHANCE_YOUR_CALM, too_many_pings), which ends every call on the connection.

• Client keepalive 10 s with pings allowed without streams, on an idle stream: GOAWAY too_many_pings after 31 s. The client's error said so: received prior goaway: code: ENHANCE_YOUR_CALM, debug data: "too_many_pings".
• Client keepalive 60 s, timeout 20 s, on a healthy idle stream: GOAWAY too_many_pings after 181 s.

The sidecar hops need no help: link already found the dead paths above in 15 s. A keepalive that was set so a NAT or a load balancer would not drop an idle connection has nothing to protect any more, because the application's connection now ends at a local socket.

Keepalive does not find a stuck server. A deadline does

A server whose handler is stuck, waiting on a lock or a dead dependency, still has a healthy HTTP/2 transport, and that transport answers PINGs. We gave the client keepalive 10 s with a 5 s timeout, against a server that permits it, and called a handler that never returns:

• After 45 s the call was still open and the channel READY; 115 bytes of PING ACKs had come back in that time.
• The same call with a 2 s deadline ended DEADLINE_EXCEEDED at 2.0 s, and the handler's context ended at the same moment.

Keepalive answers "is the connection alive", and link answers that already. Only a deadline answers "is my call going to finish".

The exception: forward: to another host

An agent can publish a server that runs on another host, or behind a load balancer: grpc: { tcp: 50051, forward: "ledger.internal:50051" }. That last hop is a plain TCP connection from the agent, and no link watches it. We read what the agent's dialer sets: net.Dialer{Timeout: 10s} with no KeepAlive, so Go's defaults apply: TCP keepalive on, first probe after 15 s idle, then every 15 s, dead after 9 unanswered probes (ss showed the socket's keepalive timer). We made that host vanish under an idle stream in two ways: iptables DROP on the agent (a host gone silently), and docker pause (a frozen host whose kernel still answers TCP):

So for an idle long-lived stream to a forwarded server, the agent's TCP keepalive finds a vanished host in about 2.5 minutes, and only a client keepalive finds a frozen one. A 60 s keepalive needs the server to allow it, though: against grpc-go's default enforcement it drew too_many_pings after 181 s. Keep the client's keepalive time at or above what the server enforces (5 minutes for a default grpc-go server), or lower the server's minimum where you own the server. Calls with deadlines need none of this; it matters only for streams that stay idle for minutes.

Recommended settings

In Go, that is a channel without grpc.WithKeepaliveParams and a deadline on every call:

conn, _ := grpc.NewClient("unix:///run/user/1000/nfltr/ledger.sock",
    grpc.WithTransportCredentials(insecure.NewCredentials()))
// no keepalive: link watches the hops through the relay

ctx, cancel := context.WithTimeout(ctx, 2*time.Second) // every call
defer cancel()
resp, err := client.Get(ctx, req)
if status.Code(err) == codes.Unavailable {
    // retry if Get is idempotent: a new session opens
}

Migrating checklist

1. Point the channel at the mapping's local endpoint. The stubs, interceptors and TLS options stay.
2. Remove client keepalive that was there for the network: NAT or load balancer idle timeouts, a VPN. Keep it only for streams idle for minutes to a forward: server on another host, at or above the server's enforced minimum.
3. Put a deadline on every call, streams included, sized for the work.
4. Keep the retry on UNAVAILABLE for idempotent calls, with backoff.
5. Leave the server's keepalive enforcement and MaxConnectionAge as they are.
6. Keep connect timeouts short; the sidecar fails a call at once when the remote is offline.
7. Watch the sidecar's /status and /metrics: last_error.reason says drain_deadline when a relay drain cut a session.

The same guidance is kept current in nfltr's guides, next to the code: docs/guides/connect-sidecar.md ("Keepalives and deadlines": the hops, the settings, this checklist and the forward: exception) and docs/guides/grpc-over-connect.md (the gRPC settings). Every flag is in the nfltr up and nfltr dial reference.

Across several link hops

The path above is a chain: the sidecar's link to the relay, then the relay's link to the agent, joined at the relay. Longer chains (several relays, your own hops built on link) behave the same way, hop by hop.

Liveness is per hop.

• Each link connection pings only while nothing arrives, and declares its peer dead within its own interval plus timeout. The cost is per connection, not per session: an idle link carried 156 bytes in 20 s with one session on it and 208 bytes with 500.
• Each hop has its own setting. The sidecar's connection uses link's default (dead within 15 s), an agent sets its own (keepalive: in its nfltr up file), and a listener with no setting of its own adopts the dialer's.
• When a middle link dies, the hops on both sides of it detect it within that link's bound and send ABORT both ways, with CodeConnLost and the index of the hop it came from. The ends read an error, never a clean end of stream. In process, with the default preset on every link of a three-link chain and the middle link black-holed, the origin read it after 14.0 s and the far end after 15.0 s: the failed link's bound plus one trip along the chain.
• A drain does the same at its deadline with CodeDrainDeadline; that is the drain_deadline reason above.

What per-hop liveness cannot see: a process that is stuck while its connection still answers pings (a deadline finds it, as above), the plain TCP forward: hop, and a reader that stopped reading, which per-session flow control holds to its own session instead.

Deadlines across hops. link carries no deadline value from hop to hop. Each side's deadlines are its own, and giving up travels as an abort or a close, which every hop passes on.

• The open is bounded by the context the origin passes: Answered(ctx), Pool.Open(ctx, …) or Node.Open(ctx, …) covers the open across every hop. A hop can add a per-hop limit, ForwardConfig.AnswerTimeout, which refuses with CodeTimeout and its hop index. nfltr's relay sets none, so the origin's context (the sidecar's --open-timeout) governs.
• SetReadDeadline and SetWriteDeadline are local, as on a net.Conn. A forwarding hop applies no read deadline and no idle limit; a write deadline set on a spliced session aborts both ways with CodeTimeout.
• For gRPC, grpc-timeout carries the end-to-end budget in band; the handler two links away saw 2,999 ms of 3,000 ms.

link does not add a deadline of its own because the application's protocol already carries one where it is needed (gRPC, HTTP), a wall-clock deadline would need every hop to agree on the time or subtract its own delay, and the case that matters, a side giving up, already reaches every hop as an abort.

For Go code on link directly

Programs that use link themselves get the same behaviour from four calls:

• Pick a liveness preset per connection on the dialing side: LivenessFast (dead within 4.5 s), LivenessDefault (15 s), LivenessNAT (30 s), LivenessDatacenter (45 s), LivenessMobile (60 s), or link.ParseLiveness("nat,timeout=40s") from a flag. A dead peer surfaces as link.ErrDeadPeer, or across hops as a *link.Abort with CodeConnLost.
• Bound the open with a context: Open(ctx) on a Pool or Node, or Answered(ctx) after Mux.Open.
• Set deadlines on the session with SetReadDeadline and SetWriteDeadline; a passed read deadline is os.ErrDeadlineExceeded, exactly as on a net.Conn.
• Propagate giving up with Abort(code, msg) or Close, and put the budget in the header when the far end needs it.

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
s, err := m.Open([]byte("5s")) // the budget, if the far end needs it
if err != nil {
    return err
}
if err := s.Answered(ctx); err != nil { // the open, across every hop
    return err
}
stop := context.AfterFunc(ctx, func() { _ = s.Abort(link.CodeApp+1, "caller gave up") })
defer stop()
_ = s.SetReadDeadline(time.Now().Add(200 * time.Millisecond)) // this side only

This is adapted from link's Example_deadlines, which go test runs: the far end, two links away, reads the origin's abort as *link.Abort with the application's code and Hops 2. Its sibling Example_deadMiddleHop checks the dead-middle-link behaviour above on every test run.