Designing a stream-multiplexer API

October 15, 2026

A stream multiplexer carries many independent byte streams over one connection. The wire part is well understood: frames, stream ids, per-stream windows, a ping. The Go API on top of it is where libraries differ, and where a mistake lasts as long as the module's major version. Before freezing link (Apache-2.0, standard library only) at v1, we read the exported APIs of hashicorp/yamux v0.1.2, libp2p/go-yamux v5.1.0, xtaci/smux v1.5.57, quic-go v0.63.0, libp2p/go-mplex v0.7.0, go-libp2p v0.50.0, golang.org/x/net v0.61.0's http2, and the standard library's net, and compared them decision by decision. This post goes through those decisions, what link chose in v0.9.0, and what it gave up for it.

Why nfltr needed a multiplexer at all, and how link compares with yamux on throughput, memory and failure detection, is in an earlier post: link: why relay replicas stopped talking gRPC. This one is about the API only. v0.9.0 changed no byte on the wire: its test vectors and golden recording are those of v0.8.1. nfltr.xyz runs link v0.9.3, the API described here plus three patch releases, since nfltr v2.1.4.

What Open returns

Every library answers the same question first: what does a caller get back when it opens a stream, and what can it do with it?

The concrete libraries offer two forms: a net.Conn for drop-in use, and the concrete type with every method. go-libp2p is the only one that abstracts over transports, as link's peerlink.Link does (it has implementations over TCP or TLS, QUIC and gRPC), and it returns a rich interface rather than a bare net.Conn. link now does the same.

Why half-close matters

Request and reply on one stream needs a way to say "the request is complete" without closing the stream. Without it, the protocol inside the stream needs a length prefix or a delimiter, or the opener closes the whole stream and cannot read the answer. With half-close the opener sends its request, closes its write side, and keeps reading.

link's quick start does exactly this. Up to v0.8.1, peerlink.Link.Open returned a net.Conn, so the quick start had to write s.(peerlink.Session).CloseWrite(). A type assertion is the least discoverable API in Go, and an unchecked one panics on a third-party Link that breaks the contract. The cost was real in our own code: the relay had three wrappers around a session (one counting bytes, one tying it to a context, one copying and then closing the write side), and each re-forwarded CloseWrite through an interface{ CloseWrite() error } assertion. A wrapper that forgets one loses half-close silently. A wrapper that embeds peerlink.Session gets the method from the compiler. The same rule holds one layer down: since v0.9.1, link.Client and link.Server document that the net.Conn they are given should have CloseWrite, as a wrapper of a *net.TCPConn or *tls.Conn passes it on. Without it a graceful close closes the whole connection, and a TCP connection closed while bytes still arrive resets, losing what was in flight to the peer.

v0.9.0 returns peerlink.Session from Open and passes it to Serve's callback and to Listener.Handle. It is net.Conn plus CloseWrite and Reset, so it still goes to http.Transport, grpc.WithContextDialer and io.Copy unchanged:

// v0.8.1
node.Serve(func(peer string, h []byte, s net.Conn) { s.(peerlink.Session).CloseWrite() })
// v0.9.0
node.Serve(func(peer string, h []byte, s peerlink.Session) { s.CloseWrite() })

Two costs come with it. Go function types are not covariant, so an adapter that wants a func(ctx, addr) (net.Conn, error) needs a one-line closure. And an interface that callers call cannot gain methods after v1 without breaking every implementation, so peerlink.Session will never grow ResetWithError, ID or anything else; new capabilities arrive as optional interfaces on the concrete types. A concrete struct was not an option: Link has three implementations.

The lower-level Mux.Open changed for a smaller reason. It took a callback, func(sid string) ([]byte, error), so the header could carry the new session's id. All eight call sites (the pool, the node, one in the relay, five tests and examples) ignored the id and returned a fixed header. It is now Mux.Open(header []byte) (*Session, error). If a caller needs the id later, (*Session).ID() can be added without breaking anyone; yamux, smux, quic-go and libp2p all have one.

Configuration

Four designs are in use:

// hashicorp/yamux, xtaci/smux: a pointer to the defaults, checked by the constructor
cfg := yamux.DefaultConfig()
cfg.KeepAliveInterval = 10 * time.Second
sess, err := yamux.Client(conn, cfg)

// quic-go: nil, or zero fields, mean the default
c, err := quic.DialAddr(ctx, addr, tlsConf, &quic.Config{MaxIdleTimeout: 30 * time.Second})

// functional options, as in grpc-go
cc, err := grpc.NewClient(target, grpc.WithKeepaliveParams(keepalive.ClientParameters{Time: 10 * time.Second}))

// link: a value whose zero is the default, checked by the constructor
c, err := link.Client(ctx, conn, link.Config{Liveness: link.LivenessFast})

link kept the last one, and the review found no reason to change it. With yamux and smux a &Config{} is invalid (smux's has version 0), so every caller starts from DefaultConfig(), and a pointer can be changed after the session started. Functional options add one exported function per setting, and the settings cannot be printed, compared or read as one struct in godoc. A value type whose zero value is the default has neither problem. Config.Validate, PoolConfig.Validate and the constructors return an error that names the field to change; yamux's equivalent says "keep-alive interval must be positive".

Two fields treat 0 as "the default" (MaxSessions is 1024, a pool's Max is 1). That loses nothing, because 0 is never a valid value for either.

A Serve callback or an Accept loop

net.Listener, yamux, smux, quic-go's AcceptStream(ctx) and go-mplex hand the caller an Accept loop. http.Server, http2.Server.ServeConn and go-libp2p's SetStreamHandler(pid, func(network.Stream)) take a handler instead. link takes a handler: Serve(func(peer string, header []byte, s peerlink.Session)).

For the callback: the library runs one goroutine per session, so a slow handler cannot hold the receive loop or the next accept. The peer's name, the header and the session arrive together, with no tuple type, and the library decides when shutdown has finished. Against it: a caller cannot select over accepts or put its own admission queue in front of them; the only accept-side limit is Config.MaxSessions. Code that needs an Accept (http.Server.Serve, grpc.Server.Serve) uses peerlink.Listener, which adapts the callback to a net.Listener, so one primitive covers both styles.

One sharp edge remains at the low level. On a bare link.Mux, Serve is also the receive loop, so a side that only dials must still call Serve(nil), or credit never returns and its writes stall. yamux and smux start their receive goroutine inside Client and Server. link leaves goroutine ownership to the caller and, since v0.9.0, says so in the first line of NewMux's documentation. Pool and peerlink.Node run it for you.

Errors

link's errors are sentinels tested with errors.Is, plus one typed error, *VersionError, for a peer that speaks a later wire version. Each wraps its cause, and each call documents which ones it returns. The rule we care most about: io.EOF means the peer ended its writes on purpose, never that the connection was cut. A cut connection reads as ErrConnClosed, and a write after it returns io.ErrClosedPipe wrapping the reason, as smux does. A reader copying a file can tell a complete file from a truncated one.

v0.9.0 made two changes. ErrDeadPeer is now a net.Error whose Timeout() is true, following quic-go's IdleTimeoutError, because net/http, gRPC and retry libraries classify errors by Timeout(); errors.Is(err, link.ErrDeadPeer) still matches. And ErrV1, a name kept for a wire version removed in v0.5.0, is gone: a version-1 peer reads as ErrVersion.

What link did not add: error codes on reset. go-libp2p's ResetWithError, quic-go's CancelWrite(code) and HTTP/2's RST_STREAM codes carry an application code; link's reset carries none, and adding one is a wire change. An application can write its own error into the stream before closing it. If codes come later, they arrive as an optional interface on the concrete types, because peerlink.Session cannot grow.

Graceful drain

How link's drain works on the wire (a GOAWAY, then a ping that fences opens which crossed it) is in the earlier post. The API problem was shape: up to v0.8.1 there were three, a duration plus a reason on Conn, a time.Time on Mux.GoAway, and a context on Pool and Node. v0.9.0 has one, Drain(ctx) error, the shape of http.Server.Shutdown: it blocks until the connection closed, and when the context ends it closes at once and returns the context's error. The context's deadline goes into the GOAWAY, so the peer knows how long it has. A caller that wants the old fire-and-forget behaviour writes go m.Drain(ctx). Conn.GoAway stays at the frame level, for callers who run their own loop.

peerlink.Link has no Close: a Drain whose context is already done closes at once. A Close() error on the concrete Node types, as on http.Server, can be added at any time without breaking anyone; adding it to the interface later could not.

The patch releases after v0.9.0 were all about drain, with no API or wire change. The conformance suite's drain check, peerlinktest's DrainAccepter, and the relay's own tests found them under load: Pool.Close cut the last bytes of sessions that had already ended, and now lets an idle connection finish sending before it closes (v0.9.1); the QUIC implementation's Open toward a peer that is draining every connection now fails at once with ErrGoAway instead of waiting; and peerlink.Node.Open (v0.9.2) and Pool.Open (v0.9.3) pass over a connection whose writes already failed instead of returning that error.

Write deadlines and credit

Each link session has its own window: the reader grants the writer credit for as many bytes as it will buffer, and grants more as its application reads. A session whose reader stops reading stops its writer and nobody else. That property is what link exists for, and the earlier post measured it.

The review found a hole next to it. Up to v0.8.1, (*Session).SetWriteDeadline returned nil and did nothing, and SetDeadline set only the read side. net.Conn documents both deadlines, every library in the comparison implements them, and http.Server.WriteTimeout relies on them, including in link's own recipe for HTTP over a link. A writer facing a reader that had stopped would wait for credit forever while its deadline said otherwise.

Since v0.9.0 a Write returns os.ErrDeadlineExceeded, with the count of bytes it did send, once the deadline passes while it waits for credit or for room in the connection's write buffer. While a deadline is set, the network write runs from a goroutine, so no Write waits on the network itself either. It was not a breaking change; it made an existing method do what its documentation already said.

The knobs stay few. The libraries compared offer a lot here:

link has a fixed window (1 MiB by default) and a session cap. That is predictable, and its worst case, 1 GiB of buffer per connection with the defaults, is written down in the security notes. A per-connection budget could come later as a Config field enforced locally by withholding grants, with no wire change; auto-tuning could come later with no API change. Neither blocks v1. The price today is a window that must be sized for the path by the caller.

Keeping the v1 surface small

Counted at v0.8.1, before the cleanup:

link is larger than the single-package multiplexers because its root package holds three tiers: the handshake, frames and credit for callers that run their own receive loop, and sessions. Most callers never see the lower two; the README sends newcomers to peerlink. The work before v1 was to take out of the surface what should never have been frozen.

• Conn embedded *Credit. Embedding put every Credit method (Acquire, Grant, Received, Forget and more) on Conn's method set, and Credit.Close was hidden behind Conn.Close only because the latter is shallower. It is now an accessor, Conn.Credit().
• bytequeue was a public package. Its API (PopUnless(*atomic.Bool), a TryPopBatch with three results, Wake) is the receive path's internals. Freezing it would have frozen the receive path. It is internal/bytequeue now. The README's own v1 checklist has the rule: no exported name that exists only for one caller.
• Observer is an interface link calls. Adding a method to an exported interface breaks every implementation, and new events (bytes, resets, handshake failures) are likely. We considered a struct of optional functions, as httptrace.ClientTrace does, and chose grpc-go's pattern instead: the interface has an unexported method that NopObserver provides, so implementations must embed NopObserver, and the stability policy says Observer may gain methods in minor releases.

The stability policy now separates the two kinds of interface. Interfaces link calls (Observer) may grow. Interfaces callers call (peerlink.Link, peerlink.Session, Authenticator) never change, and new capabilities arrive as optional interfaces. The conformance suite, peerlinktest, may add checks in minor releases: a test suite that cannot get stricter is not much use. The QUIC and gRPC implementations are separate modules and stay v0 while their transport's module does, because quiclink's API exposes quic-go's types.

What link left out

• CloseRead. libp2p, mplex and go-yamux have it, and its meaning differs between them (libp2p calls it implementation-defined). link's Close covers both directions.
• Stream priorities. quic-go has SetPriority; HTTP/2's priority tree is deprecated by RFC 9113. link schedules sessions fairly, and priorities would need wire support with no caller asking for them.
• Names on open. mplex has NewNamedStream(ctx, name) and libp2p has protocol ids. link has an opaque header []byte on every open, which is more general; the contract guarantees 4 KiB of it.
• Any io.ReadWriteCloser underneath. yamux and smux accept one. link takes a net.Conn, because it needs deadlines during the handshake and the TLS connection state for certificate authentication; net.Pipe covers in-process use.

The trade-offs, in one place

The review put the migration of our own code at about twelve files across the relay's two repositories, every one of them a compile error, with no behaviour change except that write deadlines became real. That migration shipped with nfltr v2.1.4. The changelog for v0.9.0 has a before-and-after line for each change.

What is next

v1.0.0 freezes the API once the README's maturity table holds. The peerlink contract has to stay unchanged for two minor releases. Fuzzing stands at 4.5 cumulative hours without a crasher in the library against a target of 24, and the soak under faults at 2.35 hours against 24. The security review of the handshake was done on 14 October. An apidiff gate in CI already fails any API change that is not listed in the repository, so a break between now and then is a decision, not an accident.