How to operate a fleet safely
A fleet multiplies the ways a surface can be quietly wrong: one host runs an older agent, two writers disagree about who is a member, one dead host drags the whole map down. This guide covers the three habits that keep a fleet honest.
Gate every host on contract compatibility
Each host serves its own contract version. A fleet holds hosts at different
versions at once, so never compare version strings with == — an additive minor
bump is still compatible, and a bare string compare would flag a working host as
broken. Gate on isContractVersionCompatible(reported, expected), which passes
when the majors match and the reported minor is at or above yours.
// Read the version off the host's identity cell and gate on it BEFORE invoking
// anything else — an additive minor bump is still compatible, so never `==`.
const versionSink = (): SurfaceSink<typeof versioned.spec> => ({
cells: {
version: (v) =>
setSkew(!isContractVersionCompatible(v.contractVersion, OURS)),
},
});
Read the version off the host’s identity cell, and check it before you invoke
anything else. Turn a mismatch into a plain “restart the other side” message
rather than an opaque schema error from deep in the RPC layer. Surface the
mismatch as the entry’s failed cause, so the chip shows an actionable card.
Keep one writer of membership
Membership must have exactly one author. In a map, that author is the
MapRegistry: entries is the truth it publishes, and status is derived from
each session’s connection state. Do not keep a parallel record of which hosts
exist beside it.
- A key absent from
entriesmeans “not a member”. There is noabsentstatus arm to keep in sync. - Add, remove, or retire a host through the pool’s
add,removeandretire.entriesfollows every one. - When a host leaves mid-stream, through a user
removeor an internalretire, its subscriptions complete cleanly before the session is destroyed. They get a typed stream end, not an error. The browser socket then closes with a normal1000close whose reason names the verb (host removedorhost retired), so a consumer never sees a socket-error frame after the end. removeandretirerun the same live teardown, and they answer different questions.removeis the user’s departure: it forgets the host, and thepersisthook writes the shrunk set.retireis an internal shed of a dead session: it drops the host from liveentriesand leaves the remembered set untouched, so a membership store re-seeds it next boot.
Live membership, which is what entries and has report, is the map’s
single-writer truth. The remembered set is a separate fact the pool keeps, and a
retired host is deliberately in one but not the other.
Isolate each host’s failures
A fleet’s failures must stay local to the host that had them. cause is the
transport class, not the retry policy. The session state machine isolates two
kinds:
- A network fault (host asleep, roaming, VPN down) normally retries at capped
backoff forever, so a laptop that closes at home reconnects on its own at a
café. It projects as
warming: in motion, no action needed. - A remote fault means the host answered and the attempt failed, such as a Nix
build on the host. It is bounded: after a few consecutive failures the session
surfaces terminal
failedwith causeremote, and the map shows that one host’s chip as needing intervention. The other hosts are untouched.
One network case is terminal: a provisioning step that exhausts its budget
silently. The ssh connector kills a wedged copy or build enough consecutive
times that its progress-liveness budget gives up, and the session surfaces
terminal failed with cause network. Terminality is the phase here, not a
rewrite of the cause to remote.
On a long-running parent, call recheck() on every session when the machine
wakes or regains connectivity. A post-sleep link is often stale: the far end
dropped the socket, and local ssh does not notice until this dial’s
keepalive tolerance elapses (about 30s by
default, and longer on a dial tuned for an unattended lane). recheck()
force-cycles even a connected link. Use reconnect() for the manual
Reconnect button; it re-arms a failed session without disturbing a live
one.
For why warming and failed mean exactly this, see
Entry contracts.