How to operate a fleet safely

A fleet multiplies the ways a surface can be quietly wrong: one host runs an older agent, two writers disagree about who is a member, one dead host drags the whole map down. This guide covers the three habits that keep a fleet honest.

Gate every host on contract compatibility

Each host serves its own contract version. A fleet holds hosts at different versions at once, so never compare version strings with == — an additive minor bump is still compatible, and a bare string compare would flag a working host as broken. Gate on isContractVersionCompatible(reported, expected), which passes when the majors match and the reported minor is at or above yours.

// Read the version off the host's identity cell and gate on it BEFORE invoking
// anything else — an additive minor bump is still compatible, so never `==`.
const versionSink = (): SurfaceSink<typeof versioned.spec> => ({
  cells: {
    version: (v) =>
      setSkew(!isContractVersionCompatible(v.contractVersion, OURS)),
  },
});

Read the version off the host’s identity cell, check it before invoking anything else, and turn a mismatch into an honest “restart the other side” message rather than an opaque schema error from deep in the RPC layer. Surface the mismatch as the entry’s failed cause so the chip shows an actionable card.

Keep one writer of membership

Membership must have exactly one author. In a map, that author is the MapRegistry: entries is the truth it publishes, and status is derived from each session’s connection state — there is no second writer. Do not maintain a parallel record of “which hosts exist” beside it.

• A key absent from entries is “not a member” — there is no absent status arm to keep in sync.
• Add, remove, or retire a host through the pool’s add/remove/retire; entries follows every one.
• When a host leaves mid-stream — a user remove or an internal retire — its subscriptions complete cleanly (a typed stream end, not an error) before the session is destroyed, and the browser socket closes with a normal 1000 close whose reason names the verb (host removed / host retired). A consumer sees a clean typed end, never a socket-error frame after it.

Isolate each host’s failures

A fleet’s failures must stay local to the host that had them. cause is the transport CLASS (not the retry policy); the session state machine isolates two kinds:

• A network fault (host asleep, roaming, VPN down) normally retries at capped backoff forever, so a laptop that closes at home reconnects on its own at a café. It projects as warming — in motion, no action needed. The ONE terminal exception (#1908): a budget-exhausted silent provisioning step — the ssh connector kills a wedged copy/build enough consecutive times that its progress-liveness budget gives up — surfaces terminal failed + network (the honest transport cause; terminality is the phase, not a rewrite to remote).
• A remote fault (host answered but rejected the closure — e.g. the user isn’t in trusted-users) is bounded: after a few consecutive failures the session surfaces terminal failed + remote, and the map shows that one host’s chip as needing intervention. The other hosts are untouched.

On a long-running parent, call recheck() on every session when the machine wakes or regains connectivity — a post-sleep link is often stale (the far end dropped the socket but local ssh won’t notice for ~30s), and recheck() force-cycles even a connected link. Use reconnect() for the manual “Reconnect” button; it re-arms a failed session without disturbing a live one.

For why warming and failed mean exactly this, see Entry contracts.