kolu
Docs

How to operate a fleet safely

A fleet multiplies the ways a surface can be quietly wrong: one host runs an older agent, two writers disagree about who is a member, one dead host drags the whole map down. This guide covers the three habits that keep a fleet honest.

Gate every host on contract compatibility

Each host serves its own contract version. A fleet holds hosts at different versions at once, so never compare version strings with == — an additive minor bump is still compatible, and a bare string compare would flag a working host as broken. Gate on isContractVersionCompatible(reported, expected), which passes when the majors match and the reported minor is at or above yours.

// Read the version off the host's identity cell and gate on it BEFORE invoking
// anything else — an additive minor bump is still compatible, so never `==`.
const versionSink = (): SurfaceSink<typeof versioned.spec> => ({
  cells: {
    version: (v) =>
      setSkew(!isContractVersionCompatible(v.contractVersion, OURS)),
  },
});

Read the version off the host’s identity cell, and check it before you invoke anything else. Turn a mismatch into a plain “restart the other side” message rather than an opaque schema error from deep in the RPC layer. Surface the mismatch as the entry’s failed cause, so the chip shows an actionable card.

Keep one writer of membership

Membership must have exactly one author. In a map, that author is the MapRegistry: entries is the truth it publishes, and status is derived from each session’s connection state. Do not keep a parallel record of which hosts exist beside it.

  • A key absent from entries means “not a member”. There is no absent status arm to keep in sync.
  • Add, remove, or retire a host through the pool’s add, remove and retire. entries follows every one.
  • When a host leaves mid-stream, through a user remove or an internal retire, its subscriptions complete cleanly before the session is destroyed. They get a typed stream end, not an error. The browser socket then closes with a normal 1000 close whose reason names the verb (host removed or host retired), so a consumer never sees a socket-error frame after the end.
  • remove and retire run the same live teardown, and they answer different questions. remove is the user’s departure: it forgets the host, and the persist hook writes the shrunk set. retire is an internal shed of a dead session: it drops the host from live entries and leaves the remembered set untouched, so a membership store re-seeds it next boot.

Live membership, which is what entries and has report, is the map’s single-writer truth. The remembered set is a separate fact the pool keeps, and a retired host is deliberately in one but not the other.

Isolate each host’s failures

A fleet’s failures must stay local to the host that had them. cause is the transport class, not the retry policy. The session state machine isolates two kinds:

  • A network fault (host asleep, roaming, VPN down) normally retries at capped backoff forever, so a laptop that closes at home reconnects on its own at a café. It projects as warming: in motion, no action needed.
  • A remote fault means the host answered and the attempt failed, such as a Nix build on the host. It is bounded: after a few consecutive failures the session surfaces terminal failed with cause remote, and the map shows that one host’s chip as needing intervention. The other hosts are untouched.

One network case is terminal: a provisioning step that exhausts its budget silently. The ssh connector kills a wedged copy or build enough consecutive times that its progress-liveness budget gives up, and the session surfaces terminal failed with cause network. Terminality is the phase here, not a rewrite of the cause to remote.

On a long-running parent, call recheck() on every session when the machine wakes or regains connectivity. A post-sleep link is often stale: the far end dropped the socket, and local ssh does not notice until this dial’s keepalive tolerance elapses (about 30s by default, and longer on a dial tuned for an unattended lane). recheck() force-cycles even a connected link. Use reconnect() for the manual Reconnect button; it re-arms a failed session without disturbing a live one.

For why warming and failed mean exactly this, see Entry contracts.