R2 — kaval, the standalone PTY daemon

R2 of remote terminals — the local PTY survivor. Your terminals (process · scrollback · running agent) outlive a kolu deploy because they don’t live in kolu: they live in kaval (Tamil kāval — watch, guard), a standalone daemon kolu dials. kolu-server restarts every deploy; kaval keeps holding the fds. Shipped end-to-end (R2.1 → R2.5), closing #671. This note is the build log; the cross-host shape lives in the parent.

The first build (#1275) shipped the whole feature as one 40-commit PR — verified live in prod, then deliberately discarded: functional, but the architecture was discovered in review, not designed. What follows is the redo — same behavior, a designed boundary, sized one PR per agent session.

Architecture — one rule, then a module map

The one rule everything follows from: package boundary = process boundary = staleKey hash, with zero file-level exceptions. kaval is exactly the code that runs in the daemon — the PTY primitive, the wire, the taps, the socket, and the process entry. Its staleKey is the nix hash of those package dirs, each hashed whole; the closure test walks the import graph from kaval’s entries and answers the one question the key asks — what would a restart gain? Everything that supervises from outside lives outside the hash, because changing the supervisor never changes what a restart gains.

kaval is a program of its own; kolu is its first client. Like drishti (graduated from a monitor example) and odu (from mini-ci), kaval is a surface app that was trapped inside kolu-server — ptyHostSurface was already a defineSurface() all along. The layering: dumb-but-durable kaval ← kolu the session brain ← kolu the web UI. kaval holds fds, mirrors screens, serves taps — nothing else. kolu stays a substantive middle tier on purpose: the #1031 postmortem is binding — daemonizing the provider DAG served stale detection every deploy, so session persistence, reconciliation, the DAG, and all spawn policy stay kolu’s, re-run fresh against the surviving PTYs. kolu is not a thin client, but it is a client (kaval-tui today, an MCP face later).

The shared daemon spine (surface-daemon) is split in two: @kolu/surface-daemon (the daemon half — atomic pid-gate, the daemonMain skeleton) and @kolu/surface-daemon-supervisor (the supervisor half — endpoint state machine, waitForPidGone, composed restart, survivable-spawn driver). Both are our libraries with our consumers (kolu today, odu serve next — see odu-runner), changed freely, no backwards-compat tax. Mechanism upstreams; only kolu’s session/terminal policy stays soul. The line every PR draws: the spine adopts/recycles/serializes a connection; kolu reconciles that connection’s contents.

Multi-host readiness (shapes, not the feature). R2 ships exactly one endpoint but is host-count-agnostic in shape, so R3 retrofits nothing: the endpoint map is keyed by hostId; daemon status is a per-host collection; adoption joins on id (→ (host,id) in R3); the HostLocation discriminator ({kind:'local'} → {kind:'remote',hostId}) is the dispatch seam (R3.1/#1364 inlined it). A remote pty-host is the same kaval closure shipped by @kolu/surface-nix-host — nix copy, realise, run — exactly how odu provisions its runner; R2.1’s system.info makes spawn policy computable for a host that isn’t kolu’s machine, and initFiles lets rcfiles land where kolu’s hands can’t reach.

What this makes impossible by construction

The production failures — four from #1034, four more paid for during #1275 — each become a failing test at the phase that owns the concept.

Production failure	Killed at	How
Mis-scoped staleness key	R2.2	Closure-scoped key + import-walk guard, re-rooted at the daemon entry.
Over-prompting (key nudged every deploy)	R2.4.4	Keyed on the closure hash; a server-only change leaves it bit-identical — a falsifiable test.
Data-loss restart (kill-then-pray, #1034)	R2.4.2	One composed restart; snapshot-before-kill is the capture step; the drain fires no autosave to clobber it.
Empty-canvas lie (dead daemon → “no terminals”)	R2.3	Honest dead/degraded state ships with the door, before any survival promise.
Lossy adoption (#1275)	R2.4.3	Whole-record adopt; a schema-key round-trip test closes the class; a non-survivor is a dropped exited shell, never an autosave-clobbered restore card.
Lazy adopt-on-spawn → duplicate terminals (#1275)	R2.4.3 · R2.5	Both adopt paths adopt from the live snapshot, never re-spawn, guarding on the registry so kolu’s own spawn echo is a no-op.
Identity stale after restart (#1275)	R2.3	One status owner emitting on every transition; everything derives by subscription.
Contract-skew crash-loop masked as an “App updated” loop	R2.3	Version checked over the socket on every connect, never an import-time throw; skew → controlled recycle.

The build — five steps, each complete w.r.t. its own hazards

No feature flags — each PR ships complete to master. The refined hazard rule: each PR is complete w.r.t. the hazards it opens — the inversion opens none (byte-identical refactor), and the door can open with survival off, which empties its hazard set by policy.

Step	Ships	PR
R2.1 spawn-policy inversion	The wire becomes fully specified — spawn {id,argv,cwd,env,initFiles} + system.info → {shell,home,platform,rcDir} (contract 3.0). All kolu-isms move client-side; the daemon writes the rcfiles it’s handed and asks nothing. In-process, zero user-visible change, made at the one moment it’s free. Makes a remote kaval computable.	#1292
R2.2 the binary + its client	pty-host → kaval, pty-tui → kaval-tui, both with bins, runnable as a pair on a box where kolu was never installed. @kolu/surface-daemon born here (gate + daemonMain skeleton, hashed as a staleKey root). Full e2e: the contract corpus over both links + a coverage-ledger meta-test walking every contract key.	#1301
R2.3 the door (topology flip)	kolu-server stops serving its own pty-host socket and becomes a client of a kaval it spawns. Boot policy is always-recycle (no survivors → no survival hazard), so every deploy exercises kill → wait-for-real-exit → respawn — the exact #1034 race, with zero sessions at stake. @kolu/surface-daemon-supervisor born here. Honest degraded state + the live KAVAL rail column ship with the door.	#1310
R2.4 the survival chain	Terminals survive a deploy. Four PRs (below).	—
R2.5 live inventory	A kaval-tui create against kolu’s own daemon now appears in kolu live.	#1458

R2.4 — survival, four PRs not one

The first build did survival as one ~1800-line PR (#1326, closed): too big to review (two blocking data-loss bugs survived its own gauntlet), the spine grew mid-implementation, and a misframed edge case dragged a race-sensitive autosave swamp into the diff that wasn’t even needed (see the crux). The redo is a shallow chain — one refactor, then one capability each.

PR	Ships	Hazard killed
R2.4.1 #1330	Pure byte-identical refactor: split ensure() into liveServingHolder/killLiveHolder/spawnConnectHold; lift killHalfWiredPty (the shared reap receptacle); dedup the snapshot shape into one SessionSnapshot type. Every extraction has a live consumer (the empty→null guard moved to R2.4.2, beside its consumer — codex flagged consumerless future-API).	—
R2.4.2 #1337	Session-preserving restart of a running daemon (pick up a new build · user-initiated) and a dead one (recover) — finishes the door’s deferred “Restart kaval” button. Spine: restarting state + serializeRestart (coalesce concurrent triggers) + holdRestarting (one honest state across the recycle). Soul: setSavedSessionFromSnapshot — an unconditional autosave-cancel so the capture survives the kill.	#1034 (snapshot-before-kill)
R2.4.3 #1344	Live PTYs survive a deploy that didn’t change kaval’s source (staleKey unchanged — the common case). Spine: adoptOrEnsure — adopt a live handshake-compatible survivor (connect, never kill), recycle only an absent/dead/genuinely-skewed one; a survivor connect is retried before concluding skew, so a daemon we merely can’t reach now keeps its PTYs. Soul: reconcile.ts + adoptTerminal (whole-record, never field-by-field).	#1275 (whole-record adopt)
R2.4.4 #1353	Amber ⬆ update pending on the rail when the adopted daemon is a build behind → one-click recycle. Derived at the read site (expectedKaval vs reported staleKey), gated on state==="connected", keyed on staleKey never the per-deploy commit. CI gate: the build-id reaches the server.	#1034 over-prompting

The crux — the partial case dissolves; #1326’s swamp was a conflation. Every #1326 bug came from treating an adopt-case non-survivor like a restart-case one. They differ:

Case	A non-survivor is…	Right move
Restart — daemon killed, all PTYs die	a terminal you still want	restore it on the empty canvas (no survivors, no autosave race)
Adoption — daemon survived, one PTY gone	an exited shell	drop it — exactly what handleExit already does

So adoption’s “partial” case is trivial — adopt the live, drop the exited — no restore card, no recycle, no autosave-durability machinery. #1326 mis-applied the restart-case restore card to the adopt case, forcing a pendingRestoreCard/union cluster that burned four codex rounds for a problem that doesn’t exist.

R2.4.4’s two operands — both already baked into nix, the code just names and compares them at the read site: reported = the adopted daemon’s own staleKey (already on the wire via daemonStatus.identity); expected = the server’s KAVAL_BUILD_ID (one ${kavalBuildId} nix---sets onto both the koluBin wrapper and the kaval bin), surfaced as an additive-optional buildInfo.expectedKaval — no contract bump, so no surviving daemon is force-restarted. Because the key is a content-hash of kaval’s daemon source closure only, a server-/client-only deploy leaves it bit-identical and the nudge stays silent — the #1034 over-prompting fix, by construction.

R2.5 — live inventory, the two-way reach

R2.4.3 reconciles the daemon’s live PTYs against the saved session once, at boot — so a terminal created out-of-band (a kaval-tui create against the very daemon kolu is a client of) stayed invisible until the next restart. The contract had only per-terminal taps and a one-shot list: no way to learn that another client spawned a PTY.

R2.5 closes that with a host-global inventory stream on ptyHostSurface (contract 3.0 → 3.1, additive · minor — a 3.0 survivor lacking it is a hard skew, forced to recycle): snapshot-then-created/exited. kolu-server subscribes once and feeds every unknown id into the existing adopt path, so the daemon’s entries map becomes the single source of truth and kolu’s registry a continuous projection. The reach is now two-way: kaval-tui list reaches kolu’s terminals, and a kaval-tui create appears in kolu live. (A bare adopted PTY has no kolu OSC hooks, so its tile shows live shell + scrollback but no agent/title detection — an inherent ceiling, not a regression.)

Design notes that carry forward

The survivor is kaval only — node-pty fds + the @xterm/headless mirror + the raw VT taps + a unix socket + its own process entry. A kolu-server restart re-runs the providers against the surviving PTYs, so detection is never stale while the PTYs persist. Honest cost: metadata is no longer “warm” across a restart — a brief re-detection pass, trading warm-on-reconnect for freshness-on-deploy. Only a (rare) contract change forces terminal loss; a (frequent) provider change restarts the cheap layer with PTYs untouched. This corrects #1031, which daemonized a survivor that held the providers.

The cgroup survival mechanism (spike-verified). On Linux/systemd kaval spawns via systemd-run --user (gated on fromSource + INVOCATION_ID), landing in its own transient cgroup — a plain detached/setsid child does not survive on cgroup-v2 (KillMode=control-group walks cgroup membership, the #1031 Linux failure). macOS’s detached spawn already survives launchd. Caveats: linger on, absolute daemon path, per-spawn unique unit names (a dead unit can linger loaded). Single-instance two ways: the unit name plus the atomic pid-gate. kaval namespaces its socket+gate per kolu-server instance by listen port (kaval-<port>/) since #1313, so two servers never recycle each other’s daemon.

tmux/dtach considered and rejected: they only keep a PTY alive — no OSC-parsed taps, no headless snapshot for lazy-attach, no home for the provider DAG. You’d still build kaval’s streaming layer next to tmux and inherit its session model on top.

The #1034 hard constraints (binding on the redo)

The #1034 build (#1034) shipped the nudge + a Restart command; on prod, clicking restart destroyed a live 20-terminal session and couldn’t bring the daemon back — killAll drained all 20, the old daemon (25G RAM, thrashing box) took ~2min to exit, the respawn timed out at 30s, leaving an empty canvas indistinguishable from “no terminals.” The respawn lost the race against the slow exit. The constraints it bought:

1. Restart is recoverable, never kill-then-pray — snapshot first; if respawn fails, a loud degraded state with the session preserved.
2. Wait for real exit (kill(pid,0) → ESRCH) before spawning — the single-instance lock fights the restart.
3. Timeouts fit a loaded prod box, not an idle dev one.
4. Never lie about state — explicit connecting/degraded UI, never silent emptiness.
5. Key staleness on kaval, not the whole binary — so the nudge fires only when a restart actually gains something.

History

• #1031 → #1034 → #1275 — three dead-ends that bought the design. #1031 daemonized the provider DAG (stale detection every deploy); #1034 was the kill-then-pray postmortem above; #1275 shipped the whole feature as one 40-commit PR, verified live, then discarded for an architecture discovered in review rather than designed. Earlier prototypes (#994 remote-providers, #1010 PTY-only local daemon) seeded the thin-survivor model; the redo is their correct combination.
• The kaval reframing (2026-06-12) — a coupling audit found four ties to kolu (no bin, daemon-side spawn policy, hardcoded socket app-name, kolu build-id env). The fix: invert the wire (R2.1), then ship the standalone binary (R2.2), then flip the topology (R2.3), then survival (R2.4). surface-daemon (#1294) named the shared daemon spine with odu serve as its second tenant; #1313 namespaced the daemon per kolu-server instance.
• R2.1–R2.5 shipped (2026-06-12 → 06-14, plus R2.5 #1458) — the inversion (contract 3.0, parity tests, zero kolu-* deps), the binary (full e2e over both links + the coverage ledger), the door (the topology flip + honest degraded state), the four-PR survival chain (refactor → restart → adoption → currency nudge), and the live-inventory two-way reach. The currency nudge’s VM gate (adopt.nix = no-nudge-on-no-op, a build-skew sibling of skew.nix) was pu-verified green-on-correct, red-under-mutation. Per-PR detail (gauntlet finds, as-built deltas) lives on each PR.