Bug — remote host: kaval didn't start, Restart → Internal server error (contract skew)

What was seen

Remote-hosting a newer local kolu (pureintent) to sincereintent — a 104-day-uptime box carrying kaval daemons up to 21 days old — produced a dead workspace, and every recovery attempt re-surfaced the same failure. Full-res: imgur album.

Clicking Restart kaval → a confirm → then a generic error toast:

Root cause — a pty-host contract skew, well-typed then squandered

The remote padi couldn’t connect to its kaval:

DaemonContractSkewError: pty-host contract skew: kaval speaks 5.0, server needs 5.2
  at connectKaval        packages/padi/src/ptyHost/connect.ts:97
  at recycleKaval        packages/padi/src/servePadi.ts:446

The local kolu needs pty-host contract 5.2; the kaval on the box speaks 5.0. sincereintent was a daemon graveyard — 65 orphan /tmp/padi-* state-roots and 21-day-old kavals from weeks of CI at many kolu versions — so the binder adopted an ancient, wrong-contract kaval. (The box hygiene that was the day’s darwin-flake substrate, biting the remote-host feature directly; the box wants a reboot regardless.)

The three defects (a-f-p + perfection, 12 findings, 0 refuted)

A · The skew flattens to INTERNAL_SERVER_ERROR. surface-map’s unary handler awaits the leaf with no translation (server.ts:534) — correctly, a keyed proxy can’t know a handler’s domain — so a plain DaemonContractSkewError (not an ORPCError) is serialized by oRPC as a generic error and the toast shows “Internal server error.” The precise cause is lost between padi’s throw and the browser.P5 / end-to-end: the actionability guarantee was placed at surface-map, a layer that cannot see what the error means. It belongs at padi’s recycleKaval handler, the knowing endpoint.

B · Restart kaval is a dead end. recycleKaval is kill-then-respawn of the same on-disk binary (a build constant; no re-fetch), then the same connectKaval handshake — so it re-throws the identical skew forever. On a remote host the kaval binary comes from padi’s own nix closure, so respawn can never yield a compatible one. The user is offered a recovery action the server has already proven cannot succeed.

C · The remote log bridge crashes per line. Not an old-log-shape problem — a plain unbound pino method call: (severity === "error" ? log.error : … : log.info)({host, line}, "…") extracts a bare function reference, so pino runs with this === undefined and throws Cannot read properties of undefined (reading 'Symbol(pino.msgPrefix)') on every line. It exists in both remotePadiBinding.ts:697 and padiBinding.ts:605; a try/catch one layer up (session.ts:562) swallows it, spams stderr per line, and drops every remote log. The version skew merely exercised this remote path hard for the first time.

The fix — two tiers

Tier 1 — the instances

• A → typed rethrow at the knowing endpoint. In recycleKaval’s catch: if (isContractSkewError(err)) throw new ORPCError("KAVAL_CONTRACT_SKEW", { data: { daemonVersion, requiredVersion } }) — mirroring the existing fileGoneAsNotFound pattern. Every generic hop then forwards it typed with zero new plumbing.
• B → refuse, don’t loop. recycleKaval on skew refuses typed (one attempt is diagnosis; a loop is a lie) — it is not the actor that can fix a remote skew.
• C → index the logger, don’t extract the method. log[severity]({…}, "…") (a call on the receiver, this bound; ternary deleted) at both sinks.
• DaemonContractSkewError carries its versions as readonly daemonVersion / requiredVersion fields; the message is derived from them (parse-don’t-validate — no consumer ever regexes the prose).

Tier 2 — fortify the surface framework (make the class inexpressible)

The three defects are one class each; the framework moves that kill the class:

• A typed error channel on defineSurface. Give a procedure a declared error union (oRPC supports contract-level typed errors) so a handler can only reject with a declared error and the client catch receives a discriminated union. Then “a typed daemon error arrives as an opaque INTERNAL_SERVER_ERROR” is a compile error, not a runtime surprise — the whole flatten-at-the-boundary class, not just this instance.
• Contract-skew as a first-class daemon STATE. Add an incompatible arm to DaemonStatus (carrying both versions), emitted by the endpoint when connect/recycle sees the skew, instead of collapsing into dead. UI affordances are then a total function of the state sum — the skew arm has no restart verb, so “a restart offered against a daemon restart can’t fix” is unspellable.
• The binder owns reprovision, not recycleKaval. One authority per fact (P3): the host’s desired build is the binder’s fact. Route the kaval skew into the same convergence vocabulary the padi axis already uses (reuse, not a parallel path), whose recovery affordance is the action that changes the binary (re-realise on the remote) — never restart the old one.
• makeSession takes a log: Logger and dispatches severity internally — deleting the duplicated per-consumer seam and the compensating try/catch. “A consumer hand-rolls severity dispatch over extracted method refs” has no spellable form left.

Phase tree (the full arc, final)

Phase	Outcome
This note — root cause + two-tier fix design	plan of record
#1876 — the skew card: name it, show both versions, offer the working recovery	✓ shipped
#1881 — isolation-default (per-client private estates)	✗ abandoned — orphaned every existing remote terminal on deploy
#1883 — renew re-point + honest client toast	✗ closed, zero product code — the product’s incompatible → renewDaemon was already correct; every client-side toast shape was falsified by the gauntlet
one-time stray reap (25494)	✓ done — the field box converges via existing code
#1887 — gate-less socket-squatter recovery (three-attestation identity)	✓ shipped — kolu heals its own rendezvous; the class fix
issue #1885 — server-authored honest “didn’t converge” toast	○ filed, awaiting a lane

Field addendum — the recovery was correct but insufficient (RENEW1, #1881)

Deployed and driven against the real graveyard box, the renew recovery converged the binaries but not the box: a competing old padi supervisor re-spawned its 5.0 kaval at the shared rendezvous after every recycle — mutual recycle, forever (two padis at one rendezvous each classify the other’s kaval as skew and recycle it — the livelock, endpoint.ts adoptAt skew arm). The card’s paths are individually correct; nothing arbitrated estate ownership. Two graveyard strata clarified the field: the digest-keyed kaval scheme is only ~14 days old (W2.2 cutover 590f49e48), so the 21–26-day survivors are pre-W2.2 port-keyed (a separate, irrelevant estate) — the live fight was the 5–6-day post-W2.2 padis sharing the default state-root digest.

The follow-up fix (#1881) is isolation, not arbitration — eviction was adversarially falsified (a live client re-dials and re-provisions its old closure; an old binary can’t honor a gate that postdates it): each kolu client gets a private remote estate keyed by a persisted client UUID (--client-id; the host spells the path). The fix kills nothing: a competitor’s 5.0 kaval simply lives in an estate the browser’s binder never touches.

A seamless auto-migration was designed and then dropped, in the gauntlet, for cause. The plan was a one-time --legacy-state-root adopt bridge (modeled on the W2.2 one) so a client’s existing terminals moved to its new isolated estate instead of being left behind. The codex review proved it re-created the very disease: two isolated clients (distinct UUIDs, distinct isolated gates) could both adopt the same live default estate — arbitrated by nothing, because the gate each holds is its own estate’s, not a shared migration claim — resurrecting the livelock; and it wasn’t even one-time (every no-override dial re-probed). A migration that can bring back the disease is worse than a loud one-time cutover, so #1881 ships clean isolation with an honest cutover: the pre-isolation default-estate daemons are named plainly as abandoned-running (their terminals are not migrated; the daemons sit idle until reboot or manual cleanup), and the binder emits a loud warning when it boots isolated and detects a live default-estate daemon it is leaving behind — never adopting it, never killing it.

Seamless migration done right is a recorded follow-up, two candidate shapes: (A) an atomic default-estate migration-claim (one acquirePidGate in the default estate so exactly one client adopts, once, plus a persisted cutover-done marker so reboots never re-probe — needs a two-client race test); or (B, recommended) a user-mediated explicit adopt affordance — one user action gives natural exclusivity with no race and no new shared rendezvous, and it matches the design’s own doctrine that sharing is an explicit choice, not an accident.

Lens artifact: workflow wf_97914292-c4b — 15 agents, a-f-p (P4/P5/P3) + perfection over the fix space, every recommendation adversarially verified against the real code (0 refuted).