Bug — a gate-less socket squatter wedges kolu forever

#1887

srid’s ruling, verbatim: “I want the bug to be fixed. WHY is a stray process breaking kolu?” The field incident was closed by hand-killing pid 25494 on sincereintent after it had wedged the box for 8 days — but a hand-reap is cheating. The product must converge on its own. This note is the plan of record for making it do so.

The wedge

The single-instance gate (kaval.pid) and the rendezvous socket (pty-host.sock) are two separate files. Normally they move together: the daemon holding the socket is the one the gate names. The bug is the state where they come apart — the gate file is gone (or names a different / dead pid) but a live orphan kaval still holds the socket. That state is representable, it happens in the field, and no supervisor arm handles it.

Trace it against the code at HEAD:

1. A fresh kolu-server boots and runs its convergence probe. The orphan’s socket is accepting, so the probe reads its system.version, sees an older pty-host contract, and decide returns recycle — routing the boot to the endpoint’s adoptOrEnsure.
2. adoptOrEnsure → adoptSurvivor calls liveServingHolder, which is gate-pid-only: it reads gatePid(gatePath), finds the gate absent, and returns undefined. The recycle arm (killLiveHolder) only ever fires against a gate-recorded pid, so it never targets the orphan.
3. With no gate holder, the boot falls to spawnConnectHold. It spawns a fresh kaval — which acquires the now-absent gate, then tries to bind the socket, is refused already-served by the live orphan, releases the gate and exits. The gate is empty again.
4. spawnConnectHold meanwhile sees the socket accepting (the orphan), dials it, handshakes the orphan, gets the contract skew, and reports incompatible.
5. The client shows the incompatible card. “Update & restart” re-runs the same boot. liveServingHolder reads the empty gate, returns undefined, and the loop repeats — forever. The socket is also non-durable (F1): if the orphan ever does close(), it unlinks whatever socket path a successor managed to bind.F1 is the file-ops half of the same wound — a successor that briefly binds by clearing a stale-looking inode can later have its socket path unlinked by the original squatter’s clean close(). The fix dissolves F1 by removal: the orphan is gone before the successor binds, so there is no later close() to unlink anything.

The fix — the supervisor owns its own socket namespace

Supervision of one’s own verified daemons is the supervisor’s job. When the gate gap opens but the socket is still held, the supervisor should identify the holder itself and recycle it — using identity, not bookkeeping, as the safety line.

The arm fires only in the gate-less-squatter state and does three things:

• IDENTIFY — ask the OS which pid listens on this exact socket path. A dependency-free platform leaf: on Linux, the bound listener is the one row in /proc/net/unix carrying the path (state 01, the SO_ACCEPTCON flag), and its inode maps to a pid via /proc/<pid>/fd/* → socket:[inode]; on darwin, lsof against the path.The field case ran on sincereintent (darwin), so the lsof path is production-load-bearing, not a nicety — it is verified on a real mac in acceptance, not assumed. The Linux /proc parse is already grounded empirically (exact column layout + inode→pid mapping).
• VERIFY IDENTITY by handshake — dial the socket and run the existing kaval version exchange (the very handshake that already detects skew), then act on what it proves:
  • skewed daemon → the wedge (the 25494 case): a daemon of this estate we cannot serve through. Its disposition is the ONE thing that follows the boot policy — the SAME recycle-vs-refuse split the endpoint already makes for a gate-recorded skew:
    • under the kaval / always-recycle policy → RECYCLE it — SIGTERM the identified pid, waitForPidGone, then bind fresh (the wait/ceiling machinery is killLiveHolder’s, reused verbatim; the only new thing is the identity source).
    • under the padi binder / refuse policy (adoptOrSpawnOrRefuse) → leave it standing and report incompatible, never SIGTERM it — a client must not kill a running padi even a skewed gate-less one (#1313).Caught by the architecture-first-principles gate (C4 depends-on): the first cut hardcoded skew→kill in the shared recovery, so a gate-less skewed padi would have been SIGTERM’d under the very policy whose contract is “never kill a running padi”. The skew disposition now threads the caller’s policy, exactly as the gate-recorded path already did.
  • compatible kaval → left for the ordinary connect to adopt, exactly as today — its live PTYs are preserved, not killed. A compatible gate-less holder never wedged; recycling it would be a PTY-loss regression, so recovery is a no-op there.This is a build-time refinement of the approved design’s literal “recycle any verified orphan”: grounding showed that today a compatible gate-less holder is already connected-to (adopted) by the spawn path, so only the skewed holder ever wedged. Recycling a compatible orphan would kill PTYs that survive today — a regression the skew-only recycle avoids. The identity/safety model is unchanged; only the compatible-holder disposition is adopt-not-recycle.
  • does not speak kaval → a genuinely foreign process (or a non-conforming speaker), and we fail loud (a typed SocketSquatterForeignError naming its pid + command) and never touch it.

We also never recover a socket held by our own process: the supervisor spawns its daemon as a separate process, so a holder equal to process.pid is never a real squatter — filtering it out is both correct and a self-kill guard.

This makes the existing incompatible → renewDaemon flow self-heal: a fresh padi boots → socket held gate-less → handshake → old kaval → recycle → bind the new contract → converge. It is srid’s “stop the old kaval, start the new kaval”, extended from gate-recorded daemons to verified orphans.

The identity model (the load-bearing decision)

The orchestration teardown law — agents kill only the pids they recorded — does not govern the product’s own supervisor. A supervisor recycling a daemon it has just proven, over the wire, to be its own kind is doing its job, not violating a bookkeeping rule. The safety line is IDENTITY, made of three independent attestations that must agree before any signal is sent:

Attestation	Source	Guarantees
The socket speaks kaval	the version handshake succeeds (compatible or skewed)	it is our daemon, not a stranger
A specific pid holds this socket	the OS socket-holder lookup	we know exactly whom to signal
That pid is the one that answered	handshake self-reports its pidkaval’s system.version returns { contractVersion, pid, startedAt, identity?, lifetime? } — the daemon’s own process.pid. pid is a required wire field, present since the first standalone kaval (#1301) — so it holds for every real orphan, including a 25494-era 5.0-contract skew. The recycle target is thus a pid the daemon named over the very socket we are recycling, cross-checked against the OS’s independent view. No new wire field is added (the field already exists); the added exported-type surface is an additive pid? on DaemonContractSkewError — see “Exported-API additions”.	the process answering IS the process holding the socket

Only when all three agree — and are re-confirmed unchanged immediately before the SIGTERM — is the orphan recycled. Any disagreement (foreign process, pid the OS doesn’t corroborate, holder that moved between identify and kill) aborts the kill and fails loud. We never kill anything that didn’t handshake as kaval, and we never touch another daemon’s socket namespace.

The pid-absent arm is FOREIGN, by construction — not a weaker fallback. Because pid is a required field of the system.version schema, a response that omits it (a hypothetical pre-#1301 daemon, or any non-conforming speaker that merely accepts the socket) fails oRPC output validation — the read throws, which is exactly the “did not answer the handshake” path that classifies the holder as foreign: a loud typed error naming pid + command, and never a kill. So the identity model never degrades to a two-attestation kill when attestation 3 is missing — a missing self-reported pid is itself the proof the holder is not a conforming kaval of this estate, and the fail-fast / no-fallbacks rule holds (a degrade-to-2 path would have been the very fallback this codebase forbids). This is stated and pinned, not left as emergent behaviour.

Where it lives (structure)

The recovery arm grafts onto the endpoint spine (@kolu/surface-daemon-supervisor), beside the machinery it reuses — the endpoint already owns killLiveHolder, waitForPidGone, spawnConnectHold, and the gate read. It is a Node-only module (it already imports gatePid / isHolderLive and dials node:net), so a new sibling leaf is a clean fit:

• socketHolder.ts — a new leaf next to waitForPidGone.ts: socketHolders(socketPath) → SocketHolder[] (each { pid, command }), platform-dispatched (/proc on linux, lsof on darwin), fail-fast on an unsupported platform. It returns every pid the OS reports holding the path — a list, not a single pid: linux’s /proc/net/unix names the bound listener exactly, but darwin’s lsof can report the listener plus connected clients, so the caller’s handshake-reported pid selects the true listener from the set (the recovery kills only a pid the OS corroborates AND the daemon named over the socket). It hides a bounded OS-lookup volatility — a leaf, not electricity (no transport / reconnect / persistence lifecycle of its own), so it belongs in the package that uses it, not a new @kolu/* receptacle.
• the recovery arm — a private helper in endpoint.ts, run by recoverOrSpawn at exactly the boot points where no live gate holder was found yet the endpoint is about to spawn: ensure’s else branch and adoptSurvivor’s no-survivor branch (hence, via converge and the renew flow, every wedge path). It is deliberately not on the recycle path — that has already reaped its holder and freed the socket, so re-probing there would only re-handshake a daemon we just replaced (and, against an in-process test daemon, mis-fire).The first cut placed the arm at the top of spawnConnectHold for “uniform coverage”, but recycle calls spawnConnectHold too — so the arm ran an extra handshake on every recycle. Scoping it to the no-gate-holder branches covers the same wedge paths (a wedge is always a no-gate-holder boot) without touching the recycle path. “Speaks kaval?” reuses the existing connectSurvivor three-way verdict (adopted / skew / unreachable), so the handshake logic is not re-implemented.

Exported-API additions (named honestly for the consumer gates)

The recycle only ever fires on a skew, and a skew throws DaemonContractSkewError before a DaemonConnection is ever built — so the pid the kill cross-checks rides on the error, not the connection. The surface additions to @kolu/surface-daemon-supervisor are therefore:

• pid?: number on DaemonContractSkewError — the skewed daemon’s self-reported pid, so the gate-less-squatter recovery has its third identity attestation (additive, optional);
• SocketSquatterForeignError / isSocketSquatterForeignError and socketHolders / SocketHolder — genuinely new exports.

A pid on the connection (DaemonConnection) was not added: a compatible holder is adopted, never killed, so nothing would ever read it — a dead field is worse than none.Caught by the architecture-first-principles gate (C2 consumer-ergonomics): the first cut added pid to DaemonConnection too, but the recovery reads the pid only off the skew error (the kill path), and the adopt path disposes the connection without touching it — a field with zero readers and a doc comment that lied about who read it. Deleted. The DaemonContractSkewError.pid change is additive-optional, so no existing consumer breaks — verified, not asserted: the drishti and odu consumer greps found neither repo depends on @kolu/surface-daemon-supervisor, so the additive change has zero cross-repo impact. ref-surface-supervisor.mdx is updated in the same PR. The one genuinely new capability is the OS socket-holder lookup.

Design philosophy

• Fail fast — no fallbacks. The foreign-holder arm and every identity disagreement crash loud (a typed error naming the pid + command); there is no degrade-to-spawn-anyway path that could stomp a stranger. An unsupported platform throws rather than silently skipping recovery.
• Volatility boundary. The OS socket-holder lookup is a bounded platform leaf, co-located with waitForPidGone (its twin — pid lifecycle via the OS). It is not extracted into a package: it hides an algorithm, not electricity.
• Reuse the source of truth. The recycle wait/ceiling is killLiveHolder’s; the “speaks kaval?” verdict is connectSurvivor’s; the self-reported pid is the existing system.version field. The one genuinely new capability is the OS lookup.

Acceptance — the pins

Each pin is red on today’s code before the fix and green after.

Pin	Manufacture	Today	After
The squatter	spawn a real kaval, delete its gate file, boot/recycle	wedged: fresh cannot bind; card loops	holder identified → handshaked → recycled; fresh binds; durable (old holder gone)
Foreign holder	a plain non-kaval net server bound at the path	(falls through / opaque)	not killed; loud typed error naming pid + command
Pid-absent speaker	a socket that accepts but whose system.version omits pid (fails schema)	(falls through / opaque)	classified foreign → not killed; loud typed error
Compatible holder	a gate-less holder that speaks a compatible contract	(connected/adopted)	adopted, not killed — PTYs preserved (no regression)
Gate-present unchanged	the existing killLiveHolder / adopt suites	green	green
Version-agnostic	an OLD-contract kaval (handshake succeeds, version skews)	wedged (this IS 25494)	identified as kaval → recycled

The end-to-end proof, on a pu box (or sincereintent if offered): manufacture the gate-less squatter, then drive the real UI flow — the incompatible card’s “Update & restart” — and watch it converge one click, no manual reaping. The point is not that a test passes; it is that the product heals itself.

The adversaries this must survive (for the lens gate)

• pid-reuse between lookup and kill — the identified pid could exit and the OS reuse it before the SIGTERM. Mitigation: the kill target is the handshake’s self-reported pid, cross-checked against the OS holder, and re-confirmed as the current holder immediately before the signal; a moved holder aborts the kill. The final re-confirm→SIGTERM window is irreducible — there is no atomic check-and-kill syscall — so it is bounded, not closed: a code comment at the kill site cites the killLiveHolder precedent (endpoint.ts:415-419, which already lives with the identical race on a gate pid) so a future reader sees the window is a considered bound, not an oversight. It is strictly smaller here (three attestations agreeing vs. one).
• TOCTOU on the socket — the squatter may close between “accepting” and the handshake, or between handshake and kill. Every transition re-checks; a vanished holder simply falls through to the ordinary fresh spawn.
• the foreign-process arm — a non-kaval must be impossible to kill. It is gated behind a successful kaval handshake, never behind the OS lookup alone.
• platform lookup differences — linux (/proc) is grounded; darwin (lsof, the field platform) is verified on a real mac in acceptance, never assumed.