kolu
Docs

@kolu/surface-daemon reference

The daemon binary half — the code that runs inside the long-lived process. It carries no @kolu/* app dependency; the client half lives separately in @kolu/surface-daemon-supervisor. Production imports use the package root; mixed-version suites use the dedicated ./upgrade-window.testlib subpath; pid-gate test seams (connect-probe override) use ./pidGate.testlib — never re-exported from the production root.

The frozen control core

controlCoreFragment({ stateRoot, surfaceVersion, startedAt, commit, buildId, onDrain }) implements the immutable core.hello identity and core.drain verb. It is served beside the versioned application surface — composed as the sibling control, so the two members sit at the wire tags surface/control/core/hello and surface/control/core/drain and cannot collide with anything the application surface mints. A supervisor can therefore identify and drain a resident daemon before knowing whether that versioned surface is compatible. The hello fields are fixed: stateRoot · surfaceVersion · controlCoreVersion · startedAt · commit · buildId. CONTROL_CORE_VERSION is frozen at "1.0"; it is not an application contract version. Build id and commit are one observation: the shared reader accepts both absent, both empty (off-Nix), or both non-empty, and rejects every partial pair before convergence policy runs.

The fragment never versions — within a protocol epoch. The Effect-4 migration moved this wire’s framing (the oRPC base64+newline peer codec → Effect RPC ndjson). That is a declared flag day, not a version negotiation: negotiation happens inside the protocol being replaced, so a daemon from the previous epoch cannot be asked anything at all — its first frame is undecodable, and the supervisor observes that as unspeakable-protocol, never as an incompatible version. CONTROL_CORE_VERSION deliberately did not move with it: it versions the hello payload, the payload’s six fields are unchanged (byte-pinned in controlCore.test.ts), and a value carried inside the frame cannot describe a break in the frame itself. From this epoch forward the frozen contract holds again, unchanged.

The wire says only drain(): void; it does not promise that every daemon is drainable. A daemon whose durable state cannot survive a drain implements the verb as an honest refusal — kaval’s onDrain throws — and declares the supervisor capability not-drainable, which makes policy-driven invocation unrepresentable. The refusal must leave the daemon and its owned resources alive.

That refusal is a defect, not a member error, and deliberately so. drain declares no error schema (the frozen fragment’s shape is not a consumer’s to widen), so controlCoreFragment runs the hook under Effect.promise: an undeclared throw stays a defect rather than masquerading as something a supervisor could narrow on and “handle”. The oRPC-era PRECONDITION_FAILED code said the same thing with a string the caller had to compare.

Both members are Effect Schema-typed: hello declares ControlCoreHelloSchema as its output, drain declares nothing at all. commit / buildId are Schema.optionalKey — the key is present with a string or absent, never an explicit undefined round-tripped through null.

ExportRole
controlCoreFragment(opts)server deps for frozen hello + awaited drain; a not-drainable daemon’s callback throws, and the throw stays a defect without exiting
controlCoreSurface · controlCoreProcedureSpecstandalone fragment surface (procedures: { core: … }), plus the spec seam for a daemon retaining extra frozen members
ControlCoreFragmentthe ImplementSurfaceDeps type of that surface — what controlCoreFragment satisfies
ControlCoreHelloSchema · ControlCoreHello · CONTROL_CORE_VERSIONimmutable wire vocabulary
// Serve these deps beside the versioned application surface. `hello` remains
// readable even when that application contract is skewed; `drain` waits for
// the daemon's own persistence/shutdown hook. `implementSurfaces` mints ONE
// flat group covering both siblings — each member at
// `surface/<key>/<member>/<verb>`, so the two cannot collide (not even on the
// three framework-reserved `system/*` members every surface carries).
const control = controlCoreFragment({
  stateRoot: home.dir,
  surfaceVersion: "1.0",
  startedAt: Date.now(),
  commit: readIdentity.navigableCommit,
  buildId: readIdentity.staleKey,
  onDrain: () => controller.abort(),
});
const { group, handlers } = implementSurfaces(
  { app: surface, control: controlCoreSurface },
  {},
  { app: deps, control },
);

The upgrade-window test kit

@kolu/surface-daemon/upgrade-window.testlib carries the generic mixed-version fixture, registry matchers/sweep/message, watchdog factory, assertRecipeWired, the bidirectional previous-release harness, process reaper, socket wait, and pinPreviousShapeRecovery. Its module graph has no workspace or external runtime dependency: daemon filenames, state planting, and the spawn guard are required injected hooks. Matchers take the consumer registry as a parameter; no global registry exists in the framework. A versionField records the discriminator but does not excuse coverage—the artifact must name a disposition test that plants version+1 and observes a typed state. The fixture returns independent discriminated process, listener, and state arms, so a pid cannot exist without its child and a config path cannot exist without its state root.

The on-disk home

daemonHome({ app, placement, instance?, socketFile? }) is the one call that decides where a daemon’s files live. It creates the directory 0700, verifies it is owner-only (throws on a non-private dir), and returns the well-known paths with gate and socket side by side — so those two can never drift apart. The returned artifacts array is a pair of SharedArtifact registry entries for the gate and socket, so a mixed-version inventory (UW2) is fed by construction.

This is a spine primitive, not a helper. daemonMain({ home }) and the supervisor’s createEndpoint({ home, connect: (socketPath) => … }) accept the home and derive gate/socket (and, by default, the self-reap anchor) from it — loose path strings do not cross consumer → framework. Overrides (CLI --socket) are absorbed into home construction (socketOverride), never sprinkled at call sites. connect receives the socket path from the framework.

placementDirectory
"state"~/.local/state/<app>/ ($HOME only — ignores $XDG_STATE_HOME) — durable across reboots
"runtime"$XDG_RUNTIME_DIR/<app>/, falling back to /tmp/<app>-$UID/ — boot-wiped

Optional instance (padi’s state-root digest, or a legacy port string) is the only spelling for a decorated home: dir is <app>-<instance>/, gate/socket basenames keep the bare app stem (padi-<digest>/padi.pid, never padi-<digest>.pid). Do not stuff a pre-joined name into app. Optional socketFile overrides the socket basename (kaval’s historical pty-host.sock). Optional runtimeRoot (string | null, pure resolve only) lets discovery evaluate under another drawer without mutating process.env (null forces /tmp). kaval and padi both ride this API; golden tests pin every derived path byte-identical to the pre-migration formula.

A daemon that must outlive user sessions (anything supervised over ssh) takes "state": logind deletes the runtime dir with the user’s last session, leaving an orphan process whose socket and gate have vanished.

// Durable ⇒ state dir (never /run): a daemon supervised over ssh must outlive
// the session that spawned it; logind deletes $XDG_RUNTIME_DIR with the last
// session. Gate and socket live side by side under the home.
const home = daemonHome({ app: "fleet-top", placement: "state" });
ExportRole
daemonHome({ app, placement, instance?, socketFile?, socketOverride? })create + verify the home; returns dir · gatePath · socketPath · file(name) · artifacts
resolveDaemonHome(…)pure path algebra (no mkdir, no env mutation); path helpers, discovery, and the spine consume this
DaemonHomePathsthe face the spine speaks (dir · gatePath · socketPath)
DaemonHome · ResolvedDaemonHome · DaemonHomeOptions · ResolveDaemonHomeOptions · DaemonHomePlacement · DaemonHomeRuntimeRootreturn types and options
isPrivateOwnedDir(dir)owner-only privacy predicate (shared with the pid gate)
SharedArtifactone shared on-disk file both generations touch (inventory entry shape)

The lifecycle

daemonMain({ home, … }) is the gate → serve → teardown skeleton. Gate and socket come from home; the self-reap anchor defaults to home.dir (override when the on-disk identity is not the rendezvous home — e.g. padi’s state-root). It claims the single-instance gate, serves the surface’s { group, handlers } over the unix socket, waits for the lifetime to end — or for the anchor to be proven gone — then closes the socket, releases the gate, and returns a DaemonExit — it never calls process.exit, so the whole lifecycle is drivable in-process from a test.

daemonProcessMain({ name, run }) is the bin half of that partition: for a process that exists to be the daemon, it runs run() (daemonMain or a wrapper around it) to completion and then owns the process exit — the code classified from the DaemonExit (already-running/shutdown0, serve-failed1), a rejection narrated as name: message and exited 1 (the crash arm is swallow-proof: a failing stderr write cannot block the exit). Without it, any live resource or timer the bin never tied to the daemon’s lifetime keeps the event loop alive after the daemon’s tenure — what holds it alive, per its lifetime policy — has ended: an invisible lingering daemon. Call it as the bin’s last statement; it returns synchronously and owns the rest of the process’s life.

When the process is boundToPid (KOLU_DAEMON_BIND_PID set), daemonProcessMain also arms a sibling that does not share this event loop. The in-process boundToPid poll and the SIGTERM handler both live on the daemon’s loop; a wedged daemon (fd exhaustion, SIGSTOP) can neither poll nor honour the signal — the field that made this a requirement (juspay/kolu#2178): TERM left every orphan up, KILL reaped them. The sibling polls the bind pid on the same cadence, waits a grace so the clean pid-gone path can finish release, then SIGKILLs if the process is still up. It does not SIGTERM: that handler is already gone on the clean path, and TERM does not help a wedged loop. It is armed only here, never from in-process daemonMain tests (those share vitest’s pid; a SIGKILL would take the runner down), and is disarmed on the clean-exit and crash arms.

daemonProcessMain({
  name: "fleet-top", // crash-arm narration prefix
  run: () =>
    daemonMain({
      // gate, socket, anchor — all derived from home inside the spine
      home,
      processIdentity, // injected (pid, startUnixUs) for this process
      readProcessIdentity, // injected OS fact reader; the spine only compares
      group, // the flat RPC group the runtime serves…
      handlers, // …and its tag-keyed handlers: one pair, no router
      lifetime: { kind: "forever" }, // or { kind: "idleTimeout", ms, isIdle }
      log: stderrLogger(),
      signal: controller.signal,
      onReady: ({ socketPath, pid }) =>
        process.stderr.write(`listening on ${socketPath} (pid ${pid})\n`),
    }),
});
ExportRole
daemonMain({ home, processIdentity, readProcessIdentity, … })the gate → serve → teardown skeleton; gate/socket/default-anchor from home; identity is injected (never defaulted); resolves a DaemonExit
daemonProcessMain({ name, run })the bin half: run the daemon to completion, then own the process exit (code + crash arm). A boundToPid process also gets a sibling that waits a grace then SIGKILLs it if the bind pid dies while this loop cannot
claimPidGate(gatePath, socketPath, self, readProcessIdentity)the named full claim: acquirePidGate then confirmHeldGate when held — composition roots use this, not the two steps by convention
acquirePidGate(gatePath, self, readProcessIdentity)atomic single-instance claim via link(2); writes ${pid}\t${startUnixUs}\n; returns { kind: "acquired", release }, { kind: "held", pid }, or { kind: "dir-not-private", dir }
ProcessIdentity · ReadProcessIdentitycanonical start-qualified identity ({ pid, startUnixUs }) and the sync injected OS fact reader for the daemon claim path — the spine never imports osfacts. The supervisor’s awaitable inject is ReadProcessIdentityAsync on @kolu/surface-daemon-supervisor (beside EndpointSpec), an Effect — the sync one stays here because a synchronous gate claim must not reorder against the boot side effects it guards
readGateIdentity · gatePid · gateIdentity · liveHolderFromRecord · liveHolderPid · throwIfGateUnreadable · identitiesMatch · startTimesMatch · START_TIME_TOLERANCE_USpid-first tolerant reader: pid via legacy parseInt (leading digits); start time only from exact pid\tstart…; ±2 s match; writers emit ${pid}\t${startUnixUs}\n. Three-way law on the read outcome (exhaustive): absent → free; malformed → reap; unreadable (EACCES/EIO) → throw — never collapse into free/stale. liveHolderFromRecord takes an already-read observation; liveHolderPid reads once then delegates
confirmHeldGate(held, …)one-field (legacy) socket fence only — one gate observation drives both liveness and generation; dead socket reclaims; absent / indeterminate wait; two-field match skips (identity is truth)
isHolderLive(pid)kill(0) liveness probe for one-field holders and cheap checks
socketServeState(socketPath) · SOCKET_SERVE_PROBE_MSfour-way socket probe (serving · dead · absent · indeterminate) used by the one-field reclaim fence. Timeout is a latency ceiling only — reports indeterminate, never authorizes reclaim. Non-ENOENT lstat failures throw (fail loud)
stderrLogger() · Logger · DaemonExitthe structural logging contract and the exit record daemonMain resolves (its exit-code classification lives inside daemonProcessMain)
daemonLifetimeFromEnv(fallback) · DAEMON_BIND_PID_ENVresolve the lifetime from KOLU_DAEMON_BIND_PID — a valid pid → boundToPid, unset → fallback, any present-but-invalid value (incl. "") throws
lifetimeInfo(lifetime) · DaemonLifetimeInfothe serializable projection of a DaemonLifetime — the three kinds with the non-wire members dropped (idleTimeout’s isIdle, boundToPid’s pollMs); what a daemon publishes about itself for a UI to read
anchorGone(path) · DaemonShutdownReasonthe ENOENT-only “proven absent” probe both ends of the anchor invariant share, and the full shutdown-reason union (signal · abort · idle · pid-gone · anchor-gone · runtime-fault)
armRuntimeFaultExit({ done, log, subject, lastRites? }) · DAEMON_RUNTIME_FAULT_MARKERobserve a served runtime’s done and hand back the AbortSignal to pass as DaemonSpec.faultSignal. A rejection is FATAL: it logs the WHOLE error (marked, stack intact), runs the daemon’s last rites (padi captures its final session), and ends the tenure as reason: "runtime-fault" — a non-zero exit through the ordinary teardown, so the socket closes and the gate releases before the process ends. Last rites that throw are logged and the exit still happens
DaemonSpec.faultSignalthe owned-fault stop arm, separate from signal because they mean opposite things: signal is a graceful stop (exit 0), faultSignal is structural wiring death (exit non-zero). Omitting it leaves a runtime fault nowhere to go — the zombie class (process alive, gate held, socket answering, runtime dead)

lifetime is the policy that distinguishes tenants — three honest constructors, not forever plus flags:

lifetimeShuts down whenTenant
{ kind: "forever" }a signal or external abortkaval / padi in production
{ kind: "idleTimeout", ms, isIdle }ms of continuous idlenessodu serve
{ kind: "boundToPid", pid, pollMs? }the watched pid is gone (in-process poll kill(pid, 0)). daemonProcessMain additionally arms a sibling that SIGKILLs a wedged loop after a grace; constructing this lifetime object is not what arms the sibling — the env/daemonProcessMain path isa test/smoke-spawned kaval/padi — dies with its run

Every lifetime is additionally subject to the anchor self-reap — the table lists only each lifetime’s own trigger.

pollMs? is a test-only seam (the liveness-poll interval); production omits it and uses the fixed two-second PID_WATCH_POLL_MS. The pid must be a single-process pid (a positive integer in pid_t range) — an invalid value throws at consumption rather than being reclassified into a clean shutdown.

lifetimeInfo(lifetime) projects a live DaemonLifetime to a serializable DaemonLifetimeInfo ({ kind: "forever" } | { kind: "idleTimeout"; ms } | { kind: "boundToPid"; pid }) — dropping the isIdle closure and the test-only pollMs — so a daemon can publish which lifetime it is running under. kaval carries it on its system.version handshake and padi on its identity cell; kolu’s Kaval and Padi info dialogs each render it as a lifetime row (forever in production; bound to run pid N under a test/smoke run).

daemonLifetimeFromEnv(fallback) reads DAEMON_BIND_PID_ENV (KOLU_DAEMON_BIND_PID): a valid positive-integer pid value selects boundToPid (no liveness check — an already-dead pid still selects it, then exits at once), true absence (the var is UNSET) selects fallback (there is deliberately no way to weaken a production daemon — only a harness/smoke run opts a spawned daemon into dying with it). A set-but-malformed value — the empty string (a present-but-invalid broken expansion, distinct from unset), non-canonical decimal, fractional, or out of pid_t range — throws (fail-fast), never silently degrades to fallback. A boundToPid shutdown resolves { kind: "shutdown", reason: "pid-gone" }, exit code 0. Pid reuse inside the poll window is a documented residual, not engineered around.

The owned-runtime-fault exit

Every shutdown reason is a success (exit 0) except one: runtime-fault, raised by DaemonSpec.faultSignal when the served surface runtime’s done rejects. It is an orderly teardown — listener closed, gate released, last rites run — that scores exit 1, because the supervisor’s only channel for “that was a crash, not a stop” is the exit code. Wire it with armRuntimeFaultExit; both kaval and padi do.

Why fatal rather than “log and keep serving”: after the @kolu/surface poll ruling (a poll read’s failure is cell-local at every tick, T+0 included) a done rejection can only be structural wiring death, which does not heal by waiting. The recorded failure is juspay/kolu#2101 — a padi that logged one probe timeout and kept its gate and socket while its runtime was dead, needing an operator instead of the ~2s respawn-and-restore a crash would have produced.

The anchor self-reap

anchor defaults to () => home.dir — gate, socket, and anchor all ride the home. Override only when the on-disk identity is not the rendezvous home (kaval’s anchor is its padi’s state-root, learned from a manifest; padi’s is its state-root). The thunk is armed under every lifetime (an independent trigger, not a lifetime arm): once the directory has been proven gone for two consecutive polls (5s apart in production; anchorPollMs? is the test seam), the daemon reaps itself through the normal teardown — socket unlinked, gate released, { kind: "shutdown", reason: "anchor-gone" }, exit code 0 — instead of lingering as a zombie holding its sockets and RSS forever after a git worktree remove.

Proof is ENOENT-only (anchorGone(path)): any other lstat failure — EACCES, EIO, ENOTDIR — means “I could not read whether it exists”, which never counts toward a reap. The thunk is re-evaluated every tick, so an anchor learned after boot self-corrects (kaval reads the state-root manifest its padi writes around kaval’s own boot). () => undefined is the honest spelling for a daemon with genuinely no on-disk identity (never reaped). Omitting anchor is the normal case — it defaults to the home.

Serving the surface

The listener is serveOverUnixSocket({ socketPath, group, handlers, log }) from @kolu/surface/unix-socket — a @kolu/surface subpath, not this package. It never crashes: every failure mode resolves to a no-op listener with a machine-readable outcome. daemonMain passes its own log through, so the listener’s lifetime (bound / post-listen fault / closed) lands in the daemon’s journal beside the daemon’s own boot and teardown narration.

outcome.kindMeaning
listeningbound and serving
dir-not-private · not-a-socket · bind-failedcould not bind
already-served · probe-failedanother holder is (or may be) live

Fronting over stdio

frontDaemonOverStdio(opts) is the front half: it adopts-or-spawns the gate-held daemon and raw-byte-relays this process’s stdio onto its socket, so a remote (ssh) session survives the link — the durable counterpart to serveOverStdio. It relays with node:net only, no surface import, keeping the daemon closure contract-blind. That blindness is legal only because the stdio leg and the socket leg are byte-identical ndjson, which packages/surface/src/links/byteSplice.test.ts proves on captured raw bytes in both directions — including that nothing on the wire is non-JSON binary.

return frontDaemonOverStdio({
  socketPath: home.socketPath,
  spawnDaemon: () => reExecAsDetachedDaemon({ stripArgs: ["--stdio"] }),
  log: (msg) => process.stderr.write(`--stdio: ${msg}\n`),
});

reExecAsDetachedDaemon({ stripArgs, stderrLog? }) re-execs this binary minus the front flag as a detached, unref’d process, so it survives the SIGHUP that drops the link.

Baked identity

readBakedIdentity(prefix) reads a Nix-baked env pair (<PREFIX>_BUILD_ID / <PREFIX>_COMMIT_HASH) into { staleKey, navigableCommit }, and returns empty strings when both variables are absent off-nix rather than inventing an identity. The pair is joint: both variables must contain non-empty values or both must be absent; a half-baked or explicitly empty baked identity throws at daemon boot because a Nix build id without its knowable commit (or the reverse) is contradictory. See How to bake an identity.

Nix recipes

The package’s nix/ directory is public surface too — the build-time half of what readBakedIdentity reads back. Each file is a function of { lib }: import "${kolu}/packages/surface-daemon/nix/<file>.nix" { inherit lib; }. All of it is pure evaluation (no import-from-derivation), so nix flake check forces every output without realising a build mid-eval. The how-to is How to bake an identity.

mkDaemonIdentitynix/daemon-identity.nix

mkDaemonIdentity { name, prefix, root, behavioralFileset, pinnedSources ? { }, commitHash, override ? null }{ buildId, bakeArgs }.

ArgRole
namethe daemon’s name, used in this recipe’s error messages
prefixits identity-env namespace — "KAVAL", "PADI" — the <PREFIX> readBakedIdentity reads
rootthe lib.fileset.toSource root; must be a common ancestor of behavioralFileset
behavioralFilesetWHAT counts as this daemon’s behaviour — the consumer’s own policy decision
pinnedSourcesbehavioural members that are pins, not files: name → store path. Every value must be a string under builtins.storeDir; the assert names the offending attrs
commitHashthe navigable git ref this build was made from; "" throws
overrideTEST-ONLY: force buildId instead of hashing (build-skew arms)
ReturnValue
buildId64-char sha256 of the fileset.toSource store path, plus one sorted <name>=<store path> line per pinnedSources entry. Platform-independent by construction — it hashes source, never the runtime engine, so a cross-platform comparison is not a false mismatch
bakeArgsthe --set <PREFIX>_BUILD_ID … --set <PREFIX>_COMMIT_HASH … string to splice into a makeWrapper invocation — the pair is emitted together, so the joint invariant cannot be half-baked

With pinnedSources = { } the id is hashString "sha256" "${src}"byte-identical to what this recipe hashed before the pinned arm existed, so a workspace-only consumer’s live daemon ids do not move.

mkWorkspaceClosurenix/workspace-closure.nix

mkWorkspaceClosure { members, pinned ? [ ], mustCover ? [ ] } derives what the identity should hash from the package.json dependencies graph, instead of a hand-kept file list that can silently drift. The file also exports defaultIsHashedSource at its top level.

ArgRole
memberspackage name → package dir. A local member’s dir is a path literal; a pinned member’s is a string already under builtins.storeDir
pinnedthe member NAMES that are pins — declared, never inferred from how the value is spelled
mustCovername prefixes that must never resolve outside members ([ "@kolu/" ]) — the by-name tripwire a pinned consumer needs, having no workspace: protocol to give a stale map away
ReturnValue
membersthe checked map (same keys), which the two functions below read
depClosure { entries, stop ? [ ] }the member names reachable from entries over dependencies edges whose target is a member, minus each stop entry and everything reachable only through it
identityInputs { entries, stableLeaves ? [ ], isHashedSource ? defaultIsHashedSource }the same closure rendered as mkDaemonIdentity’s two inputs: behavioralFileset (each local member’s filtered src plus its package.json) and pinnedSources (the pinned members’ store paths). isHashedSource defaults to real .ts/.tsx, dropping .test.ts · .test.tsx · .test-d.ts · .testlib.ts

devDependencies are deliberately not followed — they never ship behaviour. The residual failure direction is over-inclusion (one extra id flip), never a silent escape. Every mistake is a loud eval failure:

FailureMessage names
a pinned name that is not a members keythe stale names
a member whose value shape disagrees with its declared kind — a path where a pin string belongs, or a string where a local path belongsthe member, and which declaration to fix
a members key that disagrees with the package’s own namethe key, the manifest name, the dir
a workspace: (or mustCover-matching) edge whose target is not a memberthe importer, the target, and the prefix that matched
a stop / stableLeaves entry that is not in the closurethe stale entries

mkProvenAgentSourcenix/agent-source.nix

The third recipe in the same directory, unrelated to identity: it assembles a remote-agent source tree and proves an agent evaluates from it before the path reaches any wrapper. See the file’s own header for the prove/expose rule.