@kolu/surface-daemon reference
The daemon binary half — the code that runs inside the long-lived process. It
carries no @kolu/* app dependency; the client half lives separately in
@kolu/surface-daemon-supervisor. Production
imports use the package root; mixed-version suites use the dedicated
./upgrade-window.testlib subpath; pid-gate test seams (connect-probe
override) use ./pidGate.testlib — never re-exported from the production root.
The frozen control core
controlCoreFragment({ stateRoot, surfaceVersion, startedAt, commit, buildId, onDrain }) implements the immutable core.hello identity and core.drain verb.
It is served beside the versioned application surface — composed as the sibling
control, so the two members sit at the wire tags surface/control/core/hello
and surface/control/core/drain and cannot collide with anything the application
surface mints. A supervisor can therefore identify and drain a resident daemon
before knowing whether that versioned surface is compatible. The hello fields are
fixed:
stateRoot · surfaceVersion · controlCoreVersion · startedAt · commit · buildId.
CONTROL_CORE_VERSION is frozen at "1.0"; it is not an application contract
version. Build id and commit are one observation: the shared reader accepts both
absent, both empty (off-Nix), or both non-empty, and rejects every partial pair
before convergence policy runs.
The fragment never versions — within a protocol epoch. The Effect-4
migration moved this wire’s framing (the oRPC base64+newline peer codec →
Effect RPC ndjson). That is a declared flag day, not a version negotiation:
negotiation happens inside the protocol being replaced, so a daemon from the
previous epoch cannot be asked anything at all — its first frame is undecodable,
and the supervisor observes that as unspeakable-protocol, never as an
incompatible version. CONTROL_CORE_VERSION deliberately did not move with
it: it versions the hello payload, the payload’s six fields are unchanged
(byte-pinned in controlCore.test.ts), and a value carried inside the frame
cannot describe a break in the frame itself. From this epoch forward the frozen
contract holds again, unchanged.
The wire says only drain(): void; it does not promise that every daemon is
drainable. A daemon whose durable state cannot survive a drain implements the
verb as an honest refusal — kaval’s onDrain throws — and declares the
supervisor capability not-drainable, which makes policy-driven invocation
unrepresentable. The refusal must leave the daemon and its owned resources alive.
That refusal is a defect, not a member error, and deliberately so. drain
declares no error schema (the frozen fragment’s shape is not a consumer’s to
widen), so controlCoreFragment runs the hook under Effect.promise: an
undeclared throw stays a defect rather than masquerading as something a supervisor
could narrow on and “handle”. The oRPC-era PRECONDITION_FAILED code said the
same thing with a string the caller had to compare.
Both members are Effect Schema-typed: hello declares
ControlCoreHelloSchema as its output, drain declares nothing at all.
commit / buildId are Schema.optionalKey — the key is present with a string
or absent, never an explicit undefined round-tripped through null.
| Export | Role |
|---|---|
controlCoreFragment(opts) | server deps for frozen hello + awaited drain; a not-drainable daemon’s callback throws, and the throw stays a defect without exiting |
controlCoreSurface · controlCoreProcedureSpec | standalone fragment surface (procedures: { core: … }), plus the spec seam for a daemon retaining extra frozen members |
ControlCoreFragment | the ImplementSurfaceDeps type of that surface — what controlCoreFragment satisfies |
ControlCoreHelloSchema · ControlCoreHello · CONTROL_CORE_VERSION | immutable wire vocabulary |
// Serve these deps beside the versioned application surface. `hello` remains
// readable even when that application contract is skewed; `drain` waits for
// the daemon's own persistence/shutdown hook. `implementSurfaces` mints ONE
// flat group covering both siblings — each member at
// `surface/<key>/<member>/<verb>`, so the two cannot collide (not even on the
// three framework-reserved `system/*` members every surface carries).
const control = controlCoreFragment({
stateRoot: home.dir,
surfaceVersion: "1.0",
startedAt: Date.now(),
commit: readIdentity.navigableCommit,
buildId: readIdentity.staleKey,
onDrain: () => controller.abort(),
});
const { group, handlers } = implementSurfaces(
{ app: surface, control: controlCoreSurface },
{},
{ app: deps, control },
);
The upgrade-window test kit
@kolu/surface-daemon/upgrade-window.testlib carries the generic mixed-version
fixture, registry matchers/sweep/message, watchdog factory,
assertRecipeWired, the bidirectional previous-release harness, process reaper,
socket wait, and pinPreviousShapeRecovery. Its module graph has no workspace
or external runtime dependency: daemon filenames, state planting, and the spawn
guard are required injected hooks. Matchers take the consumer registry as a
parameter; no global registry exists in the framework. A versionField records
the discriminator but does not excuse coverage—the artifact must name a
disposition test that plants version+1 and observes a typed state.
The fixture returns independent discriminated process, listener, and
state arms, so a pid cannot exist without its child and a config path cannot
exist without its state root.
The on-disk home
daemonHome({ app, placement, instance?, socketFile? }) is the one call that
decides where a daemon’s files live. It creates the directory 0700, verifies
it is owner-only (throws on a non-private dir), and returns the well-known paths
with gate and socket side by side — so those two can never drift apart. The
returned artifacts array is a pair of SharedArtifact registry entries for
the gate and socket, so a mixed-version inventory (UW2) is fed by construction.
This is a spine primitive, not a helper. daemonMain({ home }) and the
supervisor’s createEndpoint({ home, connect: (socketPath) => … }) accept the
home and derive gate/socket (and, by default, the self-reap anchor) from it —
loose path strings do not cross consumer → framework. Overrides (CLI --socket)
are absorbed into home construction (socketOverride), never sprinkled at call
sites. connect receives the socket path from the framework.
placement | Directory |
|---|---|
"state" | ~/.local/state/<app>/ ($HOME only — ignores $XDG_STATE_HOME) — durable across reboots |
"runtime" | $XDG_RUNTIME_DIR/<app>/, falling back to /tmp/<app>-$UID/ — boot-wiped |
Optional instance (padi’s state-root digest, or a legacy port string) is the
only spelling for a decorated home: dir is <app>-<instance>/, gate/socket
basenames keep the bare app stem (padi-<digest>/padi.pid, never
padi-<digest>.pid). Do not stuff a pre-joined name into app. Optional
socketFile overrides the socket basename (kaval’s historical pty-host.sock).
Optional runtimeRoot (string | null, pure resolve only) lets discovery
evaluate under another drawer without mutating process.env (null forces
/tmp). kaval and padi both ride this API; golden tests pin every derived path
byte-identical to the pre-migration formula.
A daemon that must outlive user sessions (anything supervised over ssh) takes
"state": logind deletes the runtime dir with the user’s last session, leaving
an orphan process whose socket and gate have vanished.
// Durable ⇒ state dir (never /run): a daemon supervised over ssh must outlive
// the session that spawned it; logind deletes $XDG_RUNTIME_DIR with the last
// session. Gate and socket live side by side under the home.
const home = daemonHome({ app: "fleet-top", placement: "state" });
| Export | Role |
|---|---|
daemonHome({ app, placement, instance?, socketFile?, socketOverride? }) | create + verify the home; returns dir · gatePath · socketPath · file(name) · artifacts |
resolveDaemonHome(…) | pure path algebra (no mkdir, no env mutation); path helpers, discovery, and the spine consume this |
DaemonHomePaths | the face the spine speaks (dir · gatePath · socketPath) |
DaemonHome · ResolvedDaemonHome · DaemonHomeOptions · ResolveDaemonHomeOptions · DaemonHomePlacement · DaemonHomeRuntimeRoot | return types and options |
isPrivateOwnedDir(dir) | owner-only privacy predicate (shared with the pid gate) |
SharedArtifact | one shared on-disk file both generations touch (inventory entry shape) |
The lifecycle
daemonMain({ home, … }) is the gate → serve → teardown skeleton. Gate and
socket come from home; the self-reap anchor defaults
to home.dir (override when the on-disk identity is not the rendezvous home —
e.g. padi’s state-root). It claims the single-instance gate, serves the
surface’s { group, handlers } over the unix socket, waits for the lifetime to end — or for the anchor to be
proven gone — then closes the socket, releases the gate, and returns a
DaemonExit — it never calls process.exit, so the whole lifecycle is drivable
in-process from a test.
daemonProcessMain({ name, run }) is the bin half of that partition: for a
process that exists to be the daemon, it runs run() (daemonMain or a
wrapper around it) to completion and then owns the process exit — the code
classified from the DaemonExit (already-running/shutdown → 0,
serve-failed → 1), a rejection narrated as name: message and exited 1
(the crash arm is swallow-proof: a failing stderr write cannot block the exit).
Without it, any live resource or timer the bin never tied to the daemon’s
lifetime keeps the event loop alive after the daemon’s tenure — what holds it
alive, per its lifetime policy — has ended: an invisible lingering daemon.
Call it as the bin’s last statement; it returns synchronously and owns the
rest of the process’s life.
When the process is boundToPid (KOLU_DAEMON_BIND_PID set), daemonProcessMain
also arms a sibling that does not share this event loop. The in-process
boundToPid poll and the SIGTERM handler both live on the daemon’s loop; a
wedged daemon (fd exhaustion, SIGSTOP) can neither poll nor honour the signal —
the field that made this a requirement (juspay/kolu#2178): TERM left every orphan
up, KILL reaped them. The sibling polls the bind pid on the same cadence, waits
a grace so the clean pid-gone path can finish release, then SIGKILLs if the
process is still up. It does not SIGTERM: that handler is already gone on the
clean path, and TERM does not help a wedged loop. It is armed only here, never
from in-process daemonMain tests (those share vitest’s pid; a SIGKILL would
take the runner down), and is disarmed on the clean-exit and crash arms.
daemonProcessMain({
name: "fleet-top", // crash-arm narration prefix
run: () =>
daemonMain({
// gate, socket, anchor — all derived from home inside the spine
home,
processIdentity, // injected (pid, startUnixUs) for this process
readProcessIdentity, // injected OS fact reader; the spine only compares
group, // the flat RPC group the runtime serves…
handlers, // …and its tag-keyed handlers: one pair, no router
lifetime: { kind: "forever" }, // or { kind: "idleTimeout", ms, isIdle }
log: stderrLogger(),
signal: controller.signal,
onReady: ({ socketPath, pid }) =>
process.stderr.write(`listening on ${socketPath} (pid ${pid})\n`),
}),
});
| Export | Role |
|---|---|
daemonMain({ home, processIdentity, readProcessIdentity, … }) | the gate → serve → teardown skeleton; gate/socket/default-anchor from home; identity is injected (never defaulted); resolves a DaemonExit |
daemonProcessMain({ name, run }) | the bin half: run the daemon to completion, then own the process exit (code + crash arm). A boundToPid process also gets a sibling that waits a grace then SIGKILLs it if the bind pid dies while this loop cannot |
claimPidGate(gatePath, socketPath, self, readProcessIdentity) | the named full claim: acquirePidGate then confirmHeldGate when held — composition roots use this, not the two steps by convention |
acquirePidGate(gatePath, self, readProcessIdentity) | atomic single-instance claim via link(2); writes ${pid}\t${startUnixUs}\n; returns { kind: "acquired", release }, { kind: "held", pid }, or { kind: "dir-not-private", dir } |
ProcessIdentity · ReadProcessIdentity | canonical start-qualified identity ({ pid, startUnixUs }) and the sync injected OS fact reader for the daemon claim path — the spine never imports osfacts. The supervisor’s awaitable inject is ReadProcessIdentityAsync on @kolu/surface-daemon-supervisor (beside EndpointSpec), an Effect — the sync one stays here because a synchronous gate claim must not reorder against the boot side effects it guards |
readGateIdentity · gatePid · gateIdentity · liveHolderFromRecord · liveHolderPid · throwIfGateUnreadable · identitiesMatch · startTimesMatch · START_TIME_TOLERANCE_US | pid-first tolerant reader: pid via legacy parseInt (leading digits); start time only from exact pid\tstart…; ±2 s match; writers emit ${pid}\t${startUnixUs}\n. Three-way law on the read outcome (exhaustive): absent → free; malformed → reap; unreadable (EACCES/EIO) → throw — never collapse into free/stale. liveHolderFromRecord takes an already-read observation; liveHolderPid reads once then delegates |
confirmHeldGate(held, …) | one-field (legacy) socket fence only — one gate observation drives both liveness and generation; dead socket reclaims; absent / indeterminate wait; two-field match skips (identity is truth) |
isHolderLive(pid) | kill(0) liveness probe for one-field holders and cheap checks |
socketServeState(socketPath) · SOCKET_SERVE_PROBE_MS | four-way socket probe (serving · dead · absent · indeterminate) used by the one-field reclaim fence. Timeout is a latency ceiling only — reports indeterminate, never authorizes reclaim. Non-ENOENT lstat failures throw (fail loud) |
stderrLogger() · Logger · DaemonExit | the structural logging contract and the exit record daemonMain resolves (its exit-code classification lives inside daemonProcessMain) |
daemonLifetimeFromEnv(fallback) · DAEMON_BIND_PID_ENV | resolve the lifetime from KOLU_DAEMON_BIND_PID — a valid pid → boundToPid, unset → fallback, any present-but-invalid value (incl. "") throws |
lifetimeInfo(lifetime) · DaemonLifetimeInfo | the serializable projection of a DaemonLifetime — the three kinds with the non-wire members dropped (idleTimeout’s isIdle, boundToPid’s pollMs); what a daemon publishes about itself for a UI to read |
anchorGone(path) · DaemonShutdownReason | the ENOENT-only “proven absent” probe both ends of the anchor invariant share, and the full shutdown-reason union (signal · abort · idle · pid-gone · anchor-gone · runtime-fault) |
armRuntimeFaultExit({ done, log, subject, lastRites? }) · DAEMON_RUNTIME_FAULT_MARKER | observe a served runtime’s done and hand back the AbortSignal to pass as DaemonSpec.faultSignal. A rejection is FATAL: it logs the WHOLE error (marked, stack intact), runs the daemon’s last rites (padi captures its final session), and ends the tenure as reason: "runtime-fault" — a non-zero exit through the ordinary teardown, so the socket closes and the gate releases before the process ends. Last rites that throw are logged and the exit still happens |
DaemonSpec.faultSignal | the owned-fault stop arm, separate from signal because they mean opposite things: signal is a graceful stop (exit 0), faultSignal is structural wiring death (exit non-zero). Omitting it leaves a runtime fault nowhere to go — the zombie class (process alive, gate held, socket answering, runtime dead) |
lifetime is the policy that distinguishes tenants — three honest constructors,
not forever plus flags:
lifetime | Shuts down when | Tenant |
|---|---|---|
{ kind: "forever" } | a signal or external abort | kaval / padi in production |
{ kind: "idleTimeout", ms, isIdle } | ms of continuous idleness | odu serve |
{ kind: "boundToPid", pid, pollMs? } | the watched pid is gone (in-process poll kill(pid, 0)). daemonProcessMain additionally arms a sibling that SIGKILLs a wedged loop after a grace; constructing this lifetime object is not what arms the sibling — the env/daemonProcessMain path is | a test/smoke-spawned kaval/padi — dies with its run |
Every lifetime is additionally subject to the anchor self-reap — the table lists only each lifetime’s own trigger.
pollMs? is a test-only seam (the liveness-poll interval); production omits it
and uses the fixed two-second PID_WATCH_POLL_MS. The pid must be a single-process
pid (a positive integer in pid_t range) — an invalid value throws at consumption
rather than being reclassified into a clean shutdown.
lifetimeInfo(lifetime) projects a live DaemonLifetime to a serializable
DaemonLifetimeInfo ({ kind: "forever" } | { kind: "idleTimeout"; ms } | { kind: "boundToPid"; pid }) — dropping the isIdle closure and the test-only pollMs — so
a daemon can publish which lifetime it is running under. kaval carries it on its
system.version handshake and padi on its identity cell; kolu’s Kaval and Padi
info dialogs each render it as a lifetime row (forever in production; bound to run pid N under a test/smoke run).
daemonLifetimeFromEnv(fallback) reads DAEMON_BIND_PID_ENV
(KOLU_DAEMON_BIND_PID): a valid positive-integer pid value selects boundToPid
(no liveness check — an already-dead pid still selects it, then exits at once),
true absence (the var is UNSET) selects fallback (there is deliberately no way
to weaken a production daemon — only a harness/smoke run opts a spawned daemon into
dying with it). A set-but-malformed value — the empty string (a present-but-invalid
broken expansion, distinct from unset), non-canonical decimal, fractional, or out of
pid_t range — throws (fail-fast), never silently degrades to fallback. A
boundToPid shutdown resolves { kind: "shutdown", reason: "pid-gone" }, exit code
0. Pid reuse inside the poll window is a documented residual, not engineered around.
The owned-runtime-fault exit
Every shutdown reason is a success (exit 0) except one: runtime-fault, raised
by DaemonSpec.faultSignal when the served surface runtime’s done rejects. It is
an orderly teardown — listener closed, gate released, last rites run — that scores
exit 1, because the supervisor’s only channel for “that was a crash, not a
stop” is the exit code. Wire it with armRuntimeFaultExit; both kaval and padi do.
Why fatal rather than “log and keep serving”: after the @kolu/surface poll ruling
(a poll read’s failure is cell-local at every tick, T+0 included) a done rejection
can only be structural wiring death, which does not heal by waiting. The recorded
failure is juspay/kolu#2101 — a padi that logged one probe timeout and kept its gate
and socket while its runtime was dead, needing an operator instead of the ~2s
respawn-and-restore a crash would have produced.
The anchor self-reap
anchor defaults to () => home.dir — gate, socket, and anchor all ride the
home. Override only when the on-disk identity is not the rendezvous home
(kaval’s anchor is its padi’s state-root, learned from a manifest; padi’s is
its state-root). The thunk is armed under every lifetime (an independent
trigger, not a lifetime arm): once the directory has been proven gone for
two consecutive polls (5s apart in production; anchorPollMs? is the test
seam), the daemon reaps itself through the normal teardown — socket unlinked,
gate released, { kind: "shutdown", reason: "anchor-gone" }, exit code 0 —
instead of lingering as a zombie holding its sockets and RSS forever after a
git worktree remove.
Proof is ENOENT-only (anchorGone(path)): any other lstat failure — EACCES,
EIO, ENOTDIR — means “I could not read whether it exists”, which never counts
toward a reap. The thunk is re-evaluated every tick, so an anchor learned after
boot self-corrects (kaval reads the state-root manifest its padi writes around
kaval’s own boot). () => undefined is the honest spelling for a daemon with
genuinely no on-disk identity (never reaped). Omitting anchor is the normal
case — it defaults to the home.
Serving the surface
The listener is serveOverUnixSocket({ socketPath, group, handlers, log }) from
@kolu/surface/unix-socket — a @kolu/surface subpath, not this package. It
never crashes: every failure mode resolves to a no-op listener with a
machine-readable outcome. daemonMain passes its own log through, so the
listener’s lifetime (bound / post-listen fault / closed) lands in the daemon’s
journal beside the daemon’s own boot and teardown narration.
outcome.kind | Meaning |
|---|---|
listening | bound and serving |
dir-not-private · not-a-socket · bind-failed | could not bind |
already-served · probe-failed | another holder is (or may be) live |
Fronting over stdio
frontDaemonOverStdio(opts) is the front half: it adopts-or-spawns the
gate-held daemon and raw-byte-relays this process’s stdio onto its socket, so a
remote (ssh) session survives the link — the durable counterpart to
serveOverStdio. It relays with node:net only, no surface import, keeping the
daemon closure contract-blind. That blindness is legal only because the stdio
leg and the socket leg are byte-identical ndjson, which
packages/surface/src/links/byteSplice.test.ts proves on captured raw bytes in
both directions — including that nothing on the wire is non-JSON binary.
return frontDaemonOverStdio({
socketPath: home.socketPath,
spawnDaemon: () => reExecAsDetachedDaemon({ stripArgs: ["--stdio"] }),
log: (msg) => process.stderr.write(`--stdio: ${msg}\n`),
});
reExecAsDetachedDaemon({ stripArgs, stderrLog? }) re-execs this binary minus
the front flag as a detached, unref’d process, so it survives the SIGHUP that
drops the link.
Baked identity
readBakedIdentity(prefix) reads a Nix-baked env pair
(<PREFIX>_BUILD_ID / <PREFIX>_COMMIT_HASH) into { staleKey, navigableCommit },
and returns empty strings when both variables are absent off-nix rather than
inventing an identity. The pair is joint: both variables must contain non-empty
values or both must be absent; a half-baked or explicitly empty baked identity
throws at daemon boot because a Nix build id without its knowable commit (or the
reverse) is contradictory. See
How to bake an identity.
Nix recipes
The package’s nix/ directory is public surface too — the build-time half of
what readBakedIdentity reads back. Each file is a function of { lib }:
import "${kolu}/packages/surface-daemon/nix/<file>.nix" { inherit lib; }. All
of it is pure evaluation (no import-from-derivation), so nix flake check
forces every output without realising a build mid-eval. The how-to is
How to bake an identity.
mkDaemonIdentity — nix/daemon-identity.nix
mkDaemonIdentity { name, prefix, root, behavioralFileset, pinnedSources ? { }, commitHash, override ? null } → { buildId, bakeArgs }.
| Arg | Role |
|---|---|
name | the daemon’s name, used in this recipe’s error messages |
prefix | its identity-env namespace — "KAVAL", "PADI" — the <PREFIX> readBakedIdentity reads |
root | the lib.fileset.toSource root; must be a common ancestor of behavioralFileset |
behavioralFileset | WHAT counts as this daemon’s behaviour — the consumer’s own policy decision |
pinnedSources | behavioural members that are pins, not files: name → store path. Every value must be a string under builtins.storeDir; the assert names the offending attrs |
commitHash | the navigable git ref this build was made from; "" throws |
override | TEST-ONLY: force buildId instead of hashing (build-skew arms) |
| Return | Value |
|---|---|
buildId | 64-char sha256 of the fileset.toSource store path, plus one sorted <name>=<store path> line per pinnedSources entry. Platform-independent by construction — it hashes source, never the runtime engine, so a cross-platform comparison is not a false mismatch |
bakeArgs | the --set <PREFIX>_BUILD_ID … --set <PREFIX>_COMMIT_HASH … string to splice into a makeWrapper invocation — the pair is emitted together, so the joint invariant cannot be half-baked |
With pinnedSources = { } the id is hashString "sha256" "${src}" —
byte-identical to what this recipe hashed before the pinned arm existed, so
a workspace-only consumer’s live daemon ids do not move.
mkWorkspaceClosure — nix/workspace-closure.nix
mkWorkspaceClosure { members, pinned ? [ ], mustCover ? [ ] } derives what the
identity should hash from the package.json dependencies graph, instead of a
hand-kept file list that can silently drift. The file also exports
defaultIsHashedSource at its top level.
| Arg | Role |
|---|---|
members | package name → package dir. A local member’s dir is a path literal; a pinned member’s is a string already under builtins.storeDir |
pinned | the member NAMES that are pins — declared, never inferred from how the value is spelled |
mustCover | name prefixes that must never resolve outside members ([ "@kolu/" ]) — the by-name tripwire a pinned consumer needs, having no workspace: protocol to give a stale map away |
| Return | Value |
|---|---|
members | the checked map (same keys), which the two functions below read |
depClosure { entries, stop ? [ ] } | the member names reachable from entries over dependencies edges whose target is a member, minus each stop entry and everything reachable only through it |
identityInputs { entries, stableLeaves ? [ ], isHashedSource ? defaultIsHashedSource } | the same closure rendered as mkDaemonIdentity’s two inputs: behavioralFileset (each local member’s filtered src plus its package.json) and pinnedSources (the pinned members’ store paths). isHashedSource defaults to real .ts/.tsx, dropping .test.ts · .test.tsx · .test-d.ts · .testlib.ts |
devDependencies are deliberately not followed — they never ship behaviour.
The residual failure direction is over-inclusion (one extra id flip), never a
silent escape. Every mistake is a loud eval failure:
| Failure | Message names |
|---|---|
a pinned name that is not a members key | the stale names |
| a member whose value shape disagrees with its declared kind — a path where a pin string belongs, or a string where a local path belongs | the member, and which declaration to fix |
a members key that disagrees with the package’s own name | the key, the manifest name, the dir |
a workspace: (or mustCover-matching) edge whose target is not a member | the importer, the target, and the prefix that matched |
a stop / stableLeaves entry that is not in the closure | the stale entries |
mkProvenAgentSource — nix/agent-source.nix
The third recipe in the same directory, unrelated to identity: it assembles a remote-agent source tree and proves an agent evaluates from it before the path reaches any wrapper. See the file’s own header for the prove/expose rule.