@kolu/surface-remote reference
Run a typed @kolu/surface agent on a remote machine over ssh, or read one
listening on a unix socket beside you. The package owns provisioning
(ship a .drv, realise it on the target), the long-lived ssh
subprocess (ref-counting, reconnect, backoff), and the observable state of the
link’s own lifecycle.
It has two public subpaths: @kolu/surface-remote and the browser-safe
@kolu/surface-remote/connection.
The session
A session is the reconnect appliance. makeSession is the transport-agnostic loop, and a connector is the transport you plug into it: sshConnector for a remote host, unixSocketConnector for a daemon on this machine. You own each session. There is no shared pool: key your own map and destroy() each session yourself.
| Export | Subpath | What it is |
|---|---|---|
makeSession(opts) | . | The reconnect, backoff, give-up and watchdog loop. |
sshConnector<S>(opts) | . | The ssh Connector: provisions the agent on a host and dials it. |
SshKeepalive / sshKeepalive(intervalS, countMax) | . | The ssh dead-peer policy for a dial. |
sshDialOpts(keepalive) | . | The ssh options to use when you build your own ssh command. |
unixSocketConnector(opts) | . | Dials a daemon listening on a unix socket on this machine. |
dialAgentOnce<S>(opts) | . | A one-shot dial for a CLI. |
SURFACE_AGENT_FLAKE_REF_ENV / readBakedAgentSource() | . | The source flake the Nix wrapper bakes, and its reader. |
resolveBakedAgentDrv(host, packageName, opts) | . | Resolves an agent from the baked source inside your own dial. |
resolveAgentDrv(host, flakeRef, packageName, { signal, onProgress, budget }) | . | Probes a host’s Nix and resolves one agent package. |
AgentDerivation / directAgentDerivation(drvPath, binaryCache) | . | What provisioning ships: a validated derivation. |
AgentBinaryCache / agentBinaryCache({ substituters, trustedPublicKeys }) | . | Where provisioning may prefetch an agent’s binaries from. |
readBakedDeclaration(flakeRef) → Result<BakedDeclaration, AgentBinaryCacheUnbakedError> | . | Reads the binary caches and closure index a source declares. |
ClosureIndex | . | Where a source’s builds are published. |
closureIndexUrl(index, key, system, attr) | . | The URL of one entry in a closure index. |
readPublishedClosure(fetchText, index, key, system, attr) → Promise<ClosureLookup> | . | Looks up one published build. |
IndexedClosure | . | Where one build is published for one host. |
closureRoute(hostNix, entry) → ClosureRoute | . | Which form of a published build a host pulls. |
isDarwinSystem(system) → boolean | . | Whether a Nix system tuple names macOS. |
Connector / Connection / ConnectError | . | The transport seam a session dials through. |
EndpointAbsentError | . | A connector’s verdict that nothing is serving there. |
Admit / AdmitVerdict | . | An optional per-dial supervision hook. |
makeSession(opts)
The reconnect loop. It handles backoff, giving up and the liveness watchdog, and returns a Session<AgentClient, Prov>.
- It owns no transport. You supply
connectOnce, usually a connector from this section. - Its watchdog and timers are described under Lifecycle invariants.
sshConnector<S>(opts)
The ssh Connector. On each dial it resolves the .drv, provisions it, spawns ssh <host> <binary> --stdio, awaits the agent’s readiness banner, and then wires stdio to a typed client.
Options: { surface, host, binary, resolveDrvPath, localEnv, extraArgs?, keepalive? }.
| Option | Meaning |
|---|---|
surface | The Surface<S> the remote agent serves, passed as a value. |
host | Any ssh target. "localhost" short-circuits ssh. |
binary | The executable inside the realised closure. It is also the package name, so the two cannot drift apart. |
resolveDrvPath | Resolves the agent for the target’s arch. See below. |
localEnv | Required. The whole environment of a localhost dial. See below. |
extraArgs? | Arguments appended after --stdio on the agent command. POSIX-quoted for a real remote, verbatim for localhost. Their meaning is the caller’s concern. |
keepalive? | An SshKeepalive. Defaults to DEFAULT_SSH_KEEPALIVE. |
surfaceis a value, not a type parameter. Effect RPC builds the client fromsurface.group, and the face is re-nested fromsurface.spec. Neither can be recovered from a type.resolveDrvPathruns inside the owning dial’s cancellation and progress lifetime. For a direct path, returndirectAgentDerivation(drvPath, binaryCache). Get thebinaryCachefromreadBakedDeclaration(source)’scacheswherever the deployment bakes an agent source, so it comes from your flake’s ownnixConfig. UseagentBinaryCache({…})inline when there is no baked source.resolveAgentDrvreturns the flake-backed arm, which survives local garbage collection.- Inside
resolveDrvPath,const sys = await ctx.resolveSystem()runs the arch probe on this dial’s host, signal, progress sink and keepalive policy. There is no options bag to assemble. - The package reads the exact, minimal source flake that the Nix wrapper itself bakes.
resolveDrvPath is deferred
resolveDrvPath: ({ signal, onProgress, resolveSystem, resolveAgentDrv }) => Promise<AgentDerivation> resolves the agent derivation for the target’s arch, as an AgentDerivation.
Pass resolveDrvPath as a thunk. Eager resolution runs the ssh probe before the session exists, so an unreachable host would throw out of construction. Deferred, it becomes a retryable network fault inside the spawn cycle.
localEnv on a localhost dial
localEnv: Record<string, string> is required. A localhost dial runs the agent binary directly with spawn, and its environment comes only from localEnv.
- Omitting
localEnvis a compile error, onsshConnectoranddialAgentOncealike. - If
localEnvisundefinedat runtime (an untyped caller, anas any, a.d.tsor build skew), the localhost arm throws. It never falls back to inheritingprocess.env, so a local agent cannot inherit identity variables, secrets or a parent agent’s session markers. - The package neither composes nor scrubs the environment. The consumer hands over a clean one. kolu composes it from a fixed allowlist:
kolu-pty’scomposeSpawnEnv/SPAWN_ENV_ALLOWLIST. - On a real ssh host,
localEnvis unused. That child is the local ssh client, which inherits the caller’s environment forSSH_AUTH_SOCKand~/.ssh, and the agent’s environment is owned remotely. buildAgentCommandreturnsenv: Record<string, string>on the localhost arm andenv: undefinedon the ssh arm, so one place decides the environment per arm.
The readiness banner
After the spawn, sshConnector awaits the agent’s readiness banner (awaitStdioReadiness, @kolu/surface/links/readiness), raced against the child’s own exit. Only then does it build the link: the proof is what stdioLink requires.
- Your agent must greet.
serveOverStdiodoes this when the process is the agent. A daemon front, which relays into a durable daemon, must write the banner itself once it has settled that daemon. - A gate failure is a
"remote"fault, not a"network"one. A refusal, a deadline expiry or an unrecognisable prelude counts toward the session’s bounded remote budget. It goes terminal after a bounded number of attempts, provided the host stays up. - An interleaved
"network"failure means the host went away, and restarts that count. - A host that is merely down fails at spawn or exit before the banner. It keeps its
"network"classification and is retried forever. - A refusal’s opaque
anomalyrides theConnectError, so a binder can render a typed verdict instead of a spinner. - The wait is bounded at 3 minutes. That figure comes from the front’s own convergence ceilings (takeover reap 120s + 5s, socket-ready 30s, the probe’s 8s silence deadline, the front’s 10s daemon wait), not from any ping cadence.
SshKeepalive / sshKeepalive(intervalS, countMax)
keepalive?: SshKeepalive is a dial’s ssh dead-peer policy: ServerAliveInterval and ServerAliveCountMax. ssh may go intervalS × countMax seconds without an answer before it declares the transport dead and exits non-zero. Without it, an ssh parked on a half-open socket waits for the OS TCP stack, effectively forever.
| Name | Value |
|---|---|
DEFAULT_SSH_KEEPALIVE | 10 × 3 ≈ 30s. Right for an interactive tool: a host that stopped answering stops looking connected. |
sshKeepalive(30, 10) | Five minutes. For an unattended dial whose ssh should not be torn down over a slow probe answer. |
MAX_SSH_KEEPALIVE_TOLERANCE_S | One hour. The ceiling on how long a dead ssh may go unnoticed. |
sshKeepalive(intervalS, countMax)is the only way to make one. An out-of-range policy throws at the call you wrote, and it is never clamped.- The policy reaches every ssh the dial spawns: the arch probe, the cache prefetch, the warm validity check, the GC-root pin, the closure ship, the
NIX_SSHOPTSthat the remote-storenix buildforks its own ssh with, and the agent command. dialAgentOncetakes the same option and forwards it. Itslivenessoption is forwarded toMakeSessionOptions.liveness. That is a separate passthrough, not the other half ofkeepalive.- The cost is symmetric: a dead host keeps a long-keepalive dial parked for the same window before the loop retries.
What keepalive does not bound
keepalive only answers how long ssh may get no answer before it declares the transport dead. It is not a survival budget for the work riding that ssh. It is one of four independent bounds on how long a link may be silent:
- On a
connectedlink, the session’s heartbeat decides first (about 25s at its defaults), thenkeepalive(30s by default). - The link’s own keep-alive is a backstop above both. It ends a link that has received nothing for 60s, so 60–65s. A policy raised past it is inert on a connected link.
- During
provisioning, a separate per-step child-lifetime budget bounds the tolerance you request. It is a 30s deadline for a quick probe, an escalating 120s→960s allowance for a required build, and a flat 600s for a speculative copy. ssh keepalives never reset it, since they are protocol traffic and produce no child stdout.
The full breakdown of the four bounds is the keepalive option’s doc comment on SshConnectorOptions (packages/surface-remote/src/sshConnector.ts). packages/surface-remote/src/keepaliveOrdering.test.ts pins its figures against arithmetic. The doc comment is authoritative: if this page and it disagree, it is the one to correct.
sshDialOpts(keepalive)
The ssh options to use when you build your own ssh command outside this package. It is the same set the builders above emit: the dead-peer policy and this package’s ControlMaster decision. That decision is a per-policy socket or an explicit ControlPath=none, never silence.
- The commands of one dial share a single ssh
ControlMasterconnection. - OpenSSH applies
ServerAliveInterval/ServerAliveCountMaxfrom the process that opened the master. Every later ssh on that socket inherits them and ignores its own. - So the control socket is keyed by the keepalive policy. Two policies to the same host get two masters, and neither gets the other’s dead-peer behaviour.
- As a result, a CI-tuned dial and an interactive dial to the same host each keep their own warm connection.
ControlPersistis 10 minutes, and spans invocations. - Within one dial, every ssh carries the same policy by construction.
unixSocketConnector(opts)
The unix-socket Connector. It dials a daemon that is already listening on a path on this machine, and wires the socket to a typed client.
Options: { socketPath, group, client, describe }.
socketPathis a thunk (a function that returns the path), not a string. A daemon’s path comes from its environment, so a path captured once could go stale.- The connector owns the socket it dials: it hands it to
socketDuplexLinkas both halves. The socket’s ownclosetherefore resolves the connection’sclosed, and the session notices when its daemon dies. Dialling throughunixSocketLinkdoes not give you that. - A dial answered with
ENOENTorECONNREFUSEDthrowsEndpointAbsentError. It is a subclass ofConnectError, so everyinstanceof ConnectErrorcheck still matches it. It adds one fact: nothing is serving that path.
dialAgentOnce<S>(opts)
A one-shot dial for a CLI. It returns { client, dispose } with the link already proven live.
- It validates the baked source-flake ref, probes the host system, resolves only
packages.<system>.<package>.drvPath, pins it, and probes the link. - It takes the same
surfacevalue assshConnector, and the same requiredlocalEnv. package(the closure shipped) andbinary(the program run inside it) are both required, and neither defaults to the other. They differ when a host receives more than the daemon: kolu shipspadi-agentand runspadi.dispose()tears down only its own session.kaval-tui --host,padi-tui --hostandkolu-cli --hostuse it.
SURFACE_AGENT_FLAKE_REF_ENV / readBakedAgentSource()
The framework-owned handoff from the Nix wrapper, and the only adapter that reads it from the process environment. readBakedAgentSource() returns the normalized exact source, or throws AgentSourceUnbakedError.
resolveBakedAgentDrv(host, packageName, opts)
Resolves an agent from the wrapper-baked source inside a long-lived consumer’s own dial and failure boundary. The consumer never sees the environment handoff.
resolveAgentDrv(host, flakeRef, packageName, { signal, onProgress, budget })
Probes the host’s Nix and lazily resolves one package, inside the owning dial’s cancellation and progress budget. It returns { kind: "flake-installable", drvPath, installable, binaryCache }.
- The probe is
resolveHostNix: the host’s system and Nix version, as aHostNix. - The installable lets one Nix process own evaluation through remote realisation, so garbage collection cannot break the handoff.
- It reads the declaration from the baked source first, before the host probe and before any evaluation. A pre-contract wrapper therefore fails with
AgentBinaryCacheUnbakedError, not with whatever transport fault an unreachable host would have raised. - Inside a dial, the connector’s bound
ctx.resolveAgentDrvalso forms the dial’sIndexedClosure. It uses the declaration’sindexandrev, the package name and the probedHostNix, and passes it toprovisionAgentaspublished. That isnullwhen the source declares no index, or when the resolver returns a derivation of its own.
AgentDerivation / directAgentDerivation(drvPath, binaryCache)
The nominal provisioning source.
- Use
directAgentDerivationwhen you own a direct.drv. It validates its arguments. - Only
resolveAgentDrvconstructs the flake-backed arm, so a path cannot be paired with an unrelated installable. - Every arm requires an
AgentBinaryCache. A provisioning path that ignores the cache cannot be written.
AgentBinaryCache / agentBinaryCache({ substituters, trustedPublicKeys })
Where provisioning may prefetch an agent’s binaries from. It is nominal, like AgentDerivation.
- The smart constructor is the only way to make one.
- It rejects an empty or blank declaration, so a value of this type is always validated.
readBakedDeclaration(flakeRef) → Result<BakedDeclaration, AgentBinaryCacheUnbakedError>
Reads and validates the baked source’s binary-cache.json sidecar, and, when that names an index, the commit-hash beside it. Prefer it to writing a declaration by hand.
mkProvenAgentSourcewrites the sidecar from the agent flake’s ownnixConfigand itsclosureIndex.BakedDeclarationis{ caches: AgentBinaryCache; index: ClosureIndex; rev: string } | { caches; index: null; rev: null }. Both or neither: an index with no revision cannot be written.- A missing or empty
commit-hashis the typed error. - It uses the same channel as
readBakedAgentSource, so the two source-configuration faults chain:readBakedAgentSource().andThen(readBakedDeclaration).map((d) => d.caches).
ClosureIndex
Where a source’s builds are published: { repo, branch, rawBase }, a GitHub branch served raw.
binary-cache.nixderivesrawBase. Nothing in TypeScript spells an index URL.- Files are
<rawBase>/<rev>/<system>/<attr>.jsonand<rawBase>/latest/<system>/<attr>.json. - Each file is a
PublishedClosure = { rev, ia, ca? }: the full commit, the signed input-addressed path, and its content-addressed rewrite (absent on darwin).
closureIndexUrl(index, key, system, attr)
The URL of one entry.
keyis{ rev }(the publisher’s key, amkProvenAgentSourcesource’scommit-hash) or"latest".- A segment that could climb out of the index (
/, a leading., empty) throws.
readPublishedClosure(fetchText, index, key, system, attr) → Promise<ClosureLookup>
Looks up one published build with one HTTPS GET. The default fetcher is fetchClosureIndex: fetch, with a 30 s timeout. Tests pass any FetchText.
| Result | When |
|---|---|
{ kind: "published", url, entry } | a 200 |
{ kind: "not-published", url } | the index’s own 404: a dev build or an unpublished revision. A fact, not a fault. |
Errors: anything else throws a ClosureIndexError naming the URL, with kind: "unreadable" (no answer, another status) or "malformed" (not { rev, ia, ca? }, or a rev-keyed entry naming another revision).
IndexedClosure
The value of ProvisionOptions.published: where one build is published for one host. { index: Pick<ClosureIndex, "rawBase">; rev: string; attr: string; host: HostNix }.
- It is a plain value. The ssh connector forms it per dial from a baked source.
- A caller that owns its own
.drv(directAgentDerivation) states it.
closureRoute(hostNix, entry) → ClosureRoute
Which form of a published build a host pulls.
| Route | When |
|---|---|
{ kind: "content-addressed", storePath } | A Linux Nix that verifies the content-addressed form the publisher writes. |
{ kind: "signed", storePath, because: "mach-o" | "nix-line" } | macOS always, since a rewrite breaks Mach-O code signatures. Also any other Linux Nix line. |
- The forms are a private table, each named by the first Nix line that writes it:
2.17(the oldest line that imports a rewrite) and2.35. 2.35hashes a self-referencing path differently, so 2.34 and 2.35 refuse each other’s rewrite:ca hash mismatch importing path.- A Nix verifies only its own line’s form. Publishers rewrite with 2.34.8, so hosts on 2.17–2.34 pull
ca. - The sender’s own staging (
deliverClosure) reads the same table. - A Linux entry with no
caon the publisher’s form throws: the publication is broken. signedBecause(because, hostNix)words the reason for a refusal message.
isDarwinSystem(system) → boolean
Whether a Nix system tuple names macOS (*-darwin). closureRoute and closure staging use this one test to take the signed, input-addressed form, because a Mach-O rewritten to content-addressed loses its code signature. Call it to word a refusal by route rather than spelling the suffix.
Connector / Connection / ConnectError
The transport seam. A Connector dials once and returns a Connection, or rejects with a classified ConnectError. Write your own for a non-ssh transport.
- A
Connectionmay carry an optionalprocessAlivesame-box life-oracle. A local or stdio arm supplies it, and ssh omits it. See the liveness watchdog under Lifecycle invariants. ConnectErrortakes a fourth, optionalanomaly: unknown(defaultnull). It is the app’s own typed verdict for the failure, carried opaquely from the connector that classified it to the binder that renders it.- A binder therefore never parses
messageto tell “this daemon speaks a previous protocol epoch” from “this host is down”.
EndpointAbsentError
A connector’s verdict that nothing is serving there: absent rather than broken. It puts the session in its quiet absent arm instead of escalating toward failed.
- Only a connector that can tell absence apart may throw it. A unix-path
connect(2)answersENOENT(no socket file) orECONNREFUSED(a stale file nobody is bound to), and both mean the daemon is not here. - A websocket dial cannot tell, since it cannot distinguish absence from a network that swallowed the SYN.
- It is exported so a consumer tests for it by class, not by its message.
Admit / AdmitVerdict
An optional per-dial supervision hook. Its verdicts are adopt, refuse(state) and replaced(reason). Omit it and every connection is adopted.
The mirror pump
pumpRemoteSurface is the consume side of makeSession. It folds each successive client’s frames into your sink until the link dies, then waits for the next spawn. It pins nothing, so its first wait may legitimately be parked before any pin: reServeSurface decides by demand who holds the link open.
| Export | Subpath | What it is |
|---|---|---|
pumpRemoteSurface(opts) | . | Mirrors each spawn into a fresh sink until that link dies. |
SurfaceSink / makeSink | . | The sink role, and the factory you supply for it. |
reServeSurface(opts) | . | Re-serves one binding downstream, on demand, under a policy. |
makeClientCursor(session) | . | The raw-loop primitive: wait for the next fresh client. |
pumpRemoteSurface(opts)
Per spawn, it runs mirrorRemoteSurface into a makeSink() until that link dies, then awaits the next spawn. It pins nothing.
Options: { source, session, makeSink, liveProcedures?, liveClient?, onLinkDown?, connection?, signal?, log?, onFault? }.
signalstops the pump. It aborts the active spawn’s mirror, whose per-key pumps settle viasignal.reason, and it never destroys the caller-owned session.makeSinktakes{ seq }, the spawn counter, not a client. It is rebuilt every spawn, so no per-client state survives a reconnect.onLinkDownfires after holders clear. It is the cue to drop any per-link local fold, so the next spawn rebuilds from a fresh snapshot.
log and onFault
Two channels, with two levels. log is chatter (reconnects, link ends), so filter it freely, at debug if you like. onFault is a structured MirrorFault ({ label, err, scope: "member" | "key" }) and must be wired at error level.
- A
member-scoped fault also rejects the mirror’sdone. - A
key-scoped fault is deliberately key-local and never reachesdone, soonFaultis its only notice. - Both are forwarded verbatim from
reServeSurfacethrough the pump intomirrorRemoteSurface. - Why: Why faults get a channel of their own.
How the pump ends
- A rejecting
mirror.done(a member’s upstream stream faulting, or your sink throwing) is not reconnected: re-mirroring over a link that is still up would rebuild exactly the mirror that just failed. The pump narrates the death onlogand rethrows. - That rejection reaches
reServeSurface’sdone, where the consumer’s policy lives. kolu-server exits for the default host and retires a guest. The reconnect loop still owns the case it is for, a link that ended cleanly. - A mirror may also end while the link is still alive, when every subscription has settled and the far end closed its streams. No spawn is coming, so the wait for the next client would park forever serving nothing.
- Each mirror end therefore races that wait against the framework-reserved
system/liveround-trip on the client that just ended. A far end that still answers ends the pump, and itsdoneresolves. - A dead transport, or a probe that never answers, leaves the ordinary reconnect wait in charge. The session’s own liveness watchdog owns the half-open case.
SurfaceSink / makeSink
The sink role and its caller-supplied factory. Its members receive folded frames: cells.<n>.set(v), collections.<n>.{upsert,remove}.
- A collection that declares the
deltasverb is mirrored through its single batched snapshot-then-delta stream, folded into the same{ upsert, remove }sink: one protocol across the wire. - A collection without the verb keeps the per-key
keys+getfan-out. - The choice is read from the spec, never probed on the link.
reServeSurface(opts)
Policy-driven re-serve for one binding. It owns the pump, counts downstream demand, and returns a supervised { surface, group, handlers, done, close }, the same value pair implementSurface returns, so a host merges it into its own served surface with nothing to re-prefix.
Supervision and close()
close()aborts its pump via that signal and releases its runtime’s owned sources. It is idempotent.donealso rejects on an owned runtime fault.- The pump is a terminal source: it ends on its own when the session is destroyed. Supervision routes through
@kolu/surface’ssuperviseTerminalSource(runtime, terminal), the framework combinator that pairs a passive runtime with a terminal driver. - The terminal close is attempted first, and the runtime close is always attempted afterward. One close failure is rethrown unchanged; several are surfaced together as an
AggregateError.
policy
policy (Record<member, "value" | "delta">) splits members by how they survive an upstream drop.
- A
valuemember (cell, collection, value pulse) is held open and replayed across the drop. - A
deltamember (byte or liveness stream) fails through. A mid-stream upstream drop ends the downstream stream withSurfaceRelayTransportLost(@kolu/surface/errors), the one retryable framework error the shared retry fence accepts, so the client re-subscribes end to end and a snapshot only ever leads a fresh stream. - Both ends were built from that error’s class, which is what lets its
_tagsurvive the parent’s decode-and-re-encode. A raw re-raise would reach the downstream as an opaque, non-retriable failure and strand it. - A subscribe before any live spawn is up waits for the next spawn, on the same pre-connect arm as hold-open, rather than raising that retryable end every second.
- Direct callers of
relayFailThroughStream(and its hold-open sibling) must pass anObservableHolderfromobservableHolder(). The pre-bind wait needswhenChanged(), which a plainLiveSpawnHolderdoes not expose. - Holding open a byte stream is a compile error.
Demand gating
Every downstream call enters through one of reServeSurface’s own handlers, so that is where demand is counted: one gate for cells, collections, streams, events and procedures alike.
- The session is pinned on the crossing from nobody to somebody, and released on the way back.
- The release waits a fixed grace (
RELEASE_GRACE_MS, 30s), so a consumer whose subscriptions re-key crosses zero for free. - An upstream nobody is subscribed to is therefore never dialled, which is the point for an upstream that is optional, such as a daemon most machines do not run.
- A parent that wants an always-on link pins the session itself, and the count never reaches zero. kolu-server does that per host. A host’s link state is itself something the fleet shows, and a host dialled only once a browser subscribed would read
connectingwhile nobody looked at it. - Why: Why demand is counted in one place.
absent frames
The optional absent (Record<member, frame>) names, per value member, the frame a subscriber gets while the upstream is absent. It tells a hold-open subscriber that there is nothing there.
- A frame is sent once per absence episode, and replaced by the next live frame.
- The frame is a value the source contract declares (odu’s
CHECKOUT_NONE), never one this relay invents. - Absence is read off the session’s own
absentarm, not off the client holder going null, which also means “between spawns”. Only a session whose connector can tell absence from unreachability ever sends one, and every other re-serve omits the option. - Naming a member that is not a
value-classified stream or event throws at build. - Why: Why absence gets a frame of its own.
A standing refusal
A daemon that answers and speaks a contract this build cannot read is a refusal, the opposite fact, and never an absent frame. The admit verdict’s standing refusal is published on the session itself: session.refusal(), the sentence or null.
- A hold-open relay reading it stops holding and fails its subscription with
SurfaceUpstreamRefused(@kolu/surface/errors), carrying that sentence. - It is raised as a defect, because a streaming member declares no error channel, and a defect is never retried.
- Every hold-open relay gets the signal, whether or not its member declares an
absentframe. - Why: Why a refusal stops the relay.
makeClientCursor(session)
A stateful cursor, and the raw-loop primitive under the pump. cursor.next(signal?) blocks until a fresh client arrives after a reconnect, or wakes early when signal aborts.
The fleet fan-out
Two exports turn many sessions into one fleet: a keyed pool of sessions, and the adapter that publishes them as a host map.
| Export | Subpath | What it is |
|---|---|---|
buildRemotePool(opts) | . | The keyed pool of sessions a ?host= dispatcher reads. |
serveHostMap(map, pool, opts) | . | Publishes that pool as a @kolu/surface-map registry. |
buildRemotePool(opts)
A keyed Map<host, { session, handler }> that a ?host= dispatcher reads. Its verbs are add/remove/retire/reconnect/recheckAll, plus per-host socket eviction.
Options: { buildEntry, persist?, controls? }. buildEntry(host) returns the host’s { session, handler }.
remove(host) and retire(host)
Both share the same teardown mechanics: evict sockets, drop membership, destroy the session. They differ in two ways.
removeis the user’s departure. It calls thepersisthook first.retireis an internal shed, a dead session the pool sheds on its own. It does not persist, so a membership store re-seeds the host next boot.- Each verb labels its own WebSocket close reason and log line:
host removedorhost retired. - A pool with no
persisthook makes them behave the same on disk, since neither writes. The close reason and log still name the verb.
serveHostMap(map, pool, opts)
Adapts a pool into a @kolu/surface-map MapRegistry. It fuses membership with each session’s onState and projects each state into an entry status, reading the clockOffset the session’s own connected arm carries.
Options: { linkFor, failureOf, connection? }.
linkFor(host, session)is built once per host, and evicted on either removal or retirement, since both drop live membership.- There is no
offsetOfoption.makeSessionmeasures the far-end clock offset off the framework-reservedsystem.clockNowat admit and carries it on the session’s ownconnectedstate, whichserveHostMapreads directly. - Readiness is link-liveness. The entry is
connectedthe moment its link is live, carryingclockOffset: nullthrough (honestly not yet measured) until the first probe stamps it, and it is never demoted toconnecting. - Together with
buildRemotePoolit is the server half of a host map. See How to serve a map.
failureOf(host, session, state)
Required and total. It classifies a down session, and the registry publishes what it returns.
| It returns | For |
|---|---|
null | A transient down state: keep the entry warming. One meaning only, a classification verdict, never a fabrication. |
A schema-valid domain Failure | A standing down state. |
- A terminal
failedstate that yieldsnullfails loud withUnclassifiedHostFailureError: a failed entry cannot exist without a domain failure. - The distinct member-with-no-session (unknown-host) race fails loud with
UnclassifiedHostSessionError. - There is no
"other"fallback cause. - It is invoked fresh on every resolve, potentially once per member per sibling’s tick, with nothing held or cached against a frame.
- That is safe because
@kolu/surface-map’s republish gate compares the published status structurally (dequal). A classifier that mints a fresh-but-equal literal per call, as every real one does, is suppressed by the gate rather than by an adapter-side hold this module would have to keep in sync with it.
The failure’s evidence
At the same classification seam, serveHostMap staples the session frame’s retained log tail, the very frame failureOf just classified, onto the published failure record as evidence (see @kolu/surface-map’s FailureEvidence).
- Both down arms get it: a terminal
failed, and adisconnectedthe classifier called a standing refuse, which publishes asfailedtoo. - A transient
disconnectedpublisheswarmingand carries neither a reason nor evidence. SessionState.logis structurally identical toFailureEvidence, soraw.logis passed straight through. There is no injected evidence hook and no second evidence pipe.- The tail is pinned at classification, a post-mortem record rather than a live view, so it survives the client’s liveness floor with the reason it belongs to. The live
connectionword is dropped over a dead link.
The fail-loud seams
Both unclassified errors are genuine producer defects, so they crash the process even when raised deep in the internal per-member republish.
- The loop catches synchronously, so one member’s defect never freezes a sibling’s status stream mid-frame.
- It then rethrows out-of-band on the next microtask, as an uncaught exception, never a swallowed
console.errorthat would let the server degrade to a stale-but-healthy status.
Connection health
Connection health is state the remote agent cannot report, since it cannot observe the link to itself, so the parent authors it. It rides the host map’s entry as a fine connection payload: the one authority the coarse dot (EntryStatus.kind) and the fine word both derive from.
| Export | Subpath | What it is |
|---|---|---|
ConnectionInfo / ConnectionInfoSchema / DEFAULT_CONNECTION | /connection | The browser-safe type, its wire schema, and the pending value. |
projectConnection(s) | /connection | The pure leaf that turns a SessionState frame into one. |
sessionConnection(raw) | /connection | The total projection a host map feeds to serveHostMap. |
ConnectPhase | /connection | The up-but-not-yet-connected phases a progress UI narrates. |
LogEntry / LogEntrySchema | /connection | One { source, line } entry of the log tail. |
ConnectionInfo / ConnectionInfoSchema / DEFAULT_CONNECTION
The browser-safe connection type, the wire schema the host map validates an entry’s connection payload against, and DEFAULT_CONNECTION, the gate-closed pending value. All three ship from @kolu/surface-remote/connection.
ConnectionInfo is the discriminated mirror of the session’s SessionState sum minus one arm (Exclude<SessionState<SshProv>, { phase: "absent" }>), keyed on phase.
| Arm | What it says |
|---|---|
probing | The warm-check window: the agent is looked up, and nothing is shipped yet. |
provisioning | The ssh connector’s provisioning phase. |
connecting | The dial is up but not yet connected. |
connected | The link is live. It alone carries clockOffset. |
disconnected | Down. It adds error and cause ("network" transport-class, "remote" reached-and-refused). |
failed | Terminal. It carries cause: "network" | "remote". |
absentis excluded, because only a connector that can tell absence from unreachability produces it. A host is dialled over ssh or over the local endpoint, and neither can tell.- The transport class is orthogonal to terminality. A
MAX_CONSECUTIVE_FAILURESremote rejection gives up"remote", while a budget-exhausted silent provisioning step gives up"network".
Fields on every arm
| Field | What it carries |
|---|---|
log | The provenance-tagged tail: { source: "local" | "remote"; line }[]. |
sinceMs | The server-stamped elapsed of the current episode, in milliseconds. |
campaignEpoch | The monotonic campaign generation. |
sinceMsis a duration on the server’s single clock, never a foreign epoch the browser would subtract. The browser extends it smoothly with its own ticker.campaignEpochlets a client’s elapsed timer re-baseline when the same host starts a fresh campaign that reopens atsinceMs: 0after a quiet multi-minute stretch.- There is no separate
progressLinesorremoteProgressLines. Provenance is thesourcefield.
What a campaign scopes
Both log and sinceMs are scoped to the current campaign. They are stamped and reset at campaign birth: the first (or re-pin) dial, a user reconnect() or recheck(), and a drop of a connected link.
- They are not reset on a backoff retry within an ongoing connect campaign.
- So a wedged provision that retries for ten minutes reads ten minutes, not the per-attempt second or so.
- The log survives the retry dials (capped), so a coming-up card can render the attempt tail.
clockOffset on the connected arm
clockOffset: number | null is the far-end host’s wall-clock offset in milliseconds against the serving process. It is measured off the framework-reserved system.clockNow at admit, round-trip compensated (measureSurfaceClockOffset in @kolu/surface/clock-now).
- It is born
nulland re-stamped once the probe lands.nullmeans not yet measured. - Readiness is link-liveness, so
serveHostMapkeeps the entryconnectedwith anulloffset. A0placeholder would be a lie on a skewed host. - This is where the
clockOffseta keyedSurfaceMapfolds intoEntryStatus.connectedoriginates. It rides the session’s own state, not an injectedoffsetOf.
When the clock probe fails
A clock probe that fails is never swallowed. The session stays connected with a null offset, re-attempts on a fixed cadence, and surfaces the failure on its diagnostic logger, MakeSessionOptions.log.
- That logger is the
@kolu/logLoggershape: four level methodsdebug/info/warn/error, each(obj: Record<string, unknown>, msg: string) => void. Pass a pino child directly. - The session dispatches the severity itself, with a receiver-bound
log[severity]({ line }, label)call. Consumers never dispatch severity over extracted method references, so the unbound-thiscrash class has no spellable form. - A genuine failure (the wedged-clock deadline, a transport fault) is classified
error. - The expected-absent case is classified
debug: an agent dial whose client carries no reservedsystem.clockNow, which means the tool is not installed rather than that something failed. - Omit
logand diagnostic lines fall to rawprocess.stderr, the plain-CLI default. - A throwing logger crashes the session loop, deliberately: fail fast.
projectConnection(s)
The pure, browser-safe projectConnection leaf a consumer applies to a SessionState frame, or to the entry’s frame, to get a ConnectionInfo. It is the one named projection both the server and a client use.
- It is a near-identity. Every arm but one passes through untouched, since
ConnectionInfoisSessionState<SshProv>minusabsent. absentfolds todisconnected, carrying the absence’s own sentence. It is folded rather than cast away, because a sum with an unhandled arm is how a future connector’s new state reaches a browser asundefined.- The server reaches it through
sessionConnection, which isserveHostMap’sconnection.project. There is never a second server-side subscription. serveHostMapproduces the coarse status and the fineconnectionfrom the sameSessionStateframe in oneresolve, through its injectedconnection: { project, isConnected }.projectyields the word.serveHostMapasserts that the coarse dot andisConnected(word)agree, and fails loud before publication when they do not.- A consumer reads the word off the same entry it reads the dot from:
map.entry(host).state().connection, never a second subscription.
sessionConnection(raw)
The total (raw: SessionState<string> | undefined) → ConnectionInfo projection a host map feeds to serveHostMap’s connection.project.
- For a not-yet-seeded member it folds the gate-closed pending value.
- It validates the erased frame against
ConnectionInfoSchema, failing loud on a non-ssh or malformed frame. - Otherwise it returns the frame, reference-stable.
ConnectPhase
The up-but-not-yet-connected phase subset a connect or progress UI narrates: Exclude<ConnectionInfo["phase"], "connected" | "disconnected" | "failed">, that is SshProv | "connecting".
- It is the one vocabulary a UI’s exhaustive
switchorRecordkeys on, so a new provisioning phase fails to compile until it is handled.
LogEntry / LogEntrySchema
One { source, line } entry of the log tail. They are aliases of @kolu/surface-map’s EvidenceLine and EvidenceLineSchema: the same concept, so the same type and the same validator.
- They are taken from the schema-only
@kolu/surface-map/evidenceleaf, so this browser-safe module still pulls in no group-assembly code.
A mirror, end to end
const src = base;
const runtime = implementSurface(src, {
cells: {
load: { store: inMemoryStore(DEFAULT_LOAD) },
},
collections: {
processes: {
readAll: () => new Map<Pid, Proc>(),
upsert: () => {},
remove: () => {},
},
},
});
void session.pin();
void pumpRemoteSurface({
source: src,
session,
makeSink: ({ seq: _seq }) => ({
cells: { load: (v) => runtime.ctx.cells.load.set(v) },
collections: { processes: { upsert: () => {}, remove: () => {} } },
}),
});
Lifecycle invariants
What a session guarantees about its own state, its children and its retries, and the provisioning steps a dial runs.
| Export or topic | Subpath | What it is |
|---|---|---|
onState | . | Snapshot, then every later transition. |
currentState() | . | The synchronous twin of onState, and the liveness read. |
currentClient() | . | The dialling-or-connected client promise. Not a liveness gate. |
pin() / release() / destroy() | . | Intent to stay connected, and giving it back. |
| Timers and the process lifetime | . | Which of a session’s timers can hold a process open. |
| The dial’s process group | . | What a connector teardown kills. |
| Fault classes and the give-up budget | . | Which faults retry, which are bounded, which end it. |
| Absence | . | The steady state a missing endpoint settles in. |
| A host with no runnable Nix | . | The terminal probe verdicts, by exit code. |
| An ssh gate that cannot be answered | . | Credential and host-key refusals, terminal by design. |
recheck() and reconnect() | . | The two manual cycles, and what each disturbs. |
nudge() | . | Fire a scheduled reconnect now, and nothing else. |
runCapture and LifetimePolicy | . | The bounded lifetime every provisioning child runs under. |
| The pre-connected liveness backstop | . | The baked bound on a campaign that says nothing. |
provisionAgent(opts) → ProvisionResult | . | Realise an AgentDerivation on a host. |
resolveHostNix(host, opts) → Promise<HostNix> | . | The one-round-trip host probe every dial opens with. |
deliverClosure(opts) → Promise<DeliverClosureResult> | . | Get one exact store path into a host’s store and root it. |
pullClosure(opts) → Promise<PullClosureResult> | . | Have the host pull a published path straight from a cache. |
| The liveness watchdog | . | What a silent connected link means, per arm. |
| State progression | . | The phase order, and the backoff on a drop. |
| A mirror never fabricates a value | . | Why a mirrored cell serves nothing before its first fold. |
onState
Snapshot, then deltas. A listener attached at any point sees the current state synchronously before any later transition, the same contract useCell consumers expect.
currentState()
The synchronous twin of onState. It returns the current published SessionState frame, the point-read companion to the snapshot-then-delta subscription, reading the same cell onState publishes.
- It always returns the freshest frame.
onState’s delta delivery is microtask-deferred, while the initial snapshot fires synchronously.- One synchronous frame can drive two transitions (
disconnectedto a give-upfailed), so a listener delivered one delta frame may read a later one here. - Honest liveness is
currentState().phase === "connected".
currentClient()
The in-flight or current client promise. It is not a liveness gate.
- It is legitimately non-null while merely dialling: during
connectingand provisioning, and across an entire reconnect backoff window, becausescheduleReconnectretains the rejected dial. - The pump and
markConnectedhandshake depend on that dialling-or-connected meaning. - So
currentClient() !== nullnever means the far end is live. ReadcurrentState().phasefor that.
pin() / release() / destroy()
pin() is intent, and release() gives it back.
pin()bumps the ref-count unconditionally, so the session keeps reconnecting even if the first spawn fails, and resolves with the first client.release()decrements the same count. At zero the session lets its connection go after a fixed grace (RELEASE_GRACE_MS, 30s), then re-arms on the nextpin()with a fresh dial.- The grace is a constant, never an option: a consumer whose subscriptions re-key drops to zero pins and back within one tick, and the grace makes that crossing free.
destroy()drops the pin regardless of the count.
Timers and the process lifetime
A session’s internal timers never hold the process alive. The reconnect backoff, the connect watchdog and the clock probes are unref()’d, so a pinned but abandoned session with a down link cannot keep an otherwise-finished process running.
- A live link is different: its ssh child or socket is a real event-loop hold while it lasts.
- A consumer whose sole purpose is waiting on a session must hold the process by its own means: a server socket, stdin, a live transport.
- So a parked
ClientCursor.next()or onState-derived wait is not a process hold. Across a reconnect gap it is settled only by the backoff firing, and if nothing else holds the loop it goes silently unsettled as the process exits. - The one deliberately ref’d timer is the admit-handshake timeout. It must fire even as the last handle standing, because it rejects a
pin()a caller is awaiting.
The dial’s process group
An agent dial owns its whole process group. The long-lived agent transport starts in a fresh POSIX process group, and every connector teardown signals that exact group: SIGTERM, then SIGKILL after a short grace.
- That includes an ssh client’s
ProxyCommandshell and any nested ssh it starts, so a failed dial that retries leaves no subprocess behind. - The same private ownership primitive backs
runCapture’s bounded provisioning children. Session policy and one-shot lifetime policy remain separate consumers of it.
Fault classes and the give-up budget
The budget is a classified ledger (@kolu/surface’s makeFailureLedger), one run per failure class, so no class can spend another’s.
| Fault | What it does |
|---|---|
network | Transport-class: the host is unreachable, or a transport wedged or was killed. Retries at capped backoff (60s). |
remote | The host answered and the attempt still failed: a Nix build error, an agent that would not start. Bounded. |
- A
networkfault does not itself count toward give-up, and it resets the remote run. An unreachable gap means the host went away, so the next remote blip is fresh evidence of a sleeping host. - A
remotefault is bounded byMAX_CONSECUTIVE_FAILURESconsecutive remote failures. It then surfaces terminalfailedplusremote, with a give-up line derived from the ledger’s verdict, so it can only name the run that actually tripped, followed by the last failure’s own reason, never a guessed remedy. - Each retry line says what the attempt ran into (
attempt N failed: <reason> — retrying in …) rather than a fixed label.networkis a retry class, not proof the host is gone. - An agent dial’s exit 255 is
transport-failedonly when ssh’s stderr named a transport failure, read as whole lines, with a short bounded wait for that line after the exit. A remote agent that exits 255 by itself is an ordinaryexit. - One
networkfault does terminate: a budget-exhausted silent provisioning step. The ssh connector’s per-step progress-liveness budget gives up after N consecutive silent kills, surfacing terminalfailedplusnetwork, the honest transport cause. Terminality is the phase, not a rewrite of the cause. - Why: Why the session sorts silence into kinds.
Absence
Absence is a steady state, not a failure. A connector that throws EndpointAbsentError puts the session in its absent arm, carrying the dial’s own sentence as error and no cause, because nothing failed: the endpoint is simply not there.
- It records nothing in the give-up ledger, so an absence never ages into a terminal
failed. - It re-probes at the reconnect ladder’s own ceiling rather than ramping, since nobody’s daemon gets started by polling for it.
- It publishes at debug level. A line per probe at
infowould fill an operator’s log with the news that software they do not run is not running. - It is never terminal, so the next
pin()on a daemon that has since started connects normally. - Why: Why the session sorts silence into kinds.
A host with no runnable Nix
Provisioning realises the agent with the host’s own Nix, so a probe whose shell exits 127 (the POSIX “command not found” for nix-instantiate) is terminal.
| Rejection | When |
|---|---|
{ kind: "nix-unavailable", failureCause: "remote", terminal: true } | The probe shell exited 127. |
{ kind: "unavailable", failureCause: "remote", terminal: false } | nix-instantiate ran and failed (non-127, and ssh reported no transport failure). It carries Nix’s own error. |
- The 127 case is classified by the exit code, never by the shell’s wording: bash, dash and fish word it three different ways.
- The non-127 case is retried, but bounded, because the host answered. Probe output that is not a nix-system string rejects the same way, commonly a shell startup file printing to stdout.
- Our own kills and a transient local spawn fault stay untyped retried errors.
- Why: Why the session sorts silence into kinds.
An ssh gate that cannot be answered
Every ssh this package spawns is non-interactive (BatchMode=yes, no TTY), so a host that demands a credential or an unverified host key does not prompt. ssh declines and exits 255.
- The arch probe every dial opens with reads those two refusals off ssh’s stderr and rejects a
ResolveDrvErrorwhose resolution is{ kind: "auth-refused" | "host-key-unverified", failureCause: "remote", terminal: true }. - It is
remotebecause the peer was reached and refused us, and terminal because no redial can supply the missing answer. So a password-only host settles onfailedwith an actionable reason in seconds. - Classification is conservative. It requires ssh’s own exit 255 as well as the stderr text, since a remote command’s stderr rides the same stream.
- Any refusal it does not recognize stays an untyped, retried transport error.
- The probe is the natural chokepoint: a refusal met later (Nix’s own ssh fork, the agent dial) kills that dial, and the redial’s probe meets it here.
recheck() and reconnect()
Two manual cycles, with different reach.
| Call | What it does |
|---|---|
reconnect() | Re-arms a failed or idle session, and will not disturb a live link. The manual button. |
recheck() | Force-cycles whatever is there, including a connected link and an in-flight dial. The wake or network-change companion. |
recheck()suits a post-sleep link, which is often stale.- On an in-flight dial,
recheck()aborts that dial (the connector’sctx.signalfires, group-killing any provisioning child) and redials now, with no backoff. - Every
recheck()andreconnect()is a fresh campaign: freshsinceMsand log, fresh connector budgets.
nudge()
A scheduled reconnect fires now. Every other state is a no-op.
- It is the signal a consumer sends when the world plausibly changed under a down session. kolu-server fires it on every websocket client accept, because a waking laptop’s first observable act is its browser reconnecting. The alternative is up to a full capped backoff (60s) of dead remote panes.
- It is the narrow verb, not a second spelling of
recheck(). It fires the same attempt the backoff had already scheduled, keeping its attempt number and its position in the give-up budget, and records nothing in the failure ledger. recheck()instead calls the ledger’ssuccess(), refilling every class’s run, and stamps a fresh campaign. A permanently-broken host nudged often enough would otherwise never reach its terminal verdict.- It adds no unboundedness: it only fast-forwards an already-armed timer, so it cannot create an attempt the existing schedule would not have made.
- The fired-now path narrates on the session log:
nudged — firing the scheduled retry now … give-up budget unchanged.
The three no-op arms
| State | Why nothing happens |
|---|---|
| A live link | Nothing is scheduled to fast-forward, and a client connecting is no evidence this link is stale. Cycling it is recheck()’s job. |
| An in-flight dial | The probe is already running. This no-op is the wake-storm coalescing, with no lock of its own. |
Terminal failed | Reviving a spent budget is reconnect()’s job, never an ambient signal. |
runCapture and LifetimePolicy
The fire-and-collect helper (/ root) that provisioning children run under. It takes a required LifetimePolicy.
| Policy | For |
|---|---|
{ kind: "deadline"; ms } | A quick step. |
{ kind: "progress-liveness"; silenceMs } | The minutes-long provision: killed only on real output-silence, never a healthy slow transfer. |
- Any output resets that budget, unless the caller passes
isActivity(line) → booleanfor a structured stderr that also carries heartbeats. Then only stdout and the lines it answerstruefor count. - A kill settles the distinct
ExitResultarmlifetime-expired(retryable); a user abort settlesaborted. Both group-kill the child (and its ssh grandchild) and settle at the kill, never awaiting aclosethat may never fire. - The exported
runCaptureis location-blind and gets only the local group-kill. A caller that hands it an ssh argv of its own does not get the remote kill below.
ConnectContext and remote kills
ConnectContext carries a per-dial signal (the abort) and a campaignEpoch (fresh connector budgets per campaign), plus activity().
activity()is a liveness signal with no log line. It resets the pre-connected backstop exactly as a progress line does.- A connector calls it for child output it deliberately does not narrate, so the backstop keeps dominating the child’s own silence bound.
- The ssh connector feeds it every activity line of a long-running provisioning Nix command, through
ProvisionOptions.onActivity(required). That is every stderr line except a Nix progress result whosedonecounter has not moved, which Nix re-sends about once a second even while a transfer is stalled. - Every one-shot command this package runs on a remote host (the provisioning Nix steps, the warm check, the arch probe) goes through one internal runner that wraps it to report its PID first. A run that ends aborted, or killed by its own lifetime policy, is
SIGKILLed on the host by that PID before the step returns, so a retry never queues behind the abandoned one. - That remote kill is needed because killing the local ssh does not reach a tty-less remote command, and a Nix blocked in a socket write survives
SIGTERM. - No provisioning Nix command passes
-v. At that level a daemon withauto-optimise-storeforwards a line per unpacked file to a client that is busy writing the NAR, and the two deadlock.
The pre-connected liveness backstop
While a campaign is coming up, if no progress line arrives and no phase advances for the baked DEFAULT_PRE_CONNECTED_LIVENESS_MS (20min), the session cycles the attempt (abort and redial), narrated.
- The bound is not configurable. It must stay above the connector’s maximum step-silence budget so the per-step budget always ends the step first; a lower bound would let the backstop reset the budget and loop a silent copy forever.
- It is liveness, so a chatty build resets it every line, and it is never capped.
- It guards the seam the per-child policies cannot: a wedge in a non-helper await such as
resolveDrvPath. - Why: Why the session sorts silence into kinds.
provisionAgent(opts) → ProvisionResult
Realise an AgentDerivation on a host (/ root). It is the ssh connector’s provisioning step, usable standalone.
- A flake-backed derivation is evaluated locally and provisioned into the remote
ssh-ngstore by onenix build, so Nix owns the temporary roots across evaluation, transfer and realisation. A parent on darwin can front a target on linux. - There is no tarball, Docker or prebuilt fallback.
- A warm host short-circuits through an ask-only check: a sender-local
nix-store -q --outputs, then a boundednix-store --check-validityon the host, never a substituting--realise. - Every path succeeds only after
nix-store --realise --add-rootcommits the output behind its stable target-store GC root. agentPathis the path delivery rooted, the content-addressed one when it travelled content-addressed.
Route order
- The warm check.
- The index step, when
publishedis non-null and the host is remote. A localhost staging reads its own store first. - Delivery of the agent’s output closure, before the cold build.
- The cold build on the host, with its own substituters.
- The index step is
readPublishedClosureforpublished.rev,closureRouteforpublished.host, andpullClosureon the host from each declared substituter in turn, rooted at the agent’s GC root under the dial’sbudgets.provisioning. - A landed index step returns there: no rewrite, prefetch, ship or build, and no trust on the host for a content-addressed path.
not-published, an unreachable index, a pull no cache would give (the signed path to a host that does not trust it), or a pull that went silent past the copy bound each narrate and move on.- A malformed entry fails the provision (
cause: "remote").
Delivering the closure
Delivery never builds. It sends the closure in the form the target takes, decided once the warm check has missed, by the target’s system (read off the .drv with nix derivation show) and, for Linux, by three Nix versions asked fresh each delivery.
| Nix asked | How |
|---|---|
The binder’s nix | nix --version |
| The binder’s store | nix store info --json, its daemon’s version |
| The target’s store | nix store info --json --store ssh-ng://<host>; a local target’s store is the binder’s |
- To a Linux target whose three Nixes all handle the same content-addressed form, the closure is rewritten content-addressed in the binder’s store and shipped with
nix copy --to ssh-ng://<host>. Any Nix on that form accepts it with notrusted-usersentry and no key. - The rewrite is
nix store make-content-addressed, from the local store when it holds the closure, else from each of the derivation’sbinaryCachesubstituters in order. - To a darwin target, a system nobody named, or a Linux target whose Nixes disagree on the form, the signed input-addressed closure is prefetched into the binder’s store (
nix copy --from) and shipped withnix copy --no-check-sigs --to ssh-ng://<host>. The disagreement is narrated:sending the closure as built: …, naming the versions. - Without
--no-check-sigsthe copy asks for signature checks even from a trusted user, so a locally-built closure never lands. - Rewriting a Mach-O breaks its code signature, and no rewrite is attempted that a store is known to refuse.
- A version probe that cannot answer stops the delivery at its own seat.
Where the rewrite is rooted
- A Linux target’s rewrite is rooted in the binder’s store per output path and form (
~/.local/state/kolu/surface-remote/content-addressed/<name>/<hash>@<form>). That record is what a later dial, after a restart too, asks the host about first, and it is reused only while its form is still the route’s. - The staging and the ship are both load-bearing. The local store is the only seat where a declared cache can act, since a remote-store realisation substitutes with the remote daemon’s own nix.conf.
- The cold build itself transfers only the derivation closure: locally-valid outputs are never consulted by the remote daemon.
- An as-built ship lands only when the target trusts it: a
trusted-usersssh user, or signatures the host trusts. - A miss, a signature refusal, or a rejected ship each narrate Nix’s own error into the progress tail and fall back to realising on the host with its own substituters, never failing the dial.
Options and result
ProvisionOptions: { host, derivation, published, onProgress, onActivity, onProvisioning?, budgets, keepalive, signal? }.
published: IndexedClosure | null is required: null states “no index”.
| Option | Meaning |
|---|---|
onActivity | Receives every activity line of the long-running Nix steps, narrated or filtered. Wire it to ctx.activity. |
budgets | A connector-owned ProvisionBudgets (makeProvisionBudgets()). Its evaluation and provisioning progress-liveness budgets persist across a campaign’s retry dials. |
signal | The abort. |
ProvisionResult is { ok: true; agentPath } or { ok: false; reason; cause: "network" | "remote"; terminal? }.
terminalmarks a budget-exhausted silent step that must give up at once. It is decoupled fromMAX_CONSECUTIVE_FAILURES, so the backstop cannot reset it away.
How a failure is classified
Every Nix command provisioning runs uses --log-format internal-json, and is read by what Nix tagged.
- A failure’s
reasonis Nix’s first error plus the failed build’s last log line. The dependents’ “1 dependency failed” cascade is fallout, and the root error is re-narrated last so a bounded tail keeps it. - A failure is classified
networkonly on what can speak for the connection: ssh’s own untagged stderr, or the headline of Nix’s own error (a failed ssh connection, an unreachable download). - It is never classified from a builder’s log, nor from Nix quoting one.
resolveHostNix(host, opts) → Promise<HostNix>
The host probe every dial opens with (/ root), with the full signature resolveHostNix(host, opts: ResolveSystemOptions) → Promise<HostNix>.
- It is one
nix-instantiate --eval --strict --jsonround-trip:{ system = builtins.currentSystem; nixVersion = builtins.nixVersion; }. - It is classified exactly as A host with no runnable Nix and An ssh gate that cannot be answered describe.
HostNixis{ system, nixVersion }. There is noresolveSystemexport; a resolver inside a dial keepsctx.resolveSystem(), which is this probe’ssystemhalf.- The version matters for a content-addressed path: one is only verifiable within the Nix line that wrote it, since Nix 2.35 hashes a self-referencing path differently from 2.34 and neither accepts the other’s. A caller choosing between a published path’s content-addressed and signed forms chooses by it.
deliverClosure(opts) → Promise<DeliverClosureResult>
Get one exact, already-evaluated store path that the caller holds (or a declared cache does) into a host’s store, and root it there (/ root, signature deliverClosure(opts: DeliverClosureOptions)). It is provisionAgent’s delivery step on its own, and never a build.
Its steps are the ask-only warm check, the index step and the staging provisionAgent describes:
- The warm check is remote only: the content-addressed form this sender rooted for
outPath, in whichever form it was written, thenoutPathitself. A hit is rooted and returned. - The index step runs when
publishedis non-null. - To a Linux target whose Nix and the sender’s (its
nixand its store) all handle the same content-addressed form, the closure is rewritten content-addressed in the sender’s store. It ships withnix copy --to ssh-ng://<host>, which that Nix accepts with no trust. The rewrite reads the local store when it holds the closure, elsebinaryCache. - To darwin, to a system nobody named, and to a Linux target across Nix 2.35 from the sender (or a sender whose
nixand store disagree), the input-addressed closure goes as built. It is signed when a cache signed it, and by no one when the caller built it. It is prefetched when absent, and shipped withnix copy --no-check-sigs, which lands only when the host trusts the login. - Then comes the required root commit at
rootPath. - kolu-server ships the profiles it builds for the Agents setting through it.
DeliverClosureOptions
{ host, system, outPath, binaryCache, published, rootPath, onProgress, onActivity, onDelivering?, budget, signal?, keepalive }.
| Option | Meaning |
|---|---|
system | system: () => Promise<string | undefined>, asked only when the warm check misses. |
binaryCache | binaryCache: AgentBinaryCache, required even for a path the caller built: delivery is never cache-blind. |
published | IndexedClosure | null. |
rootPath | As for pullClosure. |
budget | budget: StepBudget, the root commit’s budget. |
DeliverClosureResult
| Result | What it says |
|---|---|
{ ok: true; outPath } | The path the target holds: the content-addressed one when it travelled content-addressed. |
{ ok: false; kind: "not-delivered"; reason; at: "sender" | "target"; asBuiltBecause: string | null } | The closure is not in the target store, and nothing was rooted. |
{ ok: false; kind: "failed" } plus ProvisionResult’s failure shape | An abort, or a failed root commit. |
On a not-delivered result:
reasonis Nix’s words verbatim: a cache miss, a refused rewrite, a refused ship, or a Nix that would not say its version.- The arm in full is
{ ok: false; kind: "not-delivered"; reason; at: "sender" \| "target"; asBuiltBecause: string \| null }. atis the seat it stopped at.sendermeans this machine never held the closure in the form to ship, because the rewrite, the prefetch, or asking its own Nix failed.targetmeans the ship, or asking the target’s Nix, failed.asBuiltBecausewords why it was sent as built (Mach-O, an unnamed system, or the Nix versions that disagree). It isnullwhen it travelled content-addressed, or the route was never decided.
pullClosure(opts) → Promise<PullClosureResult>
Have a host pull one published store path straight from a binary cache and root it (/ root, signature pullClosure(opts: PullClosureOptions)). The caller moves no bytes, and evaluates, rewrites and builds nothing.
Every step runs at the target, over ssh for a remote host and directly for localhost:
- An ask-only
nix-store --check-validity <storePath>. A hit skips the copy. - Otherwise
nix --extra-experimental-features nix-command copy --from <cacheUrl> <storePath>, under the fixed copy boundPROVISION_COPY_SILENCE_MS. - The required
nix-store --realise --add-root <rootPath> --indirectcommit.
- The feature flag is a client-side parse gate any login may lift, so a stock host Nix needs no setting.
- Nix’s per-path events and advancing progress prove a multi-gigabyte download alive, and a stalled transfer’s repeated, unmoved progress does not, so a wedged copy is killed, on the host too.
--fromnames a store to read rather than proposing a substituter, so no trust setting filters it. A content-addressed path is accepted by any daemon with no signature, key ortrusted-usersentry, so an untrusted login with nonix.confchange works.- An input-addressed path is taken only when the host’s Nix trusts its signature, so the caller names the signing key as
signedBy(passed as--extra-trusted-public-keys, honoured for a trusted login; an untrusted one needs the key in the host’strusted-public-keys). - Content-addressed is no option for a macOS target: rewriting a Mach-O invalidates its code signature and macOS kills it on launch. Nor for a host whose Nix line differs from the one that wrote the path (
resolveHostNix).
PullClosureOptions
{ host, cacheUrl, storePath, signedBy?, rootPath, onProgress, onActivity?, keepalive, signal?, budget }.
| Option | Meaning |
|---|---|
budget | budget: StepBudget is required: the root commit’s progress-liveness budget, the caller’s own, so repeated silent commits end terminal. A dial passes budgets.provisioning; a one-off caller makeStepBudget(PROVISION_STEP_SILENCE_BASE_MS, PROVISION_STEP_MAX_EXPIRIES). |
signedBy | Only for an input-addressed storePath. |
rootPath | Resolved against the ssh login’s home when relative, and it must be absolute for localhost. homeGcRoot(isLocal, rel) forms it. |
keepalive | Required, as for every ssh this package spawns. |
- The root commit runs under a fresh provisioning
StepBudget.
PullClosureResult
| Result | What it says |
|---|---|
{ ok: true } | storePath is valid in the target store, and rooted. |
{ ok: false; kind: "uncopied"; reason } | The host’s copy from the cache did not land: not in the cache, no egress, or silent past the copy bound. A miss, like every speculative copy. reason is Nix’s words verbatim. |
{ ok: false; kind: "failed" } plus ProvisionResult’s failure shape | A dropped ssh (cause: "network"), an abort, or a failed root commit (terminal once its budget is spent). |
The liveness watchdog
Default-on. While connected, the session probes the reserved system.live round-trip on an interval. A rejection still counts as alive, because the round-trip completed. Opt out with liveness: false.
What a timeout does depends on the arm’s structure, through the optional Connection.processAlive same-box oracle:
| Arm | On a timeout |
|---|---|
| Remote (ssh, no oracle) | Reads silence as death and force-cycles the child, because the network can die silently. |
Local (endpoint or stdio, with processAlive) | Consults the same-box process table. Process gone: force-cycle now. Process alive: the link is merely slow under load, so it keeps probing. |
- Link teardown cannot restart that same-box process, so an arbitrary heartbeat-silence ceiling would neither distinguish load from deadlock nor heal a deadlocked daemon.
- This is arm structure, not a knob.
State progression
probing → provisioning → connecting → connected. On a drop, disconnected → probing → … at exponential backoff capped at 60s.
- A warm target still crosses
provisioningfor its mandatory root refresh. That is normally near-instant, but a GC race may require restoration under the same minutes-long lifetime contract. - An uncached exact-source evaluation can enter the phase earlier.
A mirror never fabricates a value
A re-served (mirrored) cell serves no frame until the authority’s first real fold.
- Its store still seeds the declared default, so
store.get()is a typedT. - The framework’s
hasSnapshotgate withholds that seed until the fold writes the authority’s first frame. On a mirror the only writer is the fold, never a wire client. - So the reader’s
T | undefined(“no frame yet”) holds end to end, and the declared default belongs to the one writer. - Why: Why values replay and byte streams break.