kolu
Docs

@kolu/surface-remote reference

Run a typed @kolu/surface agent on a remote machine over ssh, or read one listening on a unix socket beside you. The package owns provisioning (ship a .drv, realise it on the target), the long-lived ssh subprocess (ref-counting, reconnect, backoff), and the observable state of the link’s own lifecycle.

It has two public subpaths: @kolu/surface-remote and the browser-safe @kolu/surface-remote/connection.

The session

A session is the reconnect appliance. makeSession is the transport-agnostic loop, and a connector is the transport you plug into it: sshConnector for a remote host, unixSocketConnector for a daemon on this machine. You own each session. There is no shared pool: key your own map and destroy() each session yourself.

ExportSubpathWhat it is
makeSession(opts).The reconnect, backoff, give-up and watchdog loop.
sshConnector<S>(opts).The ssh Connector: provisions the agent on a host and dials it.
SshKeepalive / sshKeepalive(intervalS, countMax).The ssh dead-peer policy for a dial.
sshDialOpts(keepalive).The ssh options to use when you build your own ssh command.
unixSocketConnector(opts).Dials a daemon listening on a unix socket on this machine.
dialAgentOnce<S>(opts).A one-shot dial for a CLI.
SURFACE_AGENT_FLAKE_REF_ENV / readBakedAgentSource().The source flake the Nix wrapper bakes, and its reader.
resolveBakedAgentDrv(host, packageName, opts).Resolves an agent from the baked source inside your own dial.
resolveAgentDrv(host, flakeRef, packageName, { signal, onProgress, budget }).Probes a host’s Nix and resolves one agent package.
AgentDerivation / directAgentDerivation(drvPath, binaryCache).What provisioning ships: a validated derivation.
AgentBinaryCache / agentBinaryCache({ substituters, trustedPublicKeys }).Where provisioning may prefetch an agent’s binaries from.
readBakedDeclaration(flakeRef) → Result<BakedDeclaration, AgentBinaryCacheUnbakedError>.Reads the binary caches and closure index a source declares.
ClosureIndex.Where a source’s builds are published.
closureIndexUrl(index, key, system, attr).The URL of one entry in a closure index.
readPublishedClosure(fetchText, index, key, system, attr) → Promise<ClosureLookup>.Looks up one published build.
IndexedClosure.Where one build is published for one host.
closureRoute(hostNix, entry) → ClosureRoute.Which form of a published build a host pulls.
isDarwinSystem(system) → boolean.Whether a Nix system tuple names macOS.
Connector / Connection / ConnectError.The transport seam a session dials through.
EndpointAbsentError.A connector’s verdict that nothing is serving there.
Admit / AdmitVerdict.An optional per-dial supervision hook.

makeSession(opts)

The reconnect loop. It handles backoff, giving up and the liveness watchdog, and returns a Session<AgentClient, Prov>.

  • It owns no transport. You supply connectOnce, usually a connector from this section.
  • Its watchdog and timers are described under Lifecycle invariants.

sshConnector<S>(opts)

The ssh Connector. On each dial it resolves the .drv, provisions it, spawns ssh <host> <binary> --stdio, awaits the agent’s readiness banner, and then wires stdio to a typed client.

Options: { surface, host, binary, resolveDrvPath, localEnv, extraArgs?, keepalive? }.

OptionMeaning
surfaceThe Surface<S> the remote agent serves, passed as a value.
hostAny ssh target. "localhost" short-circuits ssh.
binaryThe executable inside the realised closure. It is also the package name, so the two cannot drift apart.
resolveDrvPathResolves the agent for the target’s arch. See below.
localEnvRequired. The whole environment of a localhost dial. See below.
extraArgs?Arguments appended after --stdio on the agent command. POSIX-quoted for a real remote, verbatim for localhost. Their meaning is the caller’s concern.
keepalive?An SshKeepalive. Defaults to DEFAULT_SSH_KEEPALIVE.
  • surface is a value, not a type parameter. Effect RPC builds the client from surface.group, and the face is re-nested from surface.spec. Neither can be recovered from a type.
  • resolveDrvPath runs inside the owning dial’s cancellation and progress lifetime. For a direct path, return directAgentDerivation(drvPath, binaryCache). Get the binaryCache from readBakedDeclaration(source)’s caches wherever the deployment bakes an agent source, so it comes from your flake’s own nixConfig. Use agentBinaryCache({…}) inline when there is no baked source. resolveAgentDrv returns the flake-backed arm, which survives local garbage collection.
  • Inside resolveDrvPath, const sys = await ctx.resolveSystem() runs the arch probe on this dial’s host, signal, progress sink and keepalive policy. There is no options bag to assemble.
  • The package reads the exact, minimal source flake that the Nix wrapper itself bakes.

resolveDrvPath is deferred

resolveDrvPath: ({ signal, onProgress, resolveSystem, resolveAgentDrv }) => Promise<AgentDerivation> resolves the agent derivation for the target’s arch, as an AgentDerivation.

Pass resolveDrvPath as a thunk. Eager resolution runs the ssh probe before the session exists, so an unreachable host would throw out of construction. Deferred, it becomes a retryable network fault inside the spawn cycle.

localEnv on a localhost dial

localEnv: Record<string, string> is required. A localhost dial runs the agent binary directly with spawn, and its environment comes only from localEnv.

  • Omitting localEnv is a compile error, on sshConnector and dialAgentOnce alike.
  • If localEnv is undefined at runtime (an untyped caller, an as any, a .d.ts or build skew), the localhost arm throws. It never falls back to inheriting process.env, so a local agent cannot inherit identity variables, secrets or a parent agent’s session markers.
  • The package neither composes nor scrubs the environment. The consumer hands over a clean one. kolu composes it from a fixed allowlist: kolu-pty’s composeSpawnEnv/SPAWN_ENV_ALLOWLIST.
  • On a real ssh host, localEnv is unused. That child is the local ssh client, which inherits the caller’s environment for SSH_AUTH_SOCK and ~/.ssh, and the agent’s environment is owned remotely.
  • buildAgentCommand returns env: Record<string, string> on the localhost arm and env: undefined on the ssh arm, so one place decides the environment per arm.

The readiness banner

After the spawn, sshConnector awaits the agent’s readiness banner (awaitStdioReadiness, @kolu/surface/links/readiness), raced against the child’s own exit. Only then does it build the link: the proof is what stdioLink requires.

  • Your agent must greet. serveOverStdio does this when the process is the agent. A daemon front, which relays into a durable daemon, must write the banner itself once it has settled that daemon.
  • A gate failure is a "remote" fault, not a "network" one. A refusal, a deadline expiry or an unrecognisable prelude counts toward the session’s bounded remote budget. It goes terminal after a bounded number of attempts, provided the host stays up.
  • An interleaved "network" failure means the host went away, and restarts that count.
  • A host that is merely down fails at spawn or exit before the banner. It keeps its "network" classification and is retried forever.
  • A refusal’s opaque anomaly rides the ConnectError, so a binder can render a typed verdict instead of a spinner.
  • The wait is bounded at 3 minutes. That figure comes from the front’s own convergence ceilings (takeover reap 120s + 5s, socket-ready 30s, the probe’s 8s silence deadline, the front’s 10s daemon wait), not from any ping cadence.

SshKeepalive / sshKeepalive(intervalS, countMax)

keepalive?: SshKeepalive is a dial’s ssh dead-peer policy: ServerAliveInterval and ServerAliveCountMax. ssh may go intervalS × countMax seconds without an answer before it declares the transport dead and exits non-zero. Without it, an ssh parked on a half-open socket waits for the OS TCP stack, effectively forever.

NameValue
DEFAULT_SSH_KEEPALIVE10 × 3 ≈ 30s. Right for an interactive tool: a host that stopped answering stops looking connected.
sshKeepalive(30, 10)Five minutes. For an unattended dial whose ssh should not be torn down over a slow probe answer.
MAX_SSH_KEEPALIVE_TOLERANCE_SOne hour. The ceiling on how long a dead ssh may go unnoticed.
  • sshKeepalive(intervalS, countMax) is the only way to make one. An out-of-range policy throws at the call you wrote, and it is never clamped.
  • The policy reaches every ssh the dial spawns: the arch probe, the cache prefetch, the warm validity check, the GC-root pin, the closure ship, the NIX_SSHOPTS that the remote-store nix build forks its own ssh with, and the agent command.
  • dialAgentOnce takes the same option and forwards it. Its liveness option is forwarded to MakeSessionOptions.liveness. That is a separate passthrough, not the other half of keepalive.
  • The cost is symmetric: a dead host keeps a long-keepalive dial parked for the same window before the loop retries.

What keepalive does not bound

keepalive only answers how long ssh may get no answer before it declares the transport dead. It is not a survival budget for the work riding that ssh. It is one of four independent bounds on how long a link may be silent:

  • On a connected link, the session’s heartbeat decides first (about 25s at its defaults), then keepalive (30s by default).
  • The link’s own keep-alive is a backstop above both. It ends a link that has received nothing for 60s, so 60–65s. A policy raised past it is inert on a connected link.
  • During provisioning, a separate per-step child-lifetime budget bounds the tolerance you request. It is a 30s deadline for a quick probe, an escalating 120s→960s allowance for a required build, and a flat 600s for a speculative copy. ssh keepalives never reset it, since they are protocol traffic and produce no child stdout.

The full breakdown of the four bounds is the keepalive option’s doc comment on SshConnectorOptions (packages/surface-remote/src/sshConnector.ts). packages/surface-remote/src/keepaliveOrdering.test.ts pins its figures against arithmetic. The doc comment is authoritative: if this page and it disagree, it is the one to correct.

sshDialOpts(keepalive)

The ssh options to use when you build your own ssh command outside this package. It is the same set the builders above emit: the dead-peer policy and this package’s ControlMaster decision. That decision is a per-policy socket or an explicit ControlPath=none, never silence.

  • The commands of one dial share a single ssh ControlMaster connection.
  • OpenSSH applies ServerAliveInterval/ServerAliveCountMax from the process that opened the master. Every later ssh on that socket inherits them and ignores its own.
  • So the control socket is keyed by the keepalive policy. Two policies to the same host get two masters, and neither gets the other’s dead-peer behaviour.
  • As a result, a CI-tuned dial and an interactive dial to the same host each keep their own warm connection. ControlPersist is 10 minutes, and spans invocations.
  • Within one dial, every ssh carries the same policy by construction.

unixSocketConnector(opts)

The unix-socket Connector. It dials a daemon that is already listening on a path on this machine, and wires the socket to a typed client.

Options: { socketPath, group, client, describe }.

  • socketPath is a thunk (a function that returns the path), not a string. A daemon’s path comes from its environment, so a path captured once could go stale.
  • The connector owns the socket it dials: it hands it to socketDuplexLink as both halves. The socket’s own close therefore resolves the connection’s closed, and the session notices when its daemon dies. Dialling through unixSocketLink does not give you that.
  • A dial answered with ENOENT or ECONNREFUSED throws EndpointAbsentError. It is a subclass of ConnectError, so every instanceof ConnectError check still matches it. It adds one fact: nothing is serving that path.

dialAgentOnce<S>(opts)

A one-shot dial for a CLI. It returns { client, dispose } with the link already proven live.

  • It validates the baked source-flake ref, probes the host system, resolves only packages.<system>.<package>.drvPath, pins it, and probes the link.
  • It takes the same surface value as sshConnector, and the same required localEnv.
  • package (the closure shipped) and binary (the program run inside it) are both required, and neither defaults to the other. They differ when a host receives more than the daemon: kolu ships padi-agent and runs padi.
  • dispose() tears down only its own session.
  • kaval-tui --host, padi-tui --host and kolu-cli --host use it.

SURFACE_AGENT_FLAKE_REF_ENV / readBakedAgentSource()

The framework-owned handoff from the Nix wrapper, and the only adapter that reads it from the process environment. readBakedAgentSource() returns the normalized exact source, or throws AgentSourceUnbakedError.

resolveBakedAgentDrv(host, packageName, opts)

Resolves an agent from the wrapper-baked source inside a long-lived consumer’s own dial and failure boundary. The consumer never sees the environment handoff.

resolveAgentDrv(host, flakeRef, packageName, { signal, onProgress, budget })

Probes the host’s Nix and lazily resolves one package, inside the owning dial’s cancellation and progress budget. It returns { kind: "flake-installable", drvPath, installable, binaryCache }.

  • The probe is resolveHostNix: the host’s system and Nix version, as a HostNix.
  • The installable lets one Nix process own evaluation through remote realisation, so garbage collection cannot break the handoff.
  • It reads the declaration from the baked source first, before the host probe and before any evaluation. A pre-contract wrapper therefore fails with AgentBinaryCacheUnbakedError, not with whatever transport fault an unreachable host would have raised.
  • Inside a dial, the connector’s bound ctx.resolveAgentDrv also forms the dial’s IndexedClosure. It uses the declaration’s index and rev, the package name and the probed HostNix, and passes it to provisionAgent as published. That is null when the source declares no index, or when the resolver returns a derivation of its own.

AgentDerivation / directAgentDerivation(drvPath, binaryCache)

The nominal provisioning source.

  • Use directAgentDerivation when you own a direct .drv. It validates its arguments.
  • Only resolveAgentDrv constructs the flake-backed arm, so a path cannot be paired with an unrelated installable.
  • Every arm requires an AgentBinaryCache. A provisioning path that ignores the cache cannot be written.

AgentBinaryCache / agentBinaryCache({ substituters, trustedPublicKeys })

Where provisioning may prefetch an agent’s binaries from. It is nominal, like AgentDerivation.

  • The smart constructor is the only way to make one.
  • It rejects an empty or blank declaration, so a value of this type is always validated.

readBakedDeclaration(flakeRef) → Result<BakedDeclaration, AgentBinaryCacheUnbakedError>

Reads and validates the baked source’s binary-cache.json sidecar, and, when that names an index, the commit-hash beside it. Prefer it to writing a declaration by hand.

  • mkProvenAgentSource writes the sidecar from the agent flake’s own nixConfig and its closureIndex.
  • BakedDeclaration is { caches: AgentBinaryCache; index: ClosureIndex; rev: string } | { caches; index: null; rev: null }. Both or neither: an index with no revision cannot be written.
  • A missing or empty commit-hash is the typed error.
  • It uses the same channel as readBakedAgentSource, so the two source-configuration faults chain: readBakedAgentSource().andThen(readBakedDeclaration).map((d) => d.caches).

ClosureIndex

Where a source’s builds are published: { repo, branch, rawBase }, a GitHub branch served raw.

  • binary-cache.nix derives rawBase. Nothing in TypeScript spells an index URL.
  • Files are <rawBase>/<rev>/<system>/<attr>.json and <rawBase>/latest/<system>/<attr>.json.
  • Each file is a PublishedClosure = { rev, ia, ca? }: the full commit, the signed input-addressed path, and its content-addressed rewrite (absent on darwin).

closureIndexUrl(index, key, system, attr)

The URL of one entry.

  • key is { rev } (the publisher’s key, a mkProvenAgentSource source’s commit-hash) or "latest".
  • A segment that could climb out of the index (/, a leading ., empty) throws.

readPublishedClosure(fetchText, index, key, system, attr) → Promise<ClosureLookup>

Looks up one published build with one HTTPS GET. The default fetcher is fetchClosureIndex: fetch, with a 30 s timeout. Tests pass any FetchText.

ResultWhen
{ kind: "published", url, entry }a 200
{ kind: "not-published", url }the index’s own 404: a dev build or an unpublished revision. A fact, not a fault.

Errors: anything else throws a ClosureIndexError naming the URL, with kind: "unreadable" (no answer, another status) or "malformed" (not { rev, ia, ca? }, or a rev-keyed entry naming another revision).

IndexedClosure

The value of ProvisionOptions.published: where one build is published for one host. { index: Pick<ClosureIndex, "rawBase">; rev: string; attr: string; host: HostNix }.

  • It is a plain value. The ssh connector forms it per dial from a baked source.
  • A caller that owns its own .drv (directAgentDerivation) states it.

closureRoute(hostNix, entry) → ClosureRoute

Which form of a published build a host pulls.

RouteWhen
{ kind: "content-addressed", storePath }A Linux Nix that verifies the content-addressed form the publisher writes.
{ kind: "signed", storePath, because: "mach-o" | "nix-line" }macOS always, since a rewrite breaks Mach-O code signatures. Also any other Linux Nix line.
  • The forms are a private table, each named by the first Nix line that writes it: 2.17 (the oldest line that imports a rewrite) and 2.35.
  • 2.35 hashes a self-referencing path differently, so 2.34 and 2.35 refuse each other’s rewrite: ca hash mismatch importing path.
  • A Nix verifies only its own line’s form. Publishers rewrite with 2.34.8, so hosts on 2.17–2.34 pull ca.
  • The sender’s own staging (deliverClosure) reads the same table.
  • A Linux entry with no ca on the publisher’s form throws: the publication is broken.
  • signedBecause(because, hostNix) words the reason for a refusal message.

isDarwinSystem(system) → boolean

Whether a Nix system tuple names macOS (*-darwin). closureRoute and closure staging use this one test to take the signed, input-addressed form, because a Mach-O rewritten to content-addressed loses its code signature. Call it to word a refusal by route rather than spelling the suffix.

Connector / Connection / ConnectError

The transport seam. A Connector dials once and returns a Connection, or rejects with a classified ConnectError. Write your own for a non-ssh transport.

  • A Connection may carry an optional processAlive same-box life-oracle. A local or stdio arm supplies it, and ssh omits it. See the liveness watchdog under Lifecycle invariants.
  • ConnectError takes a fourth, optional anomaly: unknown (default null). It is the app’s own typed verdict for the failure, carried opaquely from the connector that classified it to the binder that renders it.
  • A binder therefore never parses message to tell “this daemon speaks a previous protocol epoch” from “this host is down”.

EndpointAbsentError

A connector’s verdict that nothing is serving there: absent rather than broken. It puts the session in its quiet absent arm instead of escalating toward failed.

  • Only a connector that can tell absence apart may throw it. A unix-path connect(2) answers ENOENT (no socket file) or ECONNREFUSED (a stale file nobody is bound to), and both mean the daemon is not here.
  • A websocket dial cannot tell, since it cannot distinguish absence from a network that swallowed the SYN.
  • It is exported so a consumer tests for it by class, not by its message.

Admit / AdmitVerdict

An optional per-dial supervision hook. Its verdicts are adopt, refuse(state) and replaced(reason). Omit it and every connection is adopted.

The mirror pump

pumpRemoteSurface is the consume side of makeSession. It folds each successive client’s frames into your sink until the link dies, then waits for the next spawn. It pins nothing, so its first wait may legitimately be parked before any pin: reServeSurface decides by demand who holds the link open.

ExportSubpathWhat it is
pumpRemoteSurface(opts).Mirrors each spawn into a fresh sink until that link dies.
SurfaceSink / makeSink.The sink role, and the factory you supply for it.
reServeSurface(opts).Re-serves one binding downstream, on demand, under a policy.
makeClientCursor(session).The raw-loop primitive: wait for the next fresh client.

pumpRemoteSurface(opts)

Per spawn, it runs mirrorRemoteSurface into a makeSink() until that link dies, then awaits the next spawn. It pins nothing.

Options: { source, session, makeSink, liveProcedures?, liveClient?, onLinkDown?, connection?, signal?, log?, onFault? }.

  • signal stops the pump. It aborts the active spawn’s mirror, whose per-key pumps settle via signal.reason, and it never destroys the caller-owned session.
  • makeSink takes { seq }, the spawn counter, not a client. It is rebuilt every spawn, so no per-client state survives a reconnect.
  • onLinkDown fires after holders clear. It is the cue to drop any per-link local fold, so the next spawn rebuilds from a fresh snapshot.

log and onFault

Two channels, with two levels. log is chatter (reconnects, link ends), so filter it freely, at debug if you like. onFault is a structured MirrorFault ({ label, err, scope: "member" | "key" }) and must be wired at error level.

  • A member-scoped fault also rejects the mirror’s done.
  • A key-scoped fault is deliberately key-local and never reaches done, so onFault is its only notice.
  • Both are forwarded verbatim from reServeSurface through the pump into mirrorRemoteSurface.
  • Why: Why faults get a channel of their own.

How the pump ends

  • A rejecting mirror.done (a member’s upstream stream faulting, or your sink throwing) is not reconnected: re-mirroring over a link that is still up would rebuild exactly the mirror that just failed. The pump narrates the death on log and rethrows.
  • That rejection reaches reServeSurface’s done, where the consumer’s policy lives. kolu-server exits for the default host and retires a guest. The reconnect loop still owns the case it is for, a link that ended cleanly.
  • A mirror may also end while the link is still alive, when every subscription has settled and the far end closed its streams. No spawn is coming, so the wait for the next client would park forever serving nothing.
  • Each mirror end therefore races that wait against the framework-reserved system/live round-trip on the client that just ended. A far end that still answers ends the pump, and its done resolves.
  • A dead transport, or a probe that never answers, leaves the ordinary reconnect wait in charge. The session’s own liveness watchdog owns the half-open case.

SurfaceSink / makeSink

The sink role and its caller-supplied factory. Its members receive folded frames: cells.<n>.set(v), collections.<n>.{upsert,remove}.

  • A collection that declares the deltas verb is mirrored through its single batched snapshot-then-delta stream, folded into the same { upsert, remove } sink: one protocol across the wire.
  • A collection without the verb keeps the per-key keys+get fan-out.
  • The choice is read from the spec, never probed on the link.

reServeSurface(opts)

Policy-driven re-serve for one binding. It owns the pump, counts downstream demand, and returns a supervised { surface, group, handlers, done, close }, the same value pair implementSurface returns, so a host merges it into its own served surface with nothing to re-prefix.

Supervision and close()

  • close() aborts its pump via that signal and releases its runtime’s owned sources. It is idempotent.
  • done also rejects on an owned runtime fault.
  • The pump is a terminal source: it ends on its own when the session is destroyed. Supervision routes through @kolu/surface’s superviseTerminalSource(runtime, terminal), the framework combinator that pairs a passive runtime with a terminal driver.
  • The terminal close is attempted first, and the runtime close is always attempted afterward. One close failure is rethrown unchanged; several are surfaced together as an AggregateError.

policy

policy (Record<member, "value" | "delta">) splits members by how they survive an upstream drop.

  • A value member (cell, collection, value pulse) is held open and replayed across the drop.
  • A delta member (byte or liveness stream) fails through. A mid-stream upstream drop ends the downstream stream with SurfaceRelayTransportLost (@kolu/surface/errors), the one retryable framework error the shared retry fence accepts, so the client re-subscribes end to end and a snapshot only ever leads a fresh stream.
  • Both ends were built from that error’s class, which is what lets its _tag survive the parent’s decode-and-re-encode. A raw re-raise would reach the downstream as an opaque, non-retriable failure and strand it.
  • A subscribe before any live spawn is up waits for the next spawn, on the same pre-connect arm as hold-open, rather than raising that retryable end every second.
  • Direct callers of relayFailThroughStream (and its hold-open sibling) must pass an ObservableHolder from observableHolder(). The pre-bind wait needs whenChanged(), which a plain LiveSpawnHolder does not expose.
  • Holding open a byte stream is a compile error.

Demand gating

Every downstream call enters through one of reServeSurface’s own handlers, so that is where demand is counted: one gate for cells, collections, streams, events and procedures alike.

  • The session is pinned on the crossing from nobody to somebody, and released on the way back.
  • The release waits a fixed grace (RELEASE_GRACE_MS, 30s), so a consumer whose subscriptions re-key crosses zero for free.
  • An upstream nobody is subscribed to is therefore never dialled, which is the point for an upstream that is optional, such as a daemon most machines do not run.
  • A parent that wants an always-on link pins the session itself, and the count never reaches zero. kolu-server does that per host. A host’s link state is itself something the fleet shows, and a host dialled only once a browser subscribed would read connecting while nobody looked at it.
  • Why: Why demand is counted in one place.

absent frames

The optional absent (Record<member, frame>) names, per value member, the frame a subscriber gets while the upstream is absent. It tells a hold-open subscriber that there is nothing there.

  • A frame is sent once per absence episode, and replaced by the next live frame.
  • The frame is a value the source contract declares (odu’s CHECKOUT_NONE), never one this relay invents.
  • Absence is read off the session’s own absent arm, not off the client holder going null, which also means “between spawns”. Only a session whose connector can tell absence from unreachability ever sends one, and every other re-serve omits the option.
  • Naming a member that is not a value-classified stream or event throws at build.
  • Why: Why absence gets a frame of its own.

A standing refusal

A daemon that answers and speaks a contract this build cannot read is a refusal, the opposite fact, and never an absent frame. The admit verdict’s standing refusal is published on the session itself: session.refusal(), the sentence or null.

  • A hold-open relay reading it stops holding and fails its subscription with SurfaceUpstreamRefused (@kolu/surface/errors), carrying that sentence.
  • It is raised as a defect, because a streaming member declares no error channel, and a defect is never retried.
  • Every hold-open relay gets the signal, whether or not its member declares an absent frame.
  • Why: Why a refusal stops the relay.

makeClientCursor(session)

A stateful cursor, and the raw-loop primitive under the pump. cursor.next(signal?) blocks until a fresh client arrives after a reconnect, or wakes early when signal aborts.

The fleet fan-out

Two exports turn many sessions into one fleet: a keyed pool of sessions, and the adapter that publishes them as a host map.

ExportSubpathWhat it is
buildRemotePool(opts).The keyed pool of sessions a ?host= dispatcher reads.
serveHostMap(map, pool, opts).Publishes that pool as a @kolu/surface-map registry.

buildRemotePool(opts)

A keyed Map<host, { session, handler }> that a ?host= dispatcher reads. Its verbs are add/remove/retire/reconnect/recheckAll, plus per-host socket eviction.

Options: { buildEntry, persist?, controls? }. buildEntry(host) returns the host’s { session, handler }.

remove(host) and retire(host)

Both share the same teardown mechanics: evict sockets, drop membership, destroy the session. They differ in two ways.

  • remove is the user’s departure. It calls the persist hook first.
  • retire is an internal shed, a dead session the pool sheds on its own. It does not persist, so a membership store re-seeds the host next boot.
  • Each verb labels its own WebSocket close reason and log line: host removed or host retired.
  • A pool with no persist hook makes them behave the same on disk, since neither writes. The close reason and log still name the verb.

serveHostMap(map, pool, opts)

Adapts a pool into a @kolu/surface-map MapRegistry. It fuses membership with each session’s onState and projects each state into an entry status, reading the clockOffset the session’s own connected arm carries.

Options: { linkFor, failureOf, connection? }.

  • linkFor(host, session) is built once per host, and evicted on either removal or retirement, since both drop live membership.
  • There is no offsetOf option. makeSession measures the far-end clock offset off the framework-reserved system.clockNow at admit and carries it on the session’s own connected state, which serveHostMap reads directly.
  • Readiness is link-liveness. The entry is connected the moment its link is live, carrying clockOffset: null through (honestly not yet measured) until the first probe stamps it, and it is never demoted to connecting.
  • Together with buildRemotePool it is the server half of a host map. See How to serve a map.

failureOf(host, session, state)

Required and total. It classifies a down session, and the registry publishes what it returns.

It returnsFor
nullA transient down state: keep the entry warming. One meaning only, a classification verdict, never a fabrication.
A schema-valid domain FailureA standing down state.
  • A terminal failed state that yields null fails loud with UnclassifiedHostFailureError: a failed entry cannot exist without a domain failure.
  • The distinct member-with-no-session (unknown-host) race fails loud with UnclassifiedHostSessionError.
  • There is no "other" fallback cause.
  • It is invoked fresh on every resolve, potentially once per member per sibling’s tick, with nothing held or cached against a frame.
  • That is safe because @kolu/surface-map’s republish gate compares the published status structurally (dequal). A classifier that mints a fresh-but-equal literal per call, as every real one does, is suppressed by the gate rather than by an adapter-side hold this module would have to keep in sync with it.

The failure’s evidence

At the same classification seam, serveHostMap staples the session frame’s retained log tail, the very frame failureOf just classified, onto the published failure record as evidence (see @kolu/surface-map’s FailureEvidence).

  • Both down arms get it: a terminal failed, and a disconnected the classifier called a standing refuse, which publishes as failed too.
  • A transient disconnected publishes warming and carries neither a reason nor evidence.
  • SessionState.log is structurally identical to FailureEvidence, so raw.log is passed straight through. There is no injected evidence hook and no second evidence pipe.
  • The tail is pinned at classification, a post-mortem record rather than a live view, so it survives the client’s liveness floor with the reason it belongs to. The live connection word is dropped over a dead link.

The fail-loud seams

Both unclassified errors are genuine producer defects, so they crash the process even when raised deep in the internal per-member republish.

  • The loop catches synchronously, so one member’s defect never freezes a sibling’s status stream mid-frame.
  • It then rethrows out-of-band on the next microtask, as an uncaught exception, never a swallowed console.error that would let the server degrade to a stale-but-healthy status.

Connection health

Connection health is state the remote agent cannot report, since it cannot observe the link to itself, so the parent authors it. It rides the host map’s entry as a fine connection payload: the one authority the coarse dot (EntryStatus.kind) and the fine word both derive from.

ExportSubpathWhat it is
ConnectionInfo / ConnectionInfoSchema / DEFAULT_CONNECTION/connectionThe browser-safe type, its wire schema, and the pending value.
projectConnection(s)/connectionThe pure leaf that turns a SessionState frame into one.
sessionConnection(raw)/connectionThe total projection a host map feeds to serveHostMap.
ConnectPhase/connectionThe up-but-not-yet-connected phases a progress UI narrates.
LogEntry / LogEntrySchema/connectionOne { source, line } entry of the log tail.

ConnectionInfo / ConnectionInfoSchema / DEFAULT_CONNECTION

The browser-safe connection type, the wire schema the host map validates an entry’s connection payload against, and DEFAULT_CONNECTION, the gate-closed pending value. All three ship from @kolu/surface-remote/connection.

ConnectionInfo is the discriminated mirror of the session’s SessionState sum minus one arm (Exclude<SessionState<SshProv>, { phase: "absent" }>), keyed on phase.

ArmWhat it says
probingThe warm-check window: the agent is looked up, and nothing is shipped yet.
provisioningThe ssh connector’s provisioning phase.
connectingThe dial is up but not yet connected.
connectedThe link is live. It alone carries clockOffset.
disconnectedDown. It adds error and cause ("network" transport-class, "remote" reached-and-refused).
failedTerminal. It carries cause: "network" | "remote".
  • absent is excluded, because only a connector that can tell absence from unreachability produces it. A host is dialled over ssh or over the local endpoint, and neither can tell.
  • The transport class is orthogonal to terminality. A MAX_CONSECUTIVE_FAILURES remote rejection gives up "remote", while a budget-exhausted silent provisioning step gives up "network".

Fields on every arm

FieldWhat it carries
logThe provenance-tagged tail: { source: "local" | "remote"; line }[].
sinceMsThe server-stamped elapsed of the current episode, in milliseconds.
campaignEpochThe monotonic campaign generation.
  • sinceMs is a duration on the server’s single clock, never a foreign epoch the browser would subtract. The browser extends it smoothly with its own ticker.
  • campaignEpoch lets a client’s elapsed timer re-baseline when the same host starts a fresh campaign that reopens at sinceMs: 0 after a quiet multi-minute stretch.
  • There is no separate progressLines or remoteProgressLines. Provenance is the source field.

What a campaign scopes

Both log and sinceMs are scoped to the current campaign. They are stamped and reset at campaign birth: the first (or re-pin) dial, a user reconnect() or recheck(), and a drop of a connected link.

  • They are not reset on a backoff retry within an ongoing connect campaign.
  • So a wedged provision that retries for ten minutes reads ten minutes, not the per-attempt second or so.
  • The log survives the retry dials (capped), so a coming-up card can render the attempt tail.

clockOffset on the connected arm

clockOffset: number | null is the far-end host’s wall-clock offset in milliseconds against the serving process. It is measured off the framework-reserved system.clockNow at admit, round-trip compensated (measureSurfaceClockOffset in @kolu/surface/clock-now).

  • It is born null and re-stamped once the probe lands. null means not yet measured.
  • Readiness is link-liveness, so serveHostMap keeps the entry connected with a null offset. A 0 placeholder would be a lie on a skewed host.
  • This is where the clockOffset a keyed SurfaceMap folds into EntryStatus.connected originates. It rides the session’s own state, not an injected offsetOf.

When the clock probe fails

A clock probe that fails is never swallowed. The session stays connected with a null offset, re-attempts on a fixed cadence, and surfaces the failure on its diagnostic logger, MakeSessionOptions.log.

  • That logger is the @kolu/log Logger shape: four level methods debug/info/warn/error, each (obj: Record<string, unknown>, msg: string) => void. Pass a pino child directly.
  • The session dispatches the severity itself, with a receiver-bound log[severity]({ line }, label) call. Consumers never dispatch severity over extracted method references, so the unbound-this crash class has no spellable form.
  • A genuine failure (the wedged-clock deadline, a transport fault) is classified error.
  • The expected-absent case is classified debug: an agent dial whose client carries no reserved system.clockNow, which means the tool is not installed rather than that something failed.
  • Omit log and diagnostic lines fall to raw process.stderr, the plain-CLI default.
  • A throwing logger crashes the session loop, deliberately: fail fast.

projectConnection(s)

The pure, browser-safe projectConnection leaf a consumer applies to a SessionState frame, or to the entry’s frame, to get a ConnectionInfo. It is the one named projection both the server and a client use.

  • It is a near-identity. Every arm but one passes through untouched, since ConnectionInfo is SessionState<SshProv> minus absent.
  • absent folds to disconnected, carrying the absence’s own sentence. It is folded rather than cast away, because a sum with an unhandled arm is how a future connector’s new state reaches a browser as undefined.
  • The server reaches it through sessionConnection, which is serveHostMap’s connection.project. There is never a second server-side subscription.
  • serveHostMap produces the coarse status and the fine connection from the same SessionState frame in one resolve, through its injected connection: { project, isConnected }. project yields the word.
  • serveHostMap asserts that the coarse dot and isConnected(word) agree, and fails loud before publication when they do not.
  • A consumer reads the word off the same entry it reads the dot from: map.entry(host).state().connection, never a second subscription.

sessionConnection(raw)

The total (raw: SessionState<string> | undefined) → ConnectionInfo projection a host map feeds to serveHostMap’s connection.project.

  • For a not-yet-seeded member it folds the gate-closed pending value.
  • It validates the erased frame against ConnectionInfoSchema, failing loud on a non-ssh or malformed frame.
  • Otherwise it returns the frame, reference-stable.

ConnectPhase

The up-but-not-yet-connected phase subset a connect or progress UI narrates: Exclude<ConnectionInfo["phase"], "connected" | "disconnected" | "failed">, that is SshProv | "connecting".

  • It is the one vocabulary a UI’s exhaustive switch or Record keys on, so a new provisioning phase fails to compile until it is handled.

LogEntry / LogEntrySchema

One { source, line } entry of the log tail. They are aliases of @kolu/surface-map’s EvidenceLine and EvidenceLineSchema: the same concept, so the same type and the same validator.

  • They are taken from the schema-only @kolu/surface-map/evidence leaf, so this browser-safe module still pulls in no group-assembly code.

A mirror, end to end

const src = base;
const runtime = implementSurface(src, {
  cells: {
    load: { store: inMemoryStore(DEFAULT_LOAD) },
  },
  collections: {
    processes: {
      readAll: () => new Map<Pid, Proc>(),
      upsert: () => {},
      remove: () => {},
    },
  },
});

void session.pin();
void pumpRemoteSurface({
  source: src,
  session,
  makeSink: ({ seq: _seq }) => ({
    cells: { load: (v) => runtime.ctx.cells.load.set(v) },
    collections: { processes: { upsert: () => {}, remove: () => {} } },
  }),
});

Lifecycle invariants

What a session guarantees about its own state, its children and its retries, and the provisioning steps a dial runs.

Export or topicSubpathWhat it is
onState.Snapshot, then every later transition.
currentState().The synchronous twin of onState, and the liveness read.
currentClient().The dialling-or-connected client promise. Not a liveness gate.
pin() / release() / destroy().Intent to stay connected, and giving it back.
Timers and the process lifetime.Which of a session’s timers can hold a process open.
The dial’s process group.What a connector teardown kills.
Fault classes and the give-up budget.Which faults retry, which are bounded, which end it.
Absence.The steady state a missing endpoint settles in.
A host with no runnable Nix.The terminal probe verdicts, by exit code.
An ssh gate that cannot be answered.Credential and host-key refusals, terminal by design.
recheck() and reconnect().The two manual cycles, and what each disturbs.
nudge().Fire a scheduled reconnect now, and nothing else.
runCapture and LifetimePolicy.The bounded lifetime every provisioning child runs under.
The pre-connected liveness backstop.The baked bound on a campaign that says nothing.
provisionAgent(opts) → ProvisionResult.Realise an AgentDerivation on a host.
resolveHostNix(host, opts) → Promise<HostNix>.The one-round-trip host probe every dial opens with.
deliverClosure(opts) → Promise<DeliverClosureResult>.Get one exact store path into a host’s store and root it.
pullClosure(opts) → Promise<PullClosureResult>.Have the host pull a published path straight from a cache.
The liveness watchdog.What a silent connected link means, per arm.
State progression.The phase order, and the backoff on a drop.
A mirror never fabricates a value.Why a mirrored cell serves nothing before its first fold.

onState

Snapshot, then deltas. A listener attached at any point sees the current state synchronously before any later transition, the same contract useCell consumers expect.

currentState()

The synchronous twin of onState. It returns the current published SessionState frame, the point-read companion to the snapshot-then-delta subscription, reading the same cell onState publishes.

  • It always returns the freshest frame.
  • onState’s delta delivery is microtask-deferred, while the initial snapshot fires synchronously.
  • One synchronous frame can drive two transitions (disconnected to a give-up failed), so a listener delivered one delta frame may read a later one here.
  • Honest liveness is currentState().phase === "connected".

currentClient()

The in-flight or current client promise. It is not a liveness gate.

  • It is legitimately non-null while merely dialling: during connecting and provisioning, and across an entire reconnect backoff window, because scheduleReconnect retains the rejected dial.
  • The pump and markConnected handshake depend on that dialling-or-connected meaning.
  • So currentClient() !== null never means the far end is live. Read currentState().phase for that.

pin() / release() / destroy()

pin() is intent, and release() gives it back.

  • pin() bumps the ref-count unconditionally, so the session keeps reconnecting even if the first spawn fails, and resolves with the first client.
  • release() decrements the same count. At zero the session lets its connection go after a fixed grace (RELEASE_GRACE_MS, 30s), then re-arms on the next pin() with a fresh dial.
  • The grace is a constant, never an option: a consumer whose subscriptions re-key drops to zero pins and back within one tick, and the grace makes that crossing free.
  • destroy() drops the pin regardless of the count.

Timers and the process lifetime

A session’s internal timers never hold the process alive. The reconnect backoff, the connect watchdog and the clock probes are unref()’d, so a pinned but abandoned session with a down link cannot keep an otherwise-finished process running.

  • A live link is different: its ssh child or socket is a real event-loop hold while it lasts.
  • A consumer whose sole purpose is waiting on a session must hold the process by its own means: a server socket, stdin, a live transport.
  • So a parked ClientCursor.next() or onState-derived wait is not a process hold. Across a reconnect gap it is settled only by the backoff firing, and if nothing else holds the loop it goes silently unsettled as the process exits.
  • The one deliberately ref’d timer is the admit-handshake timeout. It must fire even as the last handle standing, because it rejects a pin() a caller is awaiting.

The dial’s process group

An agent dial owns its whole process group. The long-lived agent transport starts in a fresh POSIX process group, and every connector teardown signals that exact group: SIGTERM, then SIGKILL after a short grace.

  • That includes an ssh client’s ProxyCommand shell and any nested ssh it starts, so a failed dial that retries leaves no subprocess behind.
  • The same private ownership primitive backs runCapture’s bounded provisioning children. Session policy and one-shot lifetime policy remain separate consumers of it.

Fault classes and the give-up budget

The budget is a classified ledger (@kolu/surface’s makeFailureLedger), one run per failure class, so no class can spend another’s.

FaultWhat it does
networkTransport-class: the host is unreachable, or a transport wedged or was killed. Retries at capped backoff (60s).
remoteThe host answered and the attempt still failed: a Nix build error, an agent that would not start. Bounded.
  • A network fault does not itself count toward give-up, and it resets the remote run. An unreachable gap means the host went away, so the next remote blip is fresh evidence of a sleeping host.
  • A remote fault is bounded by MAX_CONSECUTIVE_FAILURES consecutive remote failures. It then surfaces terminal failed plus remote, with a give-up line derived from the ledger’s verdict, so it can only name the run that actually tripped, followed by the last failure’s own reason, never a guessed remedy.
  • Each retry line says what the attempt ran into (attempt N failed: <reason> — retrying in …) rather than a fixed label. network is a retry class, not proof the host is gone.
  • An agent dial’s exit 255 is transport-failed only when ssh’s stderr named a transport failure, read as whole lines, with a short bounded wait for that line after the exit. A remote agent that exits 255 by itself is an ordinary exit.
  • One network fault does terminate: a budget-exhausted silent provisioning step. The ssh connector’s per-step progress-liveness budget gives up after N consecutive silent kills, surfacing terminal failed plus network, the honest transport cause. Terminality is the phase, not a rewrite of the cause.
  • Why: Why the session sorts silence into kinds.

Absence

Absence is a steady state, not a failure. A connector that throws EndpointAbsentError puts the session in its absent arm, carrying the dial’s own sentence as error and no cause, because nothing failed: the endpoint is simply not there.

  • It records nothing in the give-up ledger, so an absence never ages into a terminal failed.
  • It re-probes at the reconnect ladder’s own ceiling rather than ramping, since nobody’s daemon gets started by polling for it.
  • It publishes at debug level. A line per probe at info would fill an operator’s log with the news that software they do not run is not running.
  • It is never terminal, so the next pin() on a daemon that has since started connects normally.
  • Why: Why the session sorts silence into kinds.

A host with no runnable Nix

Provisioning realises the agent with the host’s own Nix, so a probe whose shell exits 127 (the POSIX “command not found” for nix-instantiate) is terminal.

RejectionWhen
{ kind: "nix-unavailable", failureCause: "remote", terminal: true }The probe shell exited 127.
{ kind: "unavailable", failureCause: "remote", terminal: false }nix-instantiate ran and failed (non-127, and ssh reported no transport failure). It carries Nix’s own error.
  • The 127 case is classified by the exit code, never by the shell’s wording: bash, dash and fish word it three different ways.
  • The non-127 case is retried, but bounded, because the host answered. Probe output that is not a nix-system string rejects the same way, commonly a shell startup file printing to stdout.
  • Our own kills and a transient local spawn fault stay untyped retried errors.
  • Why: Why the session sorts silence into kinds.

An ssh gate that cannot be answered

Every ssh this package spawns is non-interactive (BatchMode=yes, no TTY), so a host that demands a credential or an unverified host key does not prompt. ssh declines and exits 255.

  • The arch probe every dial opens with reads those two refusals off ssh’s stderr and rejects a ResolveDrvError whose resolution is { kind: "auth-refused" | "host-key-unverified", failureCause: "remote", terminal: true }.
  • It is remote because the peer was reached and refused us, and terminal because no redial can supply the missing answer. So a password-only host settles on failed with an actionable reason in seconds.
  • Classification is conservative. It requires ssh’s own exit 255 as well as the stderr text, since a remote command’s stderr rides the same stream.
  • Any refusal it does not recognize stays an untyped, retried transport error.
  • The probe is the natural chokepoint: a refusal met later (Nix’s own ssh fork, the agent dial) kills that dial, and the redial’s probe meets it here.

recheck() and reconnect()

Two manual cycles, with different reach.

CallWhat it does
reconnect()Re-arms a failed or idle session, and will not disturb a live link. The manual button.
recheck()Force-cycles whatever is there, including a connected link and an in-flight dial. The wake or network-change companion.
  • recheck() suits a post-sleep link, which is often stale.
  • On an in-flight dial, recheck() aborts that dial (the connector’s ctx.signal fires, group-killing any provisioning child) and redials now, with no backoff.
  • Every recheck() and reconnect() is a fresh campaign: fresh sinceMs and log, fresh connector budgets.

nudge()

A scheduled reconnect fires now. Every other state is a no-op.

  • It is the signal a consumer sends when the world plausibly changed under a down session. kolu-server fires it on every websocket client accept, because a waking laptop’s first observable act is its browser reconnecting. The alternative is up to a full capped backoff (60s) of dead remote panes.
  • It is the narrow verb, not a second spelling of recheck(). It fires the same attempt the backoff had already scheduled, keeping its attempt number and its position in the give-up budget, and records nothing in the failure ledger.
  • recheck() instead calls the ledger’s success(), refilling every class’s run, and stamps a fresh campaign. A permanently-broken host nudged often enough would otherwise never reach its terminal verdict.
  • It adds no unboundedness: it only fast-forwards an already-armed timer, so it cannot create an attempt the existing schedule would not have made.
  • The fired-now path narrates on the session log: nudged — firing the scheduled retry now … give-up budget unchanged.

The three no-op arms

StateWhy nothing happens
A live linkNothing is scheduled to fast-forward, and a client connecting is no evidence this link is stale. Cycling it is recheck()’s job.
An in-flight dialThe probe is already running. This no-op is the wake-storm coalescing, with no lock of its own.
Terminal failedReviving a spent budget is reconnect()’s job, never an ambient signal.

runCapture and LifetimePolicy

The fire-and-collect helper (/ root) that provisioning children run under. It takes a required LifetimePolicy.

PolicyFor
{ kind: "deadline"; ms }A quick step.
{ kind: "progress-liveness"; silenceMs }The minutes-long provision: killed only on real output-silence, never a healthy slow transfer.
  • Any output resets that budget, unless the caller passes isActivity(line) → boolean for a structured stderr that also carries heartbeats. Then only stdout and the lines it answers true for count.
  • A kill settles the distinct ExitResult arm lifetime-expired (retryable); a user abort settles aborted. Both group-kill the child (and its ssh grandchild) and settle at the kill, never awaiting a close that may never fire.
  • The exported runCapture is location-blind and gets only the local group-kill. A caller that hands it an ssh argv of its own does not get the remote kill below.

ConnectContext and remote kills

ConnectContext carries a per-dial signal (the abort) and a campaignEpoch (fresh connector budgets per campaign), plus activity().

  • activity() is a liveness signal with no log line. It resets the pre-connected backstop exactly as a progress line does.
  • A connector calls it for child output it deliberately does not narrate, so the backstop keeps dominating the child’s own silence bound.
  • The ssh connector feeds it every activity line of a long-running provisioning Nix command, through ProvisionOptions.onActivity (required). That is every stderr line except a Nix progress result whose done counter has not moved, which Nix re-sends about once a second even while a transfer is stalled.
  • Every one-shot command this package runs on a remote host (the provisioning Nix steps, the warm check, the arch probe) goes through one internal runner that wraps it to report its PID first. A run that ends aborted, or killed by its own lifetime policy, is SIGKILLed on the host by that PID before the step returns, so a retry never queues behind the abandoned one.
  • That remote kill is needed because killing the local ssh does not reach a tty-less remote command, and a Nix blocked in a socket write survives SIGTERM.
  • No provisioning Nix command passes -v. At that level a daemon with auto-optimise-store forwards a line per unpacked file to a client that is busy writing the NAR, and the two deadlock.

The pre-connected liveness backstop

While a campaign is coming up, if no progress line arrives and no phase advances for the baked DEFAULT_PRE_CONNECTED_LIVENESS_MS (20min), the session cycles the attempt (abort and redial), narrated.

  • The bound is not configurable. It must stay above the connector’s maximum step-silence budget so the per-step budget always ends the step first; a lower bound would let the backstop reset the budget and loop a silent copy forever.
  • It is liveness, so a chatty build resets it every line, and it is never capped.
  • It guards the seam the per-child policies cannot: a wedge in a non-helper await such as resolveDrvPath.
  • Why: Why the session sorts silence into kinds.

provisionAgent(opts) → ProvisionResult

Realise an AgentDerivation on a host (/ root). It is the ssh connector’s provisioning step, usable standalone.

  • A flake-backed derivation is evaluated locally and provisioned into the remote ssh-ng store by one nix build, so Nix owns the temporary roots across evaluation, transfer and realisation. A parent on darwin can front a target on linux.
  • There is no tarball, Docker or prebuilt fallback.
  • A warm host short-circuits through an ask-only check: a sender-local nix-store -q --outputs, then a bounded nix-store --check-validity on the host, never a substituting --realise.
  • Every path succeeds only after nix-store --realise --add-root commits the output behind its stable target-store GC root.
  • agentPath is the path delivery rooted, the content-addressed one when it travelled content-addressed.

Route order

  1. The warm check.
  2. The index step, when published is non-null and the host is remote. A localhost staging reads its own store first.
  3. Delivery of the agent’s output closure, before the cold build.
  4. The cold build on the host, with its own substituters.
  • The index step is readPublishedClosure for published.rev, closureRoute for published.host, and pullClosure on the host from each declared substituter in turn, rooted at the agent’s GC root under the dial’s budgets.provisioning.
  • A landed index step returns there: no rewrite, prefetch, ship or build, and no trust on the host for a content-addressed path.
  • not-published, an unreachable index, a pull no cache would give (the signed path to a host that does not trust it), or a pull that went silent past the copy bound each narrate and move on.
  • A malformed entry fails the provision (cause: "remote").

Delivering the closure

Delivery never builds. It sends the closure in the form the target takes, decided once the warm check has missed, by the target’s system (read off the .drv with nix derivation show) and, for Linux, by three Nix versions asked fresh each delivery.

Nix askedHow
The binder’s nixnix --version
The binder’s storenix store info --json, its daemon’s version
The target’s storenix store info --json --store ssh-ng://<host>; a local target’s store is the binder’s
  • To a Linux target whose three Nixes all handle the same content-addressed form, the closure is rewritten content-addressed in the binder’s store and shipped with nix copy --to ssh-ng://<host>. Any Nix on that form accepts it with no trusted-users entry and no key.
  • The rewrite is nix store make-content-addressed, from the local store when it holds the closure, else from each of the derivation’s binaryCache substituters in order.
  • To a darwin target, a system nobody named, or a Linux target whose Nixes disagree on the form, the signed input-addressed closure is prefetched into the binder’s store (nix copy --from) and shipped with nix copy --no-check-sigs --to ssh-ng://<host>. The disagreement is narrated: sending the closure as built: …, naming the versions.
  • Without --no-check-sigs the copy asks for signature checks even from a trusted user, so a locally-built closure never lands.
  • Rewriting a Mach-O breaks its code signature, and no rewrite is attempted that a store is known to refuse.
  • A version probe that cannot answer stops the delivery at its own seat.

Where the rewrite is rooted

  • A Linux target’s rewrite is rooted in the binder’s store per output path and form (~/.local/state/kolu/surface-remote/content-addressed/<name>/<hash>@<form>). That record is what a later dial, after a restart too, asks the host about first, and it is reused only while its form is still the route’s.
  • The staging and the ship are both load-bearing. The local store is the only seat where a declared cache can act, since a remote-store realisation substitutes with the remote daemon’s own nix.conf.
  • The cold build itself transfers only the derivation closure: locally-valid outputs are never consulted by the remote daemon.
  • An as-built ship lands only when the target trusts it: a trusted-users ssh user, or signatures the host trusts.
  • A miss, a signature refusal, or a rejected ship each narrate Nix’s own error into the progress tail and fall back to realising on the host with its own substituters, never failing the dial.

Options and result

ProvisionOptions: { host, derivation, published, onProgress, onActivity, onProvisioning?, budgets, keepalive, signal? }.

published: IndexedClosure | null is required: null states “no index”.

OptionMeaning
onActivityReceives every activity line of the long-running Nix steps, narrated or filtered. Wire it to ctx.activity.
budgetsA connector-owned ProvisionBudgets (makeProvisionBudgets()). Its evaluation and provisioning progress-liveness budgets persist across a campaign’s retry dials.
signalThe abort.

ProvisionResult is { ok: true; agentPath } or { ok: false; reason; cause: "network" | "remote"; terminal? }.

  • terminal marks a budget-exhausted silent step that must give up at once. It is decoupled from MAX_CONSECUTIVE_FAILURES, so the backstop cannot reset it away.

How a failure is classified

Every Nix command provisioning runs uses --log-format internal-json, and is read by what Nix tagged.

  • A failure’s reason is Nix’s first error plus the failed build’s last log line. The dependents’ “1 dependency failed” cascade is fallout, and the root error is re-narrated last so a bounded tail keeps it.
  • A failure is classified network only on what can speak for the connection: ssh’s own untagged stderr, or the headline of Nix’s own error (a failed ssh connection, an unreachable download).
  • It is never classified from a builder’s log, nor from Nix quoting one.

resolveHostNix(host, opts) → Promise<HostNix>

The host probe every dial opens with (/ root), with the full signature resolveHostNix(host, opts: ResolveSystemOptions) → Promise<HostNix>.

  • It is one nix-instantiate --eval --strict --json round-trip: { system = builtins.currentSystem; nixVersion = builtins.nixVersion; }.
  • It is classified exactly as A host with no runnable Nix and An ssh gate that cannot be answered describe.
  • HostNix is { system, nixVersion }. There is no resolveSystem export; a resolver inside a dial keeps ctx.resolveSystem(), which is this probe’s system half.
  • The version matters for a content-addressed path: one is only verifiable within the Nix line that wrote it, since Nix 2.35 hashes a self-referencing path differently from 2.34 and neither accepts the other’s. A caller choosing between a published path’s content-addressed and signed forms chooses by it.

deliverClosure(opts) → Promise<DeliverClosureResult>

Get one exact, already-evaluated store path that the caller holds (or a declared cache does) into a host’s store, and root it there (/ root, signature deliverClosure(opts: DeliverClosureOptions)). It is provisionAgent’s delivery step on its own, and never a build.

Its steps are the ask-only warm check, the index step and the staging provisionAgent describes:

  • The warm check is remote only: the content-addressed form this sender rooted for outPath, in whichever form it was written, then outPath itself. A hit is rooted and returned.
  • The index step runs when published is non-null.
  • To a Linux target whose Nix and the sender’s (its nix and its store) all handle the same content-addressed form, the closure is rewritten content-addressed in the sender’s store. It ships with nix copy --to ssh-ng://<host>, which that Nix accepts with no trust. The rewrite reads the local store when it holds the closure, else binaryCache.
  • To darwin, to a system nobody named, and to a Linux target across Nix 2.35 from the sender (or a sender whose nix and store disagree), the input-addressed closure goes as built. It is signed when a cache signed it, and by no one when the caller built it. It is prefetched when absent, and shipped with nix copy --no-check-sigs, which lands only when the host trusts the login.
  • Then comes the required root commit at rootPath.
  • kolu-server ships the profiles it builds for the Agents setting through it.

DeliverClosureOptions

{ host, system, outPath, binaryCache, published, rootPath, onProgress, onActivity, onDelivering?, budget, signal?, keepalive }.

OptionMeaning
systemsystem: () => Promise<string | undefined>, asked only when the warm check misses.
binaryCachebinaryCache: AgentBinaryCache, required even for a path the caller built: delivery is never cache-blind.
publishedIndexedClosure | null.
rootPathAs for pullClosure.
budgetbudget: StepBudget, the root commit’s budget.

DeliverClosureResult

ResultWhat it says
{ ok: true; outPath }The path the target holds: the content-addressed one when it travelled content-addressed.
{ ok: false; kind: "not-delivered"; reason; at: "sender" | "target"; asBuiltBecause: string | null }The closure is not in the target store, and nothing was rooted.
{ ok: false; kind: "failed" } plus ProvisionResult’s failure shapeAn abort, or a failed root commit.

On a not-delivered result:

  • reason is Nix’s words verbatim: a cache miss, a refused rewrite, a refused ship, or a Nix that would not say its version.
  • The arm in full is { ok: false; kind: "not-delivered"; reason; at: "sender" \| "target"; asBuiltBecause: string \| null }.
  • at is the seat it stopped at. sender means this machine never held the closure in the form to ship, because the rewrite, the prefetch, or asking its own Nix failed. target means the ship, or asking the target’s Nix, failed.
  • asBuiltBecause words why it was sent as built (Mach-O, an unnamed system, or the Nix versions that disagree). It is null when it travelled content-addressed, or the route was never decided.

pullClosure(opts) → Promise<PullClosureResult>

Have a host pull one published store path straight from a binary cache and root it (/ root, signature pullClosure(opts: PullClosureOptions)). The caller moves no bytes, and evaluates, rewrites and builds nothing.

Every step runs at the target, over ssh for a remote host and directly for localhost:

  1. An ask-only nix-store --check-validity <storePath>. A hit skips the copy.
  2. Otherwise nix --extra-experimental-features nix-command copy --from <cacheUrl> <storePath>, under the fixed copy bound PROVISION_COPY_SILENCE_MS.
  3. The required nix-store --realise --add-root <rootPath> --indirect commit.
  • The feature flag is a client-side parse gate any login may lift, so a stock host Nix needs no setting.
  • Nix’s per-path events and advancing progress prove a multi-gigabyte download alive, and a stalled transfer’s repeated, unmoved progress does not, so a wedged copy is killed, on the host too.
  • --from names a store to read rather than proposing a substituter, so no trust setting filters it. A content-addressed path is accepted by any daemon with no signature, key or trusted-users entry, so an untrusted login with no nix.conf change works.
  • An input-addressed path is taken only when the host’s Nix trusts its signature, so the caller names the signing key as signedBy (passed as --extra-trusted-public-keys, honoured for a trusted login; an untrusted one needs the key in the host’s trusted-public-keys).
  • Content-addressed is no option for a macOS target: rewriting a Mach-O invalidates its code signature and macOS kills it on launch. Nor for a host whose Nix line differs from the one that wrote the path (resolveHostNix).

PullClosureOptions

{ host, cacheUrl, storePath, signedBy?, rootPath, onProgress, onActivity?, keepalive, signal?, budget }.

OptionMeaning
budgetbudget: StepBudget is required: the root commit’s progress-liveness budget, the caller’s own, so repeated silent commits end terminal. A dial passes budgets.provisioning; a one-off caller makeStepBudget(PROVISION_STEP_SILENCE_BASE_MS, PROVISION_STEP_MAX_EXPIRIES).
signedByOnly for an input-addressed storePath.
rootPathResolved against the ssh login’s home when relative, and it must be absolute for localhost. homeGcRoot(isLocal, rel) forms it.
keepaliveRequired, as for every ssh this package spawns.
  • The root commit runs under a fresh provisioning StepBudget.

PullClosureResult

ResultWhat it says
{ ok: true }storePath is valid in the target store, and rooted.
{ ok: false; kind: "uncopied"; reason }The host’s copy from the cache did not land: not in the cache, no egress, or silent past the copy bound. A miss, like every speculative copy. reason is Nix’s words verbatim.
{ ok: false; kind: "failed" } plus ProvisionResult’s failure shapeA dropped ssh (cause: "network"), an abort, or a failed root commit (terminal once its budget is spent).

The liveness watchdog

Default-on. While connected, the session probes the reserved system.live round-trip on an interval. A rejection still counts as alive, because the round-trip completed. Opt out with liveness: false.

What a timeout does depends on the arm’s structure, through the optional Connection.processAlive same-box oracle:

ArmOn a timeout
Remote (ssh, no oracle)Reads silence as death and force-cycles the child, because the network can die silently.
Local (endpoint or stdio, with processAlive)Consults the same-box process table. Process gone: force-cycle now. Process alive: the link is merely slow under load, so it keeps probing.
  • Link teardown cannot restart that same-box process, so an arbitrary heartbeat-silence ceiling would neither distinguish load from deadlock nor heal a deadlocked daemon.
  • This is arm structure, not a knob.

State progression

probing → provisioning → connecting → connected. On a drop, disconnected → probing → … at exponential backoff capped at 60s.

  • A warm target still crosses provisioning for its mandatory root refresh. That is normally near-instant, but a GC race may require restoration under the same minutes-long lifetime contract.
  • An uncached exact-source evaluation can enter the phase earlier.

A mirror never fabricates a value

A re-served (mirrored) cell serves no frame until the authority’s first real fold.

  • Its store still seeds the declared default, so store.get() is a typed T.
  • The framework’s hasSnapshot gate withholds that seed until the fold writes the authority’s first frame. On a mirror the only writer is the fold, never a wire client.
  • So the reader’s T | undefined (“no frame yet”) holds end to end, and the declared default belongs to the one writer.
  • Why: Why values replay and byte streams break.