kolu
Docs

tutorial

Make it a daemon.

In Your first surface, fleet-top died when you closed the terminal. Here we make it durable: one long-lived process that holds a single-instance gate, serves the same surface over a unix socket, and can be replaced by a fresh build while a client stays connected. The surface itself does not change at all — only how it is reached and kept alive.

Wrap the engine in a daemon

daemonMain is the entire gate → serve → teardown lifecycle. It claims an atomic single-instance gate, serves the surface’s { group, handlers } over a socket, waits for the lifetime to end, then closes the socket and releases the gate — returning a DaemonExit rather than calling process.exit, so the whole thing is drivable from a test. The bin wraps it in daemonProcessMain, which owns the process exit: when the daemon’s tenure ends, the process ends — top’s live sampler interval can’t keep a finished daemon alive.

/**
 * The daemon — part 1's `top`, now single-instance and durable.
 *
 * `daemonMain` is the whole `gate → serve → teardown` skeleton:
 *
 *   acquirePidGate(GATE_PATH)     — atomic single-instance claim (link(2)).
 *        │ held? → exit 0 (a live daemon already serves this scope)
 *        ▼
 *   serve { group, handlers }     — the @kolu/surface unix-socket listener
 *        ▼
 *   wait for lifetime to end      — { kind: "forever" }: only a signal / abort
 *        ▼
 *   close socket · release gate · return DaemonExit
 *
 * We pick `forever` — an idle `top` still watches your machine, so an idle
 * timeout would wrongly kill it. `daemonMain` never calls `process.exit`; it
 * RETURNS a `DaemonExit`, which is what makes the whole lifecycle drivable
 * in-process from a test. The BIN half — mapping that value to an exit code
 * and actually ending the process — belongs to `daemonProcessMain`: without
 * it, `top`'s live sampler interval would keep this process alive forever
 * after the daemon shut down (the lingering-daemon class).
 *
 * The same `createTop()` surface from part 1 is served verbatim — the daemon
 * changes how it's *reached* (a durable socket instead of a fresh
 * per-connection process), not what it serves. It hands the spine the SAME two
 * fields every transport takes: the flat `group` and the tag-keyed `handlers`.
 */

import {
  daemonMain,
  daemonProcessMain,
  stderrLogger,
} from "@kolu/surface-daemon";
import { HOME } from "../common/paths";
import {
  readProcessIdentity,
  selfProcessIdentity,
} from "../common/processIdentity";
import { createTop } from "./top";

daemonProcessMain({
  name: "fleet-top daemon",
  run: async () => {
    const top = createTop();

    // `finally`, not fulfilled-only: the release stage must run on the crash
    // arm too (a `top.start()` or `daemonMain` throw reaches
    // daemonProcessMain AFTER the sampler is disposed) — the same
    // wrapper-finally teardown ordering kaval and padi use. Everything after
    // `createTop()` sits inside the try so `top.dispose()` is structural.
    try {
      top.start();

      return await daemonMain({
        // gate, socket, anchor — all derived from home inside the spine
        home: HOME,
        processIdentity: selfProcessIdentity(),
        readProcessIdentity,
        group: top.runtime.group,
        handlers: top.runtime.handlers,
        lifetime: { kind: "forever" },
        log: stderrLogger(),
        onReady: ({ socketPath, pid }) =>
          process.stderr.write(
            `fleet-top daemon listening on ${socketPath} (pid ${pid})\n`,
          ),
      });
    } finally {
      top.dispose();
    }
  },
});

We choose { kind: "forever" }: an idle top still watches your machine, so an idle timeout would wrongly kill it. Note the daemon hands the spine the same two fields every transport takes — the flat group and the tag-keyed handlers — from the same createTop() runtime as part 1, served verbatim. (forever is one of three lifetimes — the Reference lists idleTimeout for a quiet coordinator and boundToPid for a daemon a test/smoke run wants to die with it.)

anchor is the one required field with no default: the directory whose deletion makes the daemon garbage, polled by the spine so the daemon reaps itself instead of outliving a deleted workspace. Once daemonHome supplies the rendezvous home, that home is the on-disk identity — fleet-top anchors to it (anchor: () => HOME_DIR) so a deleted home cannot leave a zombie daemon holding unlinked gate/socket inodes.

  1. Start the daemon.

    pnpm run daemon
    fleet-top daemon listening on /run/user/1000/fleet-top/fleet-top.sock (pid 48213)

    That friendly line comes from onReady; stderrLogger also writes a structured JSON line per event to stderr, so you will see a {…"msg":"daemon listening"…} line alongside it. Leave this running.

  2. In a second terminal, start it again.

    pnpm run daemon

    It exits immediately, logging one line that a live daemon already holds the gate (daemon already running; yielding to the live instance), and steps aside. That is the single-instance guarantee: acquirePidGate is an atomic link(2) claim.

Supervise it

A daemon you cannot spawn, watch, and replace is not much use. createEndpoint is the supervisor half of the spine — it runs in the client, never the daemon. converge(endpoint) is the only boot verb — an always-recycle boot: kill any live survivor, wait for its pid to be reaped, spawn a fresh daemon so it outlives us, connect, and handshake. Every transition is reported.

/**
 * The supervisor — spawn, watch, and LIVE-recycle the daemon.
 *
 * `createEndpoint` is the supervisor half of the spine (it runs in the CLIENT,
 * never the daemon). It takes the daemon from nothing to a live, handshaken
 * connection and reports every transition:
 *
 *   connecting → connected      (recycled, socket up, handshake passed)
 *   connecting → dead           (couldn't spawn / connect)
 *   connected  → degraded       (the daemon died mid-session)
 *
 * `converge(endpoint)` is the only boot verb: a live survivor is killed, then a
 * fresh daemon is spawned — every boot exercises kill → `waitForPidGone` →
 * spawn → connect (composed from `@kolu/surface-daemon`'s gate primitives).
 * `survivableSpawnDriver` launches the daemon so it OUTLIVES us (systemd-run
 * --user under a service; detached + unref otherwise).
 *
 * The finale is the LIVE recycle under a connected client: `recycle` runs the
 * fixed `capture → drain → recycle → reattach` sequence. This demo makes no
 * survival promise, so it supplies the degenerate steps (B2's boot recycle);
 * part 3's remote fan-out is where the same sequence carries real per-host
 * session state. The client we hold reconnects on the far side of the recycle.
 */

import { fileURLToPath } from "node:url";
import type { StreamingProcedure } from "@kolu/surface/client";
import { stderrLogger } from "@kolu/surface-daemon";
import {
  converge,
  createEndpoint,
  recycle,
  survivableSpawnDriver,
} from "@kolu/surface-daemon-supervisor";
import { Effect, Option, Stream } from "effect";
import {
  bakedOsFactsBin,
  osfactsSocketHolders,
  processIdentityAsync,
} from "osfacts-client";
import { GATE_PATH, HOME, SOCKET_PATH } from "../common/paths";
import type { Memory } from "../common/surface";
import { connectTop, type TopClient, type TopIdentity } from "./connect";

/** The first frame of a snapshot-then-deltas member. */
function snapshot<T>(
  stream: Stream.Stream<T, unknown>,
  what: string,
): Effect.Effect<T, Error> {
  // A member stream fails with `unknown`; the endpoint's `connect` contract is
  // "a plain Error unless it is the branded skew", so the narrowing happens HERE,
  // once, rather than at each caller.
  const head = Effect.mapError(Stream.runHead(stream), (err) =>
    err instanceof Error ? err : new Error(String(err)),
  );
  return Effect.flatMap(head, (head) =>
    Option.isNone(head)
      ? Effect.fail(
          new Error(`${what}: stream closed before its snapshot frame`),
        )
      : Effect.succeed(head.value),
  );
}

const main = Effect.gen(function* () {
  const log = stderrLogger();

  // The daemon binary the driver spawns. In a Nix build this is the realised
  // executable; from source we point `node` at the daemon entry through tsx.
  const daemonEntry = fileURLToPath(
    new URL("../daemon/main.ts", import.meta.url),
  );

  // ONE axis — where this program's osfacts binary lives — resolved ONCE, here,
  // and bound to BOTH OS-fact injects below: a missing bake is a loud boot
  // failure rather than a surprise mid-recovery, and there is only one place to
  // change when the bake moves.
  const osfactsBin = bakedOsFactsBin("KOLU_OSFACTS_BIN");

  const endpoint = createEndpoint<TopClient, TopIdentity>({
    hostId: "local",
    home: HOME, // SAME home declaration as the daemon — disagreement impossible
    // The Effect twin on a supervisor path, so the osfacts spawn never blocks
    // the loop — and so a supervisor that gave up mid-read KILLS the child
    // rather than leaving it to run out its deadline. The daemon half uses the
    // SYNC reader instead (`common/processIdentity.ts`), because its gate claim
    // must not reorder against the boot side effects it guards.
    readProcessIdentity: (pid) => processIdentityAsync(osfactsBin, pid),
    // The second OS-fact inject: who holds the rendezvous socket, for the
    // recovery that runs when the gate no longer names the daemon.
    readSocketHolders: osfactsSocketHolders(osfactsBin),
    policy: {
      capability: "not-drainable",
      baked: {
        contractVersion: "1.0",
        build: { kind: "known", id: "fleet-top" },
      },
      onContractSkew: { kind: "recycle" },
      onBuildMismatch: { kind: "nudge-human" },
    },
    probe: () => Effect.succeed(null),
    driver: survivableSpawnDriver({
      binPath: process.execPath, // node
      args: ["--import", "tsx/esm", daemonEntry],
      env: {
        FLEET_TOP_GATE: GATE_PATH,
        FLEET_TOP_SOCKET: SOCKET_PATH,
      },
      unitPrefix: "fleet-top",
      // Launched from source (tsx), not a built binary — so the driver forces
      // the detached branch even under a systemd session, and (as an actual from-source
      // launch) must inherit our ambient env (node/PATH/HOME) to run from source. A built
      // binary would omit `fromSource` and pass a complete env.
      fromSource: { inheritParentEnv: true },
    }),
    // the framework hands you the path
    connect: (socketPath) => connectTop(socketPath),
    log,
    onStatus: (hostId, status) =>
      process.stderr.write(`[supervisor] ${hostId}: ${status.state}\n`),
  });

  // Boot: always-recycle → spawn → connect. Fails (after reporting `dead`) if
  // it cannot bring the daemon up.
  yield* converge(endpoint);

  const conn = endpoint.current();
  if (conn === undefined)
    throw new Error("endpoint connected but current() is undefined");
  const mem = yield* snapshot(
    (conn.client.surface.memory?.get as StreamingProcedure<undefined, Memory>)(
      undefined,
    ),
    "memory",
  );
  process.stderr.write(
    `[supervisor] connected — daemon reports ${conn.identity.cores} cores, ` +
      `${(mem.used / 1e9).toFixed(1)} GB used\n`,
  );

  // The LIVE recycle: kill the daemon under us and stand a fresh one up, with
  // the status held at one honest "restarting". Degenerate steps — nothing to
  // preserve in this part.
  yield* recycle(endpoint, {
    capture: Effect.succeed(undefined),
    drain: () => Effect.void,
    reattach: () => Effect.void,
  });
  process.stderr.write("[supervisor] live recycle complete\n");

  endpoint.current()?.dispose();
});

Effect.runPromise(main).catch((err) => {
  process.stderr.write(`supervisor fatal: ${(err as Error).message}\n`);
  process.exit(1);
});

The finale is the payoff: recycle runs the fixed capture → drain → recycle → reattach sequence — a live recycle. The daemon is killed and a fresh one stood up underneath, while the connection we hold reconnects on the far side. This demo has nothing to preserve, so it passes the degenerate steps; part 3 is where the same sequence carries real per-host state.

  1. With a daemon still running from before, start the supervisor.

    pnpm run supervisor
    [supervisor] local: connecting
    [supervisor] local: connected
    [supervisor] connected — daemon reports 8 cores, 9.2 GB used
    [supervisor] local: connecting
    [supervisor] local: connected
    [supervisor] live recycle complete

    The second connecting / connected pair is the live recycle itself: recycle re-runs the endpoint’s boot sequence, so the endpoint reconnects to the fresh daemon and re-fires its status. (stderrLogger also emits a structured JSON line per event to stderr; the [supervisor]-prefixed lines above are the ones to read.)

Watch the order: the supervisor recycled the daemon you had running (the always-recycle boot), connected to the fresh one, read a cell straight off it, then recycled it again live — and the client it held simply reconnected. No dropped session, no lost socket.

Next

fleet-top now survives on one machine. In Across the hosts you will reach a fleet-top running on another box over ssh — and watch the same surface arrive in your browser from a machine you never deployed to by hand.