kolu
Docs

tutorial

Make it a daemon.

In Your first surface, fleet-top died when you closed the terminal. Here we make it a daemon: one long-lived process that holds a single-instance gate, serves the same surface over a unix socket, and can be replaced by a fresh build while a client stays connected.

The surface itself does not change. Only how it is reached and kept alive does.

Wrap the engine in a daemon

daemonMain is the whole lifecycle: gate, serve, teardown. It claims an atomic single-instance gate, serves the surface’s { group, handlers } over a socket, waits for the lifetime to end, then closes the socket and releases the gate. It returns a DaemonExit rather than calling process.exit, so a test can drive it.

The bin wraps it in daemonProcessMain, which owns the process exit: when the daemon’s tenure ends, the process ends. top’s live sampler interval cannot keep a finished daemon alive.

/**
 * The daemon — part 1's `top`, now single-instance and durable.
 *
 * `daemonMain` is the whole `gate → serve → teardown` skeleton:
 *
 *   acquirePidGate(GATE_PATH)     — atomic single-instance claim (link(2)).
 *        │ held? → exit 0 (a live daemon already serves this scope)
 *        ▼
 *   serve { group, handlers }     — the @kolu/surface unix-socket listener
 *        ▼
 *   wait for lifetime to end      — { kind: "forever" }: only a signal / abort
 *        ▼
 *   close socket · release gate · return DaemonExit
 *
 * We pick `forever` — an idle `top` still watches your machine, so an idle
 * timeout would wrongly kill it. `daemonMain` never calls `process.exit`; it
 * RETURNS a `DaemonExit`, which is what makes the whole lifecycle drivable
 * in-process from a test. The BIN half — mapping that value to an exit code
 * and actually ending the process — belongs to `daemonProcessMain`: without
 * it, `top`'s live sampler interval would keep this process alive forever
 * after the daemon shut down (the lingering-daemon class).
 *
 * The same `createTop()` surface from part 1 is served verbatim — the daemon
 * changes how it's *reached* (a durable socket instead of a fresh
 * per-connection process), not what it serves. It hands the spine the SAME two
 * fields every transport takes: the flat `group` and the tag-keyed `handlers`.
 */

import {
  daemonMain,
  daemonProcessMain,
  stderrLogger,
} from "@kolu/surface-daemon";
import { HOME } from "../common/paths";
import {
  readProcessIdentity,
  selfProcessIdentity,
} from "../common/processIdentity";
import { createTop } from "./top";

daemonProcessMain({
  name: "fleet-top daemon",
  run: async () => {
    const top = createTop();

    // `finally`, not fulfilled-only: the release stage must run on the crash
    // arm too (a `top.start()` or `daemonMain` throw reaches
    // daemonProcessMain AFTER the sampler is disposed) — the same
    // wrapper-finally teardown ordering kaval and padi use. Everything after
    // `createTop()` sits inside the try so `top.dispose()` is structural.
    try {
      top.start();

      return await daemonMain({
        // gate, socket, anchor — all derived from home inside the spine
        home: HOME,
        processIdentity: selfProcessIdentity(),
        readProcessIdentity,
        group: top.runtime.group,
        handlers: top.runtime.handlers,
        lifetime: { kind: "forever" },
        log: stderrLogger(),
        onReady: ({ socketPath, pid }) =>
          process.stderr.write(
            `fleet-top daemon listening on ${socketPath} (pid ${pid})\n`,
          ),
      });
    } finally {
      top.dispose();
    }
  },
});

We choose { kind: "forever" }: an idle top still watches your machine, so an idle timeout would wrongly kill it. The daemon hands the spine the same two fields every transport takes, the flat group and the tag-keyed handlers, from the same createTop() runtime as part 1, served verbatim.

forever is one of three lifetimes. The daemon reference lists idleTimeout for a quiet coordinator and boundToPid for a daemon a test or smoke run wants to die with it.

anchor is the one required field with no default: the directory whose deletion makes the daemon garbage, polled by the spine so the daemon reaps itself instead of outliving a deleted workspace. Once daemonHome supplies the rendezvous home, that home is the on-disk identity, and fleet-top anchors to it (anchor: () => HOME_DIR) so a deleted home cannot leave a zombie daemon holding unlinked gate and socket inodes.

  1. Start the daemon.

    pnpm run daemon
    fleet-top daemon listening on /run/user/1000/fleet-top/fleet-top.sock (pid 48213)

    That friendly line comes from onReady; stderrLogger also writes a structured JSON line per event to stderr, so you will see a {…"msg":"daemon listening"…} line alongside it. Leave this running.

  2. In a second terminal, start it again.

    pnpm run daemon

    It exits immediately, logging one line that a live daemon already holds the gate (daemon already running; yielding to the live instance), and steps aside. That is the single-instance guarantee: acquirePidGate is an atomic link(2) claim.

Supervise it

A daemon you cannot spawn, watch, and replace is not much use. createEndpoint is the supervisor half of the spine, and it runs in the client, never the daemon. converge(endpoint) is the only boot verb, an always-recycle boot: kill any live survivor, wait for its pid to be reaped, spawn a fresh daemon so it outlives us, connect, and handshake. Every transition is reported.

/**
 * The supervisor — spawn, watch, and LIVE-recycle the daemon.
 *
 * `createEndpoint` is the supervisor half of the spine (it runs in the CLIENT,
 * never the daemon). It takes the daemon from nothing to a live, handshaken
 * connection and reports every transition:
 *
 *   connecting → connected      (recycled, socket up, handshake passed)
 *   connecting → dead           (couldn't spawn / connect)
 *   connected  → degraded       (the daemon died mid-session)
 *
 * `converge(endpoint)` is the only boot verb: a live survivor is killed, then a
 * fresh daemon is spawned — every boot exercises kill → `waitForPidGone` →
 * spawn → connect (composed from `@kolu/surface-daemon`'s gate primitives).
 * `survivableSpawnDriver` launches the daemon so it OUTLIVES us (systemd-run
 * --user under a service; detached + unref otherwise).
 *
 * The finale is the LIVE recycle under a connected client: `recycle` runs the
 * fixed `capture → drain → recycle → reattach` sequence. This demo makes no
 * survival promise, so it supplies the degenerate steps (B2's boot recycle);
 * part 3's remote fan-out is where the same sequence carries real per-host
 * session state. The client we hold reconnects on the far side of the recycle.
 */

import { fileURLToPath } from "node:url";
import type { StreamingProcedure } from "@kolu/surface/client";
import { stderrLogger } from "@kolu/surface-daemon";
import {
  converge,
  createEndpoint,
  recycle,
  survivableSpawnDriver,
} from "@kolu/surface-daemon-supervisor";
import { processStateReader } from "@kolu/surface-daemon-supervisor";
import { Effect, Option, Stream } from "effect";
import {
  bakedOsFactsBin,
  osfactsSocketHolders,
  processIdentityAsync,
} from "osfacts-client";
import { GATE_PATH, HOME, SOCKET_PATH } from "../common/paths";
import type { Memory } from "../common/surface";
import { connectTop, type TopClient, type TopIdentity } from "./connect";

/** The first frame of a snapshot-then-deltas member. */
function snapshot<T>(
  stream: Stream.Stream<T, unknown>,
  what: string,
): Effect.Effect<T, Error> {
  // A member stream fails with `unknown`; the endpoint's `connect` contract is
  // "a plain Error unless it is the branded skew", so the narrowing happens HERE,
  // once, rather than at each caller.
  const head = Effect.mapError(Stream.runHead(stream), (err) =>
    err instanceof Error ? err : new Error(String(err)),
  );
  return Effect.flatMap(head, (head) =>
    Option.isNone(head)
      ? Effect.fail(
          new Error(`${what}: stream closed before its snapshot frame`),
        )
      : Effect.succeed(head.value),
  );
}

const main = Effect.gen(function* () {
  const log = stderrLogger();

  // The daemon binary the driver spawns. In a Nix build this is the realised
  // executable; from source we point `node` at the daemon entry through tsx.
  const daemonEntry = fileURLToPath(
    new URL("../daemon/main.ts", import.meta.url),
  );

  // ONE axis — where this program's osfacts binary lives — resolved ONCE, here,
  // and bound to BOTH OS-fact injects below: a missing bake is a loud boot
  // failure rather than a surprise mid-recovery, and there is only one place to
  // change when the bake moves.
  const osfactsBin = bakedOsFactsBin("KOLU_OSFACTS_BIN");

  const endpoint = createEndpoint<TopClient, TopIdentity>({
    hostId: "local",
    home: HOME, // SAME home declaration as the daemon — disagreement impossible
    // The Effect twin on a supervisor path, so the osfacts spawn never blocks
    // the loop — and so a supervisor that gave up mid-read KILLS the child
    // rather than leaving it to run out its deadline. The daemon half uses the
    // SYNC reader instead (`common/processIdentity.ts`), because its gate claim
    // must not reorder against the boot side effects it guards.
    readProcessIdentity: (pid) => processIdentityAsync(osfactsBin, pid),
    // The second OS-fact inject: who holds the rendezvous socket, for the
    // recovery that runs when the gate no longer names the daemon.
    readSocketHolders: osfactsSocketHolders(osfactsBin),
    // The third: is the holder stopped? A stopped holder cannot run a shutdown
    // handler, so the reap ladder skips its graceful window.
    readProcessState: processStateReader(osfactsBin),
    policy: {
      capability: "not-drainable",
      baked: {
        contractVersion: "1.0",
        build: { kind: "known", id: "fleet-top" },
      },
      onContractSkew: { kind: "recycle" },
      onBuildMismatch: { kind: "nudge-human" },
    },
    probe: () => Effect.succeed(null),
    driver: survivableSpawnDriver({
      binPath: process.execPath, // node
      args: ["--import", "tsx/esm", daemonEntry],
      env: {
        FLEET_TOP_GATE: GATE_PATH,
        FLEET_TOP_SOCKET: SOCKET_PATH,
      },
      unitPrefix: "fleet-top",
      // Launched from source (tsx), not a built binary — so the driver forces
      // the detached branch even under a systemd session, and (as an actual from-source
      // launch) must inherit our ambient env (node/PATH/HOME) to run from source. A built
      // binary would omit `fromSource` and pass a complete env.
      fromSource: { inheritParentEnv: true },
    }),
    // the framework hands you the path
    connect: (socketPath) => connectTop(socketPath),
    log,
    onStatus: (hostId, status) =>
      process.stderr.write(`[supervisor] ${hostId}: ${status.state}\n`),
  });

  // Boot: always-recycle → spawn → connect. Fails (after reporting `dead`) if
  // it cannot bring the daemon up.
  yield* converge(endpoint);

  const conn = endpoint.current();
  if (conn === undefined)
    throw new Error("endpoint connected but current() is undefined");
  const mem = yield* snapshot(
    (conn.client.surface.memory?.get as StreamingProcedure<undefined, Memory>)(
      undefined,
    ),
    "memory",
  );
  process.stderr.write(
    `[supervisor] connected — daemon reports ${conn.identity.cores} cores, ` +
      `${(mem.used / 1e9).toFixed(1)} GB used\n`,
  );

  // The LIVE recycle: kill the daemon under us and stand a fresh one up, with
  // the status held at one honest "restarting". Degenerate steps — nothing to
  // preserve in this part.
  yield* recycle(endpoint, {
    capture: Effect.succeed(undefined),
    drain: () => Effect.void,
    reattach: () => Effect.void,
  });
  process.stderr.write("[supervisor] live recycle complete\n");

  endpoint.current()?.dispose();
});

Effect.runPromise(main).catch((err) => {
  process.stderr.write(`supervisor fatal: ${(err as Error).message}\n`);
  process.exit(1);
});

recycle runs the fixed capture → drain → recycle → reattach sequence, a live recycle. The daemon is killed and a fresh one stood up underneath, while the connection we hold reconnects on the far side. This demo has nothing to preserve, so it passes the degenerate steps. In part 3 the same sequence carries real per-host state.

  1. With a daemon still running from before, start the supervisor.

    pnpm run supervisor
    [supervisor] local: connecting
    [supervisor] local: connected
    [supervisor] connected — daemon reports 8 cores, 9.2 GB used
    [supervisor] local: connecting
    [supervisor] local: connected
    [supervisor] live recycle complete

    The second connecting / connected pair is the live recycle itself: recycle re-runs the endpoint’s boot sequence, so the endpoint reconnects to the fresh daemon and re-fires its status. (stderrLogger also emits a structured JSON line per event to stderr; the [supervisor]-prefixed lines above are the ones to read.)

Watch the order: the supervisor recycled the daemon you had running (the always-recycle boot), connected to the fresh one, read a cell straight off it, then recycled it again live, and the client it held simply reconnected. No dropped session, no lost socket.

Next

fleet-top now survives on one machine. In Across the hosts you reach a fleet-top running on another box over ssh, and watch the same surface arrive in your browser from a machine you never deployed to by hand.